An evolutionary method for constructing complex SVM kernels

Proceedings of the 10th WSEAS International Conference on MATHEMATICS and COMPUTERS in BIOLOGY and CHEMISTRY
An evolutionary method for constructing complex SVM kernels
DANA SIMIAN
Computer Science Department
Faculty of Sciences
University “Lucian Blaga” Sibiu
Str. Dr. Ion Ratiu 5-7, 550012, Sibiu
ROMANIA
d_simian@yahoo.com
FLORIN STOICA
Computer Science Department
Faculty of Sciences
University “Lucian Blaga” Sibiu
Str. Dr. Ion Ratiu 5-7, 550012, Sibiu
ROMANIA
florin.stoica@ulbsibiu.ro
Abstract: - The aim of this paper is to construct and analyze multiple SVM kernels. The construction is based on a
genetic algorithm which uses a new co-mutation operator called LR-Mijn, capable of operating on a set of adjacent
bits in one single step.
Key-Words: -SVM kernel, Genetic algorithms, Co-mutation
1 Introduction
In recent years SVMs have gained increasing attention
and have become a very popular tool for machine
learning tasks. Applications of SVM have been done in
various fields for accomplishing tasks as classification,
regression and novelty detection ([14]).
The task of classification is to find a rule, which based
on external observations assigns an object to one of
several classes. A classification task supposes the
existence of training and testing data given in the form
of data instances. Each instance in the training set
contains one target value, named class label and several
attributes named features. The goal of SVM is to
produce a model which predicts target value of data
instances in the testing set which are given only the
attributes. Training involves optimization of a convex
cost function If the data set is separable we obtain an
optimal separating hyperplane with a maximal margin.
In order to avoid the difficulties for the non separable
data the kernel method is used. The kernel method is a
very powerful idea. Using an appropriate kernel, the data
are projected in a space with higher dimension in which
they are separable by a hyperplane ([2], [14]). Under
certain conditions, kernel functions can be interpreted as
representing the inner product of data objects implicitly
mapped into a nonlinear feature space. The”kernel trick”
is to calculate the inner product in the feature space
without knowing explicit the mapping function. There
are many standard kernels: linear, polynomial, RBF,
sigmoidal ([2]). Standard kernel-based classifiers use
only a single kernel, but practical applications require
consideration of a combination of kernels ([2], [3]).
Recent developments are oriented in finding complex
kernels and studying their behavior for different
problems. In [3], [4], [11], are presented methods for
solving this problem, using multiple kernels and genetic
programming techniques.
The aim of this paper is to construct and analyze
multiple SVM kernels. The construction is based on a
genetic algorithm which uses a new co-mutation
operator called LR-Mijn, capable of operating on a set
of adjacent bits in one single step.
In present, there is a major interest in design of
powerful mutation operators, in order to solve practical
problems which can not be efficiently resolved using
standard genetic operators. These new operators are
called co-mutation operators. In [7] was presented a co-mutation
operator called Mijn, capable of operating on a
set of adjacent bits in one single step. In [13] we
introduced and studied a new co-mutation operator
which we denoted by LR-Mijn and we proved that it
offers superior performances than Mijn operator.
The paper is organized as follows. In Section 2 we make
a brief presentation of SVM and kernel method. We also
present the basic idea of evolutionary method adopted
for multiple SVM kernel construction. In section 3 is
introduced the co-mutation operator LR-Mijn. The
evolutionary algorithm based on LR-Mijn operator, which
we use for multiple kernel construction is presented in
section 4. Section 5 contains the main results: the model
for constructing multiple SVM kernels and the
experimental results. Conclusions and further directions
of study can be found in section 6.
2 Support Vector Machines and kernels
2.1 Problem definition
SVM algorithm can solve the problem of binary or
multiclass classification. There are many known
ISSN: 1790-5125 172 ISBN: 978-960-474-062-8

methods to generalize the binary classifier to a n - class
classifier ([14]). Therefore we consider in this section
only the problem of binary classification.
Let be given the data points d
xi ∈R , i=1,…,m and their
labels yi∈{-1,1}. We are looking for a decision function
f, which associates to each input data x its correct label
y=f(x). The form of the decision function is
f(x)=sign(<w,x>+b) (2.1)
m
w = Σ
α x
i i
i
=
1
If the data set is separable then the conditions for
classification without training error are
yi(<wi; xi>+b) >0 (2.2)
For the data sets which are not linearly separable we use
the kernel method, which makes a projection of the input
data X in a feature Hilbert space F:
φ : X →R; x→φ (x) (2.3)
The functional form of the mapping φ ( x ) i does not need
to be known. It is implicitly defined by the choice of
kernel:
K ( x , x ) =<φ ( x ), φ ( x ) >
i j i j (2.4)
The kernel represents the inner product in the higher
dimensional Hilbert space of features. Feasible kernels
must satisfy Mercer’s conditions.
2.2 Multiple kernels
Usually the choice of the kernel is made empirically and
the standard SVM classifiers use a single kernel. Recent
papers proved that multiple kernels give better results
than the single ones. It is known from the algebra of
kernels that the set of operations
(+,*,exp) (2.5)
preserves the Mercer’s conditions and therefore we can
obtain multiple kernels using these operations. One
possibility is to use a linear combination of simple
kernels and to optimize the weights ([1]). For
optimization the weights two different kind of
approaches can be found. One of them reduces the
problem to a convex optimization problem. Other uses
evolutionary methods for optimizing the weights. In [3]
and [11] a hybrid approach using a genetic algorithm
and a SVM algorithm is proposed. Every chromosome
codes the expression of a multiple kernel. The quality of
a chromosome is represented by the classification
accuracy (the number of correctly classified items over
the total number of items) using the multiple kernel
coded in this chromosome and it is obtained running the
SVM algorithm. The hybrid techniques from [3] is
structured in two levels: a macro level and a micro level.
The macro level is represented by the genetic algorithm
which builds the multiple kernels. The micro level is
represented by the SVM algorithm which computes the
quality of chromosomes. The accuracy rate is computed
by the SVM algorithm on a validation set of data.
3 The LR-Mijn operator
In this section we define the co-mutation operator called
LR-Mijn. Our LR-Mijn operator finds the longest
sequence of σp elements, situated in the left or in the
right of the position p. If the longest sequence is in the
left of p, the LR-Mijn behaves as Mijn, otherwise the LR-Mijn
will operate on the set of bits starting from p and
going to the right.
Let us consider a generic alphabet A = {a1, a2, …, as}
composed by s ≥2 different symbols. The set of all
sequences of length l over the alphabet A will be denoted
with Σ = Al.
In the following we shall denote with σ a generic string,
and σ = σl-1…σ0 ∈ Σ= Al, where σq ∈ A ∀ q ∈ {0, …, l-
1}. Through σ(q,i) we denote that on position q within
the sequence σ there is the symbol ai of the alphabet A.
σz
p, j denotes the presence of z symbols aj within the
sequence σ, starting from the position p and going left
right ,
n
and
p i
σ ( , ) specify the presence of symbol ai on
left ,
m
position p within the sequence σ, between right symbols
an on the right and left symbols am on the left. We
right ,
i
suppose that σ = σ(l-1)... σ(p+left+1,m)
p i
σ ( , ) σ(p-right-
left ,
i
1,n)... σ(0).
The Mijn operator is the mutation operator defined in
[7]:
Mijn : σ ∈ Σ, p ∈ {0, … , l-1} → σ’ ∈ Σ’ ⊂ Σ, where p
is randomly chosen
(i) σ = σl-1…σp+nσp+n-1,i σn-1
p, j σp-1…σ0 ⎯⎯⎯→ Mijn
σ’ = σl-1…σp+nσp+n-1,j σn-1
p,i σp-1…σ0, for n < l – p + 1
and
(ii) σ = σ -p
l σp-1…σ0 ⎯⎯⎯→ Mijn σ’ = σ -p
p, j
l σp-1…σ0,
p,k
for n = l – p + 1, with ak ≠ aj randomly chosen in A.
In [13], we introduced and study the properties of LR-Mijn
co-mutation operator
Definition 3.1 Formally, the LR-Mijn operator is defined
as follows:
(i) If p ≠ right and p ≠ l – left – 1,
LR-Mijn(σ)=
ISSN: 1790-5125 173 ISBN: 978-960-474-062-8

⎧
σ σ σ σ
⎪ ⎪ ⎪ ⎪ ⎪
σ σ
σ σ σ σ
⎨
⎪ ⎪ ⎪ ⎪ ⎪
⎩
right ,
i
l p left i p m
( 1)... ( 1, ) ( , )
= − + +
left m
p right n for left right
( 1, )... (0)
,
− − >
,
l p left m p n
( 1)... ( 1, ) ( , )
= − + +
p right i for left right
( 1, )... (0)
,
σ σ
− − <
or for left =
right with probability
, 0.5
left
rightt
right rleft
right n
left i
σ σ
(ii) If p = right and p ≠ l – left – 1,
right ,
i
σ = σ(l-1)... σ(p+left+1,m)
p i
σ ( , ) and
left ,
i
LR-Mijn(σ) = σ(l-1)... σ(p+left+1,m)
right ,
k
p k
σ ( , ) ,
left ,
i
where k ≠ i (randomly chosen).
(iii) If p ≠ right and p = l – left – 1,
right ,
i
σ =
p i
σ ( , ) σ(p-right-1,n)... σ(0) and
left ,
i
LR-Mijn(σ) =
right ,
i
p k
σ ( , ) σ(p-right-1,n)... σ(0), where k ≠ i
left ,
k
(randomly chosen).
(iv) If p = right and p = l – left –1, σ =
right ,
i
p i
σ ( , )
left ,
i
LR-Mijn(σ)=
⎧
σ σ
⎪ ⎪ ⎪
σ σ
⎨
⎪ ⎪ ⎪
σ σ
⎩
right i
p k for left right where k i
( , ) , ,
= > ≠
left k
right k
p k for left right where k i
( , ) , ,
= < ≠
=
, 0.5
,
left ,
i
,
,
or for left right with probability
left
rightt
right rleft
As an example, let us consider the binary case, the string
σ = 11110000 and the randomly chosen application
point p = 2. In this case, σ2 = 0, so we have to find the
longest sequence of 0 within string σ, starting from
position p. This sequence goes to the right, and because
we have reached the end of the string, and no occurrence
of 1 has been met, the new string obtained after the
application of LR-Mijn is 11110111.
The commutation operator LR-Mijn allows long jumps,
thus the search can reach very far points from where the
search currently is. We proved in [13] that the LR-Mijn
operator performs more long jumps than Mijn, which
leads to a better convergence of an evolutionary
algorithm based on the LR-Mijn in comparison with an
algorithm based on the Mijn operator.
In the following, we will consider that A is the binary
alphabet, A = {0, 1}.
4 The evolutionary algorithm based on
LR-Mijn operator
The basic scheme for our algorithm, called in the
following LR-MEA, is described as follows:
Procedure LR-MEA
begin
t = 0
initialize randomly population P(t) with P elements;
Evaluate P (t) by using fitness function;
while not Terminated
for j = 1 to P-1 do
- select randomly one element among the
best T% from P(t);
- mutate it using LR-Mijn;
- evaluate the obtained offspring;
- insert it into P’(t).
end for
choose the best element from P(t) and
insert it into P’(t)
P(t+1) = P’(t)
t = t + 1
end while
end
5 Main results. The model for
constructing complex SVM kernels
Our goal is to build and analyze a multiple kernel
starting from the simple polynomial kernels:
+ K = x ⋅ x + r d where r d ∈ Ζ
d ( ) , , 1 2 (5.1)
and having two parameters, the degree d and r.
We use the idea of the model proposed in [3]. In a first
level, we will build and evaluate multiple kernels
obtained from (5.1) using a genetic algorithm and the set
of operations op ∈{+,∗, exp}, i = 1,3 i .
5.1 Representation of the multiple kernel
We consider the particular case in which parameter r is
the same for all simple kernels used in our composition
procedure.
In the figure 1 is represented the kernel
1 , 2 2 , 1 3 , 3 4 , ( ) ( ) d r d r d r d r K op K op K opK :
ISSN: 1790-5125 174 ISBN: 978-960-474-062-8

Fig. 1 General representation of multiple kernel
If a node contains the operation exp only one of its
descendants is considered (the “left” kernel).
The chromosome which codes the multiple kernel
described above has the following structure:
op1 op2 op3 d1 d2 d3 d4 r
Each operation i op is represented using two genes, for a
degree j d are allocated four genes, and the variable r is
represented using twelve genes. Thus, our chromosome
is composed from 34 genes.
The evaluation of the chromosome is made using the
SVM algorithm for a particular set of data. To do this we
divide the data into two subsets: the training subset, used
for problem modeling and test subset used for
evaluation. The training subset is also random divided
into a subset for learning and a subset for validation. The
SVM algorithm uses the data from the learning subset
for training and the subset from the validation set for
computing the classification accuracy which is used as
fitness function for the genetic algorithm.
5.2 SVM algorithm
For the implementation/testing/validation of our method
was used the “leukemia” data set from the page
LIBSVM data sets page ([2]).
In order to replace the default polynomial kernel from
libsvm, we extend the svm_parameter class with the
following attributes:
// parameters for multiple polynomial kernels
public long op[]; // operations
public long d[]; // degrees
public long r[]; // parameters
The class svm_predict was extended with the following
method:
public double predict(long op[], long d[], long r){
double result;
try
{
BufferedReader input = new BufferedReader(
new FileReader(test_file));
DataOutputStream output = new DataOutputStream(
new FileOutputStream("predict.out"));
svm_model model =
svm.svm_load_model(model_file);
model.param.op = op;
model.param.d = d;
model.param.r = new long[4];
for (int i=0; i<4; i++)
model.param.r[i]=r;
model.param.kernel_type = svm_parameter.POLY;
result=predict(input,output,model,0);
input.close();
output.close();
return result;
}
catch(FileNotFoundException e) {
exit_with_help();
}
.....
return 0;
}
The Kernel class was modified to accomplish the kernel
substitution. In the k_function method, our simple
kernels are computed as follows:
K = new double [4];
for (k=0; k<4; k++)
K[k]=powi(dot(x,y)+param.r[k],param.d[k]);
Then, the kernels are combined using operation given in
array param.op[].
In the genetic algorithm, the operations, the degrees of
simple kernels and the parameter r are obtained from a
chromosome, which is then evaluated using the result of
the predict method presented above.
Thus, the SVM algorithm uses the learning subset of
data for training the SVM model and the testing subset
for computing the classification accuracy that is the
fitness function for the genetic algorithm.
After the end of the genetic algorithm, the best
chromosome gives the multiple kernel which can be
evaluated on the test subset of data. The way of
construction this multiple kernel assures that it is a
veritable kernel, that is, it satisfies Mercer’s conditions.
5.3 Experimental results
Using the standard libsvm package, for the “leukemia”
data set is obtained the following classification accuracy:
java -classpath libsvm.jar svm_predict leu.t leu.model leu.output
Accuracy = 67.64705882352942% (23/34) (classification)
Multiple kernels obtained using genetic approach are
op1
op2 op3
d r K 1 , d r K 2 , d r K 3 , d r K 4 ,
ISSN: 1790-5125 175 ISBN: 978-960-474-062-8

improving the classification accuracy up to 91.18%. In
the figure 2 are presented results from three runs of our
genetic algorithm based on LR-Mijn operator. For each
execution, dimension of population was 35 and the
number of generations was 30.
Fig. 2 Classification accuracy using multiple kernels
One “optimal” multiple kernel obtained is depicted in
figure 3, where r = 2111.
:
Fig. 3 Representation of an optimal multiple kernel
If we impose the additional condition that di>0, i=1,…,4,
we obtain the same accuracy after 30 generation, using
only + and * operations.
Fig. 4 Classification accuracy, for di>0
One “optimal” multiple kernel obtained is depicted in
figure 5, where r = 2057.
+
* +
K2,r r K 1, r K 1, r K 3,
Fig. 5 An optimal multiple kernel, for di>0
6. Conclusions and further directions of
study
In this paper we presented a model for a SVM multiple
kernels, which contains only polynomials simple kernels
and is based on a genetic algorithm using a new co-mutation
algorithm. Experimental results prove that the
utilization of genetic algorithm based on LR-Mijn co-mutation
operator, has a better convergence and
improves the accuracy toward the classical genetic
algorithm used in [7].
We use the idea introduced in [3], but our model is
different, the structure of chromosomes which code the
multiple kernels is different and the macro level that is
the genetic algorithm used is different.
The experimental results prove that the operations + and
* have a better behavior with polynomial kernels than
the exponential operation.
As a further direction of study we want to compare the
results obtained using not only polynomial kernels and
characterize the behavior of the composition operation
+,* and exp related to the kernel type.
References:
[1]Baudat G., Anouar F., Kernel-based Methods and
Function Approximation, International Joint
Conference on Computer Network, Washington,
2001, 1244 – 1249.
[2]Chang C-C., Lin C-J., LIBSVM : a library for
support vector machines, 2001. Software available
at http://www.csie.ntu.edu.tw/ cjlin/libsvm.
[3]Diosan L., Oltean M., Rogozan A., Pecuchet J. P.,
Improving SVM performance using a linear
combination of kernels, Adaptive and Natural
Computing Algorithms, ICANNGA07, volume 4432
of LNCS, 2007, 218 - 227.
[4]Diosan L., Oltean M., Rogozan A., Pecuchet J. P.,
Genetically Designed Multiple-Kernels for
Improving the SVM Performance, portal VODEL,
http://vodel.insa-rouen.fr/publications/rfia, 2008.
[5]De Falco I., An introduction to Evolutionary
Algorithms and their application to the Aerofoil
+
exp +
1,r K r K 2, r K 3,
ISSN: 1790-5125 176 ISBN: 978-960-474-062-8

Design Problem – Part I: the Algorithms, von
Karman Lecture Series on Fluid Dynamics,
Bruxelles, April 1997
[6]De Falco I., Del Balio R, Della Cioppa A.,
Tarantino E., A Comparative Analysis of
Evolutionary Algorithms for Function Optimisation,
Research Institute on Parallel Information Systems,
National Research Council of Italy, 1998
[7]De Falco I, A. Iazzetta, A. Della Cioppa, Tarantino
E., The Effectiveness of Co-mutation in Evolutionary
Algorithms: the Mijn operator, Research Institute on
Parallel Information Systems, National Research
Council of Italy, 2000
[8]De Falco I., Iazzetta A., Della Cioppa A., Tarantino
E., Mijn Mutation Operator for Aerofoil Design
Optimisation, Research Institute on Parallel
Information Systems, National Research Council of
Italy, 2001
[9]Golub T. R.,. Slonim D. K,. Tamayo P, Huard C.,
Gaasenbeek M. ,. Mesirov J. P,. Coller H, Loh M. L.,
Downing J. R., Caligiuri M. A.,. Bloomfield C. D,
Lander E. S., Molecular classification of cancer:
class discovery and class prediction by gene
expression monitoring. Science, 286(5439):531,
1999.
[10] Muller k.-R., Mika S., Ratsch G., Tsuda K.,
Scholkopf B., An Introduction to kernel-Based
Learning Algorithms, IEEE Transactions On Neural
Networks , Vol. 12, No. 2, 2001, 181 – 202.
[11] Simian D., A Model For a Complex Polynomial
SVM Kernel, Proceedings of the 8-th WSEAS Int.
Conf. on Simulation, Modelling and Optimization.
Santander Spain, 2008, within Mathematics and
Computers in Science and Engineering, pp. 164-170
[12] Sonnenburg S., Rtsch G., Schafer C., Scholkopf
B, Large scale multiple kernel learning, Journal of
Machine Learning Research, 7, 2006, 1531 - 1565.
[13] Stoica F., Simian D., Simian C., A new co-mutation
genetic operator, Advanced topics on
evolutionary computing, Proceeding of the 9-th
Conference on Evolutionay Computing, pp. 76-82
[14] Vapnik V., The Nature of Statistical Learning
Theory, Springer Verlag, 1995.
http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/dataset
s
ISSN: 1790-5125 177 ISBN: 978-960-474-062-8

An evolutionary method for constructing complex SVM kernels

Recommended

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to An evolutionary method for constructing complex SVM kernels (20)

More from infopapers (17)

Recently uploaded (20)

An evolutionary method for constructing complex SVM kernels