An Integer Programming App! roach to Inductive Learning Using Genetic Algorithm Janusz Kacprayk and Grazyna Sekatula Systems Research Instat, Polish Academy of Sciences ‘ul Newelska 6, 01-48 17 Warsaw, Poland smal: Kacprzyk@ibspan wa pl [Abstract - We propose an improved inductive learning method to derive classification rules that correctly describe most of the examples belonging to a clase and do not deseribe mort of the ‘examples not belonging to this class. The problem Is represented ‘5 a modification of the set covering problems solved by a fenetic algorithm. The results are very encouraging. 1. INTRODUCTION Machine leaming from examples is a process of inferring a classification rule (concept description) of a class from positive and negative examples. In practice, mainly duc to imperfect data, the requirements to be satisfied by leaming procedures are * @ partial completeness, ie. that the classification rule must correctly describe (have the same attribute values), say, most ofthe positive examples, 4@ partial consistency, i. that the classification role must describe, say, almost none of the negative examples, convergence, i.e. the classification rule must be derived in a finite number of steps, the classification rule of minimal “length” is to be found, e.g. with the minimum number of attributes (or, more generally being ”simple”). Examples are described (cf. Michalski [22)) by a set of K “attribute - value” pairs platy) where a, denotes attribute j with value v, and # is a relation as, ¢.8, =, <, >, =2, ete. For instance, for the attributes: height, color_of hair, color_of_eyes, we can describe the look of a person as [height = "high"] [color_of-hair = "blond" » {color_of eyes = "blue"} We propose # modified inductive learning procedure based on Michalski’s [22] star-type methodology. The ‘method is based on elements of the authors" previous work [ef. Kacprzyk and Szkatula [15 - 20}. A pre- processing of data (examples) is. first performed based fon. an analysis of how frequent the values of the particular attributes occur in the examples. These (0-7803-7282-4102/$10.00 ©2002 IEEE. 181 frequencies are used to derive weights associated with those values, and the problem is a modification of the set covering problems, and solved by a modification of ‘a genetic algorithm (IP2_GA). IL, PROBLEM FORMULATION OF INDUCTIVE LEARNING FROM EXAMPLES ‘We have finite sets of examples U' and attributes AR lata). Va; =0h/ avi!) is a domain ay, F=lynK, V= UO Vay. f:UxA-¥ is a function i such that f(e,a,)eV,, for Va, eA, VeeU, called an informational function. Each ¢ €U is described by K attributes, A= {a,,...ax} , and is represented by e= Ala, =vi'] a where vf! = f(e.a,)€V,, denotes the jth attsbute a, taking on a value vj! for example e ‘An example ¢ in (1) is composed of K "attribute- value” pairs, denoted 5, =[a, = 7] (electors). The conjunction of 1 K "atribute-value” pairs, ic Cl= 5,5 la, =077) = 1 where J = {j1.J2s-5J1} S {le-K} is called a complex. la ornate @ Suppose we have example [et (1)] and considera complex fet.) C= fey =¥04 Jrunley, 90) that conetponds tothe set of dos Te Unt 5 flak). The et of indies Ute) Is clearly equivalent to a vector x=(2)]7, f= lhusK, such that x, if'a selector sj ={a;=vj 4] occurs in the complex C! , and 0 otherwise. For instance, for K =‘’ [color_of_eyes = "blue”}, vector [0,1,0]" is equivalent to the complex [color_of hair = "blond”); the complex [height = "high”] is equivalent to the vector (1,0,0]" . ‘Complex C' covers example e if all conditions on attributes given as selectors are covered by (equal to) the values of the respective attributes in e, ic. FC! aj) = fleaj Wiel We assume that a, is a decision attribute and = (08 aff} is @ domain of ay. Each example eeU is described by a set of attributes fai,a-aK}U{ag}- Observe that attribute a, determines a partition Vjog sY,oq os¥,ag } of set U, Mg where Yroy =leeUsfleag)= 00}, vil eVey for d, and ¥ ag UY ag =U, Yong OY aq =D “i ved spe Oe for i# j The set ¥joq is called the -th decision class (for vf © Vay) Suppose that we have a set of positive examples: } @) leeU: fleas) SOs and a sot of negative examples Ses les Usleag)# 994 and Vere SpUY og) Baye P,Mle.aj)# S(e.a))} @) So, SpUag) Swag )=@ and Sp ag #2, a " C Sy gy )#, by assumption; this assumption should hold in nontrivial problems, with larger datasets. The mule IF C! THEN [a,=vj*] is called an “elementary” rule for class Y.,,, where C! is a escription of example in terms of attributes a, j€1, and this example belongs to class ¥,,, . We consider the classification rules being the disjunction (via "U") of "elementary" rules consisting of complexes of type (2), (0-7803-7282-4/02/$10.00 ©2002 IEEE IF Chu..uC THEN [ay with: Nealt Stl 2K) ,Ch= la, ns Suppose now that we have P positive examples, e750, )y m= lungP and N negative examples, Sy Kye Bebe For each a,, each possible value occurs at some intensity (frequency). Ifa value occurs more frequently in the positive examples and less frequently in the negative ones, then it should rather appear in the rule sought. These frequencies should clearly be relative. This rationale may be formalized as: we introduce the function, foreach a,, j=1,....K and ve V,, 1S gem tee £0)= LB yee ~) © for each v eV, , where: aen=ft for yt ay, - 0 otherwise and: &" € 5p, vj! = f(e™,a))eVq,; and analogously for &(e".») So, we may expresses to what degree (from [-1, 1]) the particular values ve ¥,, of attribute a, occurs more often in the positive than negative examples. This idea is clearly more applicable to larger data sets. ‘We assume that ;(v)¢[-L] is used as a weight of value ve V,, of each a, (cf. Kacprzyk and Szkatula [19, 20). An example ey with weights is written as ev= Alay 15 8 )(¥i)) M ie. is a conjunction of weighted selectors, s¥ = 25 (0 Mie. Ce a rich and is called a weighted complex. Notice that for this Cj, vector x has the elements 1 for je, while, for je {0.2,..KIVI, x, ® Fora Ch its weighted lengths: 182dy (Cp) = LO-8 joj! ))-xj += ier x 3, La-/Of xy +g f/ x= 2 MF Jett 0) which reflects a higher relevance of those values of attributes which occur more often in the positive than i negative examples. The length of the weighted classification rule of L weighted complexes, Ry= Cf U..UCh is Ay (CH CGP) = ax, div Cif) (do) ‘The problem of is to find an optimal classification rule Ry = Cf" ULC" such that min, day (Cf) vu) ay Dealt ‘As the (exact) solution of (11) is very difficult, an auxiliary problem is solved (ef. Kacprzyk and Szkatula [6p, ie an Rj = Ch" UC is sough such that min dy (Ci) min dy (CH) (a2 ‘The solution of (12) is in general very close to that of (11), while much easier to obtain. IIL SOLUTION BY USING THE IP2.GA METHOD We have a finite set of examples U fef. (1)] and a finite set of attributes {4,243.4} U {ay}, ay isa decision attribute, ¥,, = (0%4,...f/} is a domain of a. We have the Y .d) decision classes for ¥, €%,. We have a set of positive examples Sp, © €Sp, pm IousP and a set of negative ones Sw, eM eSy, NV, Sy.Sp #D, SprSy =O The problem: to find an optimal classification rule Wooly poly” S Moos} minimizing the weighted length of the classification rule, can be represented as a modification of the set covering problems (Kacprzyk and Szkatula [16)). For e? Sp, and all the negative examples oP eSy, m=lusN, We construct a O-l matrix ZyexLeyls Mls Je hnoK, defined as (0-7803-7282-4/02/8 10.00 ©2002 IEEE 183 Jt stor see? ajy= fle **,a;) . 3) 0 for sleP aja feeP™ “ : -4)) Tis rows correspond to the consecutive negative examples €5y, m= loud” and its columns tothe subsequent attributes aj, nag Fy =1 occurs if a, takes on different values in the positive and negative example, ie. f(ef.a,) # fle"*,a,), and ry =0 otherwise, Tere are clearly no rows with al elements qual 0 since the sts of postive and negative examples are disjoint (and non-empty). Thus, for any positive and negative example there always exists at least one attribute taking ona different valuc in these examples. Consider now the following inequality ‘ aay where y=[Y1...1w]" is a zero-one vector, and 22 [sys soot]? such that x, © (0), for j=1,..,K Any vector x which satisfies Zr 2y (14) determines therefore in a unique way some complex composed of selectors from the description of the example such that the conditions of partial completeness and partial consistence are satisfied. It describes at least one example from the set of positive examples, and it does rot describe most of the examples from the set of negative examples. If vector x does not describe the n= th negative example, then y, =1; and 7, = Ootherwise “The minimization in (12), using inequality (14), is K @ min S0-8/094 yxy (5) weezy jay The minimization over the set of indices J, may be replaced by the minimization with respect to x which yields [ef @)] an Ry = Cf" U..UCH" such that rnin dy (CP) eosin dy (CHP) i 6 eZixay thray Each minimization with respect to x in (16) is therefore equivalent to the determination of a 0-1 vector x" Which uniquely determines the complex of the shortest weighted length. On the other hand, the satisfaction of ZA (A is a unit vector) guarantees that such a complex would not describe all negative examples. If rules defining clas Y.,, must deseribe almost none ofthe negative examples, problem (15) can be written as a modification ofthe set covering problem _k min Sejxy an “1 fat K Lenjxj2tn > " (18) i with an additional constraint y Sit 2N rel as) ml where =8)0)/), zy ell}, xy 0, FA book, Y= Otetwls Yee ON, given a parameter rel = 0 This isthe same as the original set covering problem with the exception that no more then rel rows are uncovered. Then, clearly that no more then rel rows can be deleted from the problem. We may, in deleting rows, lose some information about the problem that could have been better used. This reduction cannot always be applied. In the set covering problem (cf. Beasley and Chu [4]) there is only constraint (18), and 7=(tewtw]" is unit vector. The (17) ~ (19) is the problem of covering at least Nerel rows of an N-row, K-column, zero-one matrix (2y) by a subset of the columns at minimal cost c, We define x; =1 if column j with cost ¢, > 0 is in the solution, and x, =0 otherwise. Equations (18) and (19) ‘ensures that most rows (atleast N-rel rows) are covered by at least one column, It always has a feasible solution (@ unit vector x of K element), due to the required disjointness of the sets of positive and negative ‘examples and the way the matrix Z was constructed So, we look for a 0-1 vector x at minimum cost and a o- (Yie-YwT” which determines the covered rows, ¥,= 1 if n-th row is covered by solution x, and 7, 0, otherwise. By assumption, at least N-rel rows must be covered by solution x: The set covering problem is a well-known combinatorial optimization problem and is NP- complete. A number of optimal and faster heuristic algorithms have been proposed, cf. Grossman and Wool [10], Beasley and Chu [4] presented a genetic algorithm, with modified operations. For the solution of (16) we propose a new procedure, IP2_GA, based on a genetic algorithm. We vector (0-7803-7282-4/02/$10.00 ©2002 IEEE assume that the classification rules must correctly describe most of the examples, at 1east Alig tHe measure of classification accuracy Ajyring isthe percentage of examples correctly classified. We assume a K-bit binary string which represents a potential solution structure, where K is the number of variables (ic. columns in the set covering problem). ‘The value I for the j-th bit implies that column jis in solution x', ie. that x} is in the solution, In IP2.GA in each iteration all solutions are evaluated with respect to their completeness and consistency. We adopt a simple approach of using a penalty evaluation function which assigns utility to candidate solutions. The fitness of an individual solution x is x enle)= 10) Bac 2 fal) 20) Kc 0 for Sy2qj-xj >0 jet @ 1 for S29 ist Kk where: (x)= Leyxys fats)= jal ° Boa = MAKE)? feos K) ym value of the j-th column in the string corresponding to the solution x and c, isthe cost of -th column, The structure of a new population is chosen by a stochastic universal sampling (cf. Baker (1) with the wheel spun with a number of equally spaced markers equal to the population size ‘The consecutive steps of IP2_GA are: LouoN, where x, is the Step 1. Set the initial values: = Sp, ie. the whole set of examples is initially assumed to contain the positive cones, Sy is a set of negative examples, and Ry =, ie. the initial set of complexes is assumed empty, iteration j=, given parameter rel 20. Step 2. Iteration j = j + 1. Determine the weights G by analyzing (preprocessing) of the examples due to (6). Step 3. Determine an appropriate starting point; a good starting point may be a so-called centroid (ef. Kacpreyk and Szkatula [96]) that is some (possibly non existing) ‘example in which the attributes take on values that ‘occur most often in the positive examples and seldom in the negative examples. In the set of positive examples ‘we find the closest positive example e” to centroid, as the starting point for the next iterations, 184Step 4. For the ef we form the matrix Zyeq-{2y/]> Ny JK, due to (13) and form a modification of the set covering problem, due to (17), (18), (19). Step 5. We apply a genetic algorithm, Step 1”. Set = 7. Generate an initial population of random solutions P() = (&',x%, 4x"). Each solution is simply a binary string of length K. Evaluate the fitness eval(x') of individuals in the population, 1=1,2,...P- Step 2’. . For the first solutions a crossover ‘operator is applied. Two solutions are chosen and form two new sotutions. Step 3° A mutation operator is applied to each solution in the population. Step 4°. The new solution generated by the crossover and mutation procedures may not be feasible, We evaluate the fitness eval(x') of new individuals in the population, Step 5°. If a termination condition is satisfied, then STOP, and the best solution is the one with the smallest fitness; otherwise, go to Step 6” Step 6°. Select a new population P(t+/) from population P()) and return to Step 2" ‘The 0-1 vector x” sq] found determines in ‘a unique way the complex Cj and the 0-1 vector Y=[%ienty]” determines the fulfilled constraints. The complex can not describe more than rel examples (given a parameter rel 20). Now, we can go to Step 6. Step 6. Include complex Cy/ found in Step $ into the classification rule sought Rjy (i.e. that with the minimal weighted length), Ry = Ry U Cfisa(Ch), where 1 _mumberof examples covered by Cf «Cr number of all examples and discard from the set of examples 5 all examples covered by complex Cj, Step 7. If the set of examples $ remaining is small enough, STOP and the nile Riy = Ch sa Gp ICH ACH) Diyorte, © thoy K), i8 the one sought; otherwise, return to Step 2. 0-7803-7282-4/02/810.00 ©2002 IEEE 185 The IP2_GA algorithm described above is relatively simple and efficient. It requires a number of parameters, eg. the population size, probabilities of applying genetic operators, ec. IV. APPLICATION OF THE IP2_GA ALGORITHM TO, SOLVE A THYROID CANCER PROBLEM For a lack of space, we only consider the example of a medical data set of Nakache and Assclian (23) that concems 281 patients with thyroid cancer subjected to a surgery. Each patient was described by 12 attributes: sex, (male, female} age, {<40, 40-60, 60-70, >70) histology, {well differentiated, poorly differentiated}, metastasis, {yes, no}, enlargement, (uni-lobe, uni- lobetisthm, al the thyroid}, clinical lymph nodes, fyes, no}, clinical aspect, (unique nodule, multi nodules, important enlargement}, pathological lymph nodes, {yes, no}, compressive syndromes, (yes, no}, invasion, {no, small, average, large}, survival time, {in months}, length in month of survival time from the entrance in the study (between 1960 and 1980) to the time of analysis, survival, (survivor, non survivor at time of analysis}. Two attributes are important: the survival time (in month) atthe time of analysis, and the survival ‘or non survival atthe time of analysis. ‘The purpose is to find a prognostic rule for a new ‘case coming from the same population and being in the same conditions. ‘We have two classes: class 1: the patients will be alive over 7 years, class 2: the patient will dead during 7 years. IF Ry THEN [survival tim IF Rj THEN [survival time = below7 years] that must correctly describe most of the examples ‘belonging to class I and 2, with atleast Amy = 97-5 The IP2GA and IP2_GRE (with elements of a ‘greedy algorithm) (Kacprzyk and Szkatula (19, 20]) methods were applied The results of applying the methods to medical data are presented and described below. “TABLE 1. SOME PARAMETERS DESCRIBING THE PROCESS OF FINDING A CLASSIFICATION RULE POR THE CLASS 1o o ror ‘TABLE 2, SOME PARAMETERS DESCRIBING THE PROCESS OF FINDING A CLASSIFICATION RULE FORTHECLASS?- (7) Algorithm | Numberof | Number of selectors . iterations ‘neue a TF GRE, n is 110, 12 GA 2 ‘TABLE 3. SOME PARAMETERS DESCRIBING THE PROCESSOF (1), (CLASSIFICATION THE PATIENTS INTO THE FIRST OR THE SECOND CLASS 0 ‘Classification Agoriths | Aig Ys a3 ort rang % bY os sssumption ws. TPE GRE | atleast 975% | 98.7% atleast 97.5% | 98.7% 0s, ‘A classification rule forthe first class is: IF [metastasis = no] [clinical lymph nodes = no} x 16. Linvasion = no]; 0.468 U [compressive syndromes = no\alinvasion = (17. average}; 0.082 U [clinical aspect = unique nodule] {clinical lymph nodes = yes}; 0.104 ‘THEN [survival time = over 7 years] 0g. and the shortest classification rules were obtained by using the method IP2_GA. 19, V. CONCLUDING REMARKS We proposed an improved inductive leaming procedure 1P2_GA with clements of a genetic algorithm to derive (0. classification rules sets of positive and negative (9, examples. Results seem to be very encouraging. Bibliography @ 4 (0), Baker 18. (1967) Reducing bas and iene fn the sletion igor, In: Gerec, Algo and The Applitons Proceedings of the 2" Inematonal "Conftence ot Genet Algor (cd) 1. Grefesete, 1421, LEA, Canbdge, MA. (25. [2], Balas E- (1980) Cuting panes fom condo! bounds: = cw spec set vere Mate Progamming Seb 1,1 (0-7803-7282-4/02/810.00 ©2002 IEEE, 186 ‘Balas E. and Padberg MW. (1973) Set prtonng- A survey. la: (Chistes (1) Combinatorial Optimiarion, Wie, New York. ‘Beasley LE, Chu PC. (1998) ponte goth forthe et covering problem. ‘Technical Repo, The Management Scbol, Imperial Coliese Beasley JE, (1996 Agent lgvithm fo the covering problem. European Journal of Operational Research 94, 392408 (Christofides N. and Korman 5. (1978) A computational survey of ‘shod forthe et covering problem, Management Seience 21, 391- Ss. Cava. (1979) A greedy heuristic forthe st-coverng problem Math of Oper Rex.4 Q) 233.235, Croat LF. and Mason JP. (eds) (1991), Industrial Applications of ‘Neural Nevwors Springer Veg Btn, 991 ‘Garin! RS. and Nembauser G.L (1978) Integer programming. John Wiley & Sons, New York Londo Syncy-Tornt Grossman T- and Wool A. (1998) Camputstionl experience with spproxinuton algorths forthe st covering problem, Wering pope, Theoretical Division and CNLS, Lar Alamos Nation [aerator Tats Cand Sokatla G. (1991) Inductive lering supported by imegerprogamming. Computers and Ariel Inelignce 1,57 = 66 Jellies C. (1991) Code Recaption and Set Selection with Newal Networks. Bitthanse, Boston Jchason 'D.A. (1974) Approximation algorithms for combinsoil problems. J. Computer Sytem Se 9, 256278, Kacpeayk J. and nahi C. (1952) Purzy loge with linguistic ‘gant in inductive lean, Ia: LA. Zadeh and. Kaspryk (Es), Psy Lolo te Magen’ of Uncertain, Wey, 85 Kacpryk J, Satta G (1999) Machine leaning fom examples unde eosin ata, Proceedings of Fi Iaterstional Conference in Information Processing apd. Management of Unceringy in Knowiedge Based Sytems IPMU'4 Pans France, Vol2 1047-1051 Kaeprayk J abd Szatle G. (1996) An algorithm fr leming from ‘tones and incomighe examples, nt J of Iiligen St. 1, 56S ‘sk Kacprayk J and Sota G (19978) An improved inductive ening gordi with «preanaljsis od dt, In ZW. Rad and A. Skowron (ei) Foundations of nine Spec (Poceoings 2 Teh ISMIS97 Symposium, Chariot, NC, USA), Springer Veiag, Bo, Kacpayk J. and Sekaula G. (1997) Deriving I-THEN ras for Inelget decision supper via idacive learning, in N.Kasabov ct a. (eds) Progress ia Conseco Based Information Syste (Proceedings of ICONIP97, ANZIIS9T and ANNES? Conference, Dunedin, New Zealand) Spring, Singapore, vol 2, 618-821 KacprzykJ-and Sakata G. (1998) IPI = An Improved Inactive eaming Procedure witha Preprocessing of Data. Proceodings of IDEALS (ong Kong, Springer Verog. Kacprzyk J and Sztatua 6. (1999) Aa inductive leaning sort with preanayis od dt, IlerationalJoural of Kowledpe = Based ineliget Enpnceing Systems, vol. 3, 135-146 Lovase L. (1975) On eauo of optinal integral and fatoesl covers ise Math 13, 382-390 Micaski RS. (1983) A teary and methodology of inductive Teaming. Ia: R. Michal, J Crtonll and TM. Michell (Ed), “Machine Leaming Tioga Press. Naoche JP, Asselin B. (198) Medical dat set propose for the woop on ats anlysis EIASM Workshop, Ape 198, Powell D,Skolick MM. (1993) Using gente slgoritms in caaaceing design optimization with nonlinear consti, W: Fowest Se): rocntings of the Fit nernabonalCenfrence ‘Gente Algoihns: Morgan Krufman, San Mateo, CA, 424 430. Salata G(196) Machine easing from examples under eosin dua, PD. ti SRIPAS Wars Poland.