Chapter 1

Introduction

Before we introduce an emerging field, namely soft computing, let us examine the meaning
of the term: hard computing. This chapter defines this term. Another term called hybrid
computing has also been introduced in this chapter.

1.1 Hard Computing

The term: hard computing was first coined by Prof. L.A. Zadeh of the University of
California, USA, in 1996 [1], although it had been used to solve different problems for a
long time.
Let us see the steps to be followed to solve an engineering problem, which are:

• The variables related to an engineering problem are identified first and then classified
into two groups, namely input or condition variables (also known as antecedents)
and output or action variables (also called consequents).

• The input-output relationships are expressed in terms of mathematical (say differen-

tial) equations.

• Differential equations are then solved analytically or using numerical methods (if
required).

• Control action is decided based on the solutions of these mathematical equations.

The procedure stated above is nothing but the principle of hard computing.

1.1.1 Features of Hard Computing

Human beings have their natural quest for precision and as a result of which, they try to
model a problem using the principle of mathematics. Thus, hard computing could establish
itself long back, as a conventional method for solving engineering problems. It has the
following features:
2 SOFT COMPUTING:Fundamentals and Applications

• As it works based on the principle of mathematics, it may yield precise solutions.

Thus, control actions will be accurate.

• It may be suitable for the problems, which are easy to model mathematically and
whose stability is highly predictable.

1.1.2 Examples of Hard Computing

Hard computing has been used to solve a variety of problems, two such problems are stated
below.

1. Stress analysis of a mechanical member (say beam) subjected to some forms of loading
pattern
If the beam is assumed to have constant cross-section along its length, the problem
of stress analysis can be easily formulated mathematically. On the other hand, if the
beam is assumed to have varying cross-sections along its length, the principle of Finite
Element Method (FEM) can be used to determine the developed stress.

2. Determination of gain values of a Proportional Integral Derivative (PID) controller to

be used to control a process
Let us assume that a PID controller is to be used to control a motor mounted at
a robotic joint. A set of gain values (that is, KP , KI and KD ) can be obtained
mathematically for this controller, which are generally kept constant.

It is important to note that hard computing is applicable, if and only if the problem
can be formulated mathematically. Unfortunately, most of the real-world problems are so
complex and ill-defined that they cannot be modelled mathematically or the mathematical
modelling becomes highly non-linear. Moreover, some sort of imprecisions and uncertainties
are inherent to these problems. Thus, it may not always be possible to solve the complex
real-world problems using the principle of hard computing.

1.2 Soft Computing

The term: soft computing was also introduced by Prof. Zadeh, in 1992 [2]. It is a
collection of some biologically-inspired methodologies, such as Fuzzy Logic (FL), Neural
Network (NN), Genetic Algorithm (GA) and others, and their different combined forms,
namely GA-FL, GA-NN, NN-FL, GA-FL-NN, in which precision is traded for tractability,
robustness, ease of implementation and a low cost solution. Fig. 1.1 shows a schematic
view indicating different members of soft computing family and their possible interactions.
Control algorithms developed based on soft computing may be computationally tractable,
robust and adaptive in nature.
Soft Computing 3

GA

GA−FL GA−NN

GA−FL−NN
FL NN
NN−FL

Figure 1.1: A schematic view showing different members of soft computing and their possible interactions.

1.2.1 Features of Soft Computing

Soft computing is an emerging field, which has the following features:

• It does not require an extensive mathematical formulation of the problem.

• It may not be able to yield so much precise solution as that obtained by the hard
computing technique.

• Different members of this family are able to perform various types of tasks. For exam-
ple, Fuzzy Logic (FL) is a powerful tool for dealing with imprecision and uncertainty,
Neural Network (NN) is a potential tool for learning and adaptation and Genetic Al-
gorithm (GA) is an important tool for search and optimization. Each of these tools
has its inherent merits and demerits (which are discussed in detail, in the subsequent
chapters). In combined techniques (such as GA-FL, GA-NN, NN-FL, GA-FL-NN),
either two or three constituent tools are coupled to get the advantages from both of
them and remove their inherent limitations. Thus, in soft computing, the functions
of the constituent members are complementary in nature and there is no competition
among themselves.

• Algorithm developed based on soft computing is generally found to be adaptive in

nature. Thus, it can accommodate to the changes of a dynamic environment.

1.2.2 Examples of Soft Computing

Soft computing is becoming more and more popular, nowadays. It has been used by various
researchers to solve a variety of problems, two of them are discussed below.

1. Design and development of adaptive motion planners for intelligent and autonomous
robots
An intelligent and autonomous robot should be able to plan its collision-free, time-
optimal paths, while navigating in the presence of some moving obstacles in a dynamic
4 SOFT COMPUTING:Fundamentals and Applications

environment. Both FL- and NN-based adaptive motion planners have been designed
and developed for the intelligent and autonomous robots [3].

2. Soft computing-based expert systems to carry out input-output modeling of engineer-

ing systems or processes
To automate any engineering process or system, it may be required to know its input-
output relationships in both forward and reverse directions. Forward mapping prob-
lems (where the responses are expressed as the functions of input process parameters)
can be solved mathematically. However, it may not be always possible to solve the
problem of reverse mapping using the obtained mathematical equations. The princi-
ple of soft computing has been successfully used to solve the problems of both forward
and reverse mappings for a number of engineering systems or processes [4, 5].

Most of the real-world problems are too complex to model mathematically. In such cases,
hard computing will fail to provide any solution, and we will have to switch over to soft
computing, in which precision is considered to be secondary and we are primarily interested
in acceptable solutions.

1.3 Hybrid Computing

Hybrid computing is a combination of the conventional hard computing and emerging
soft computing. Fig. 1.2 shows the schematic view of a hybrid computing scheme. Both

Hard Hybrid Soft

computing computing
computing

Figure 1.2: A schematic view showing the scheme of hybrid computing.

hard computing as well as soft computing have their inherent advantages and disadvantages.
To get the advantages of both these techniques and eliminate their individual limitations,
one may switch over to hybrid computing to solve a problem more efficiently. Thus, a
part of the problem will be solved using hard computing and the remaining part can be
tackled utilizing soft computing, if it so demands. Moreover, these two techniques may be
complementary to each other, while solving some complex real-world problems. However,
the users of hard computing often do not like soft computing people, due to their inertia
and they fight among themselves.
Summary 5

1.3.1 Examples of Hybrid Computing

Nowadays, hybrid computing has been utilized by various investigators to solve different
engineering problems in a more efficient way. Ovaska et al. [6] provides a survey on various
applications of hybrid computing. Some of the possible applications of hybrid computing
are:

1. Optimal design of machine elements using Finite Element Method (FEM) and soft
computing
Several attempts have been made to design different machine elements using the FEM.
However, the quality of these solutions depends on the selection of the elements, their
size and connectivity. Moreover, material properties of these members may not remain
unchanged during their applications. Thus, there exists fuzziness in both the FEM
analysis as well as material properties. This fuzziness can be modelled using soft
computing, before the FEM analysis is finally carried out to obtain the optimal design
of machine elements [7].

2. PID controller trained by soft computing

A PID controller is one of the most widely-used controllers, nowadays. Its gain values
(KP , KI and KD ) are determined mathematically based on a fixed condition of the
environment. Thus, if there is a sudden change in the environment, the conventional
PID controller may not be able to yield an optimal control action. To overcome this
difficulty, the gain values can be tuned using the principle of soft computing. In a
dynamic environment, appropriate gain values of the PID controller can be determined
on-line, using a previously tuned FL- or NN-based expert system [8].

It is important to mention that the selection of a technique depends on the problem to

be solved. After gathering information of the problem, we generally first try to solve it
using hard computing. However, if it fails for some reasons, we may switch over to soft
computing. Sometimes, we should take the help of hybrid computing, if the problem so
demands.
This book deals with the fundamentals and applications of soft computing and a detailed
discussion on hard computing is beyond its scope.

1.4 Summary
This chapter has been summarized as follows:

1. Hard computing works based on the principle of mathematics and it generally yields
precise solutions. However, we can go for hard computing, if the problem is well-
defined and easy to model mathematically.

2. As most of the real-world problems are complex in nature and difficult to model
mathematically, hard computing may not be suitable to solve such problems. One
6 SOFT COMPUTING:Fundamentals and Applications

may switch over to soft computing to solve the above problems involving inherent
imprecisions and uncertainties.

3. Soft computing is a family composed of different members, namely FL, NN, GA and
their different combinations, in which precision is traded for tractability, robustness
and a low cost solution. It can provide some feasible solutions to the complex real-
world problems.

4. The concept of hybrid computing that combines the hard computing with soft com-
puting, has also been emerged to get advantages from both of them and eliminate
their individual limitations.

5. Selection of a suitable computing scheme for solving a problem depends on its nature.

1.5 Exercise
1. Define briefly the terms: soft computing, hard computing and hybrid computing with
some suitable examples.

2. Justify the use of the term: soft in soft computing.

3. How do you select a suitable scheme of computing (either hard computing or soft
computing or hybrid computing) to solve a particular problem ?
Chapter 2

Optimization and Some Traditional

Methods

This chapter introduces the concept of optimization and discusses some of its traditional
methods (also known as conventional or classical methods). Traditional methods of opti-
mization include both gradient-based and direct search techniques. A detailed discussion on
all these traditional methods of optimization is beyond the scope of this book. An attempt
has been made in this chapter to explain the working principles of a few methods only,
namely Exhaustive Search Method, Random Walk Method and Steepest Descent
Method.

2.1 Introduction to Optimization

Optimization is the process of finding the best one, out of all feasible solutions. To realize
the need for an optimization, let us suppose that we are going to design a suitable gear-
box to be used between a source of power and propeller shaft of a ship. We generally use
the traditional principle of machine design (which includes determination of stress-strain
relationship, checking of dynamic and wear loads, and others) for the above purpose. If
we use this traditional principle only, there is a possibility that the designed gear-box will
have no practical importance, at all. To overcome this difficulty, a designer should use an
optimization tool along with the traditional principle of design. An optimization tool can
make a design more efficient and cost effective.
The concept of optimization is defined mathematically as follows: Let us consider y to be
the function of a single variable x, that is, y = f (x). If the first derivative of this function,
 
that is, f (x) becomes equal to zero, that is, f (x) = 0, at a point x = x∗ , we say that
either the optimum (that is, minimum or maximum) or inflection point exits at that
point. An inflection point (also known as a saddle point) is a point, that is neither a
maximum nor a minimum one. To further investigate the nature of the point, we determine
the first non-zero higher order derivative denoted by n. Two different cases may arise as
stated below.
8 SOFT COMPUTING:Fundamentals and Applications

1. If n is found to be an odd number, x∗ is an inflection point.

2. If n is seen to be an even number, x∗ is a local optimum point. To investigate further
for declaring it either a local minimum or a local maximum point, the following two
conditions are checked:
• If the value of the derivative is found to be positive, x∗ is a local minimum point.
• If the value of the derivative is seen to be negative, x∗ is a local maximum point.

A Numerical Example:
x2
Determine the minimum/maximum/inflection point of the function f (x) = 2 + 125
x for the
positive values of x.

Solution:
The function is:
x2 125
f (x) = 2 + x

Fig. 2.1 shows the plot of this function. Its first order derivative is given by f (x) = x − 125
x2 .

65

60

55
f(x)

50

45

40

35
2 3 4 5 6 7 8 9 10
x
x2 125
Figure 2.1: Plot of the function f (x) = 2
+ x
.

By considering f (x) = 0 and solving it, we get x = 5.0.
Therefore, the minimum/maximum/inflection point exists at x = 5.0.

The second order derivative of this function is given by f (x) = 1 + 250
x3 .

At x = 5.0, f (x) = 3.0 = 0.0.
Therefore, n is found to be equal to 2 and it indicates that the local optimum exists at
x = 5.0.

As f (x) is seen to be equal to 3.0 (a positive number), it can be declared that the local
minimum exists at x = 5.0.
The minimum value of f (x) turns out to be equal to 37.5.
Introduction to Optimization 9

2.1.1 A Practical Example

Let us suppose that we will have to design an optimal wooden pointer generally used by a
speaker while delivering a lecture. The pointer should be as light in weight as possible, after
ensuring the conditions that there is no mechanical breakage and deflection of pointing end
is negligible. This task can simply be thought of selecting the best one, out of all pointers
contained in a bin.
This problem can be posed as an optimization problem as explained below. Fig. 2.2 shows
the pointer in 2-D. The geometry is specified by its diameter at the gripping end d (the
diameter of the pointing end has been assumed to be equal to zero) and length L . Its density
ρ has been assumed to be a constant. The geometrical parameters, namely d and L are

Figure 2.2: A wooden pointer.

known as design or decision variables and the fixed parameter ρ is called pre-assigned
parameter. The design variables have some feasible bounds, which are known as geometric
or side constraints. The aim of this optimization is to search a lighter design, for which its
weight has been expressed with the help of an objective function. Moreover, the pointer
should be able to satisfy a few constraints (in terms of its strength and deflection) during
its usage, which are known as functional or behavior constraints.
The above optimization problem can be expressed mathematically as follows:

πd2 L ρ
Minimize Mass of the pointer M = (2.1)
12
subject to

deflection δ ≤ δallowable ,
strength of the stick s ≥ srequired

and

dmin ≤ d ≤ dmax ,
Lmin ≤ L ≤ Lmax .

Here, equation (2.1) is called the objective function. The constraints related to deflection
and strength of the stick are known as the functional constraints. The lower and upper
limits of d and L are expressed as the geometric or side constraints.
10 SOFT COMPUTING:Fundamentals and Applications

2.1.2 Classification of Optimization Problems

Optimization problems have been classified in a number of ways as discussed below.

1. Depending on the nature of equations involved in the objective function and con-
straints, they are divided into two groups, namely linear and non-linear optimiza-
tion problems, which are defined below.

• Linear optimization problem – An optimization problem is called linear, if

both the objective function as well as all the constraints are found to be linear
functions of design variables.
Example:
Maximize y = f (x1 , x2 ) = 2x1 + x2 (2.2)
subject to
x1 + x2 ≤ 3,
5x1 + 2x2 ≤ 10
and
x1 , x2 ≥ 0.
• Non-linear optimization problem – An optimization problem is known as
non-linear, if either the objective function or any one of the functional constraints
is non-linear function of design variables. Moreover, both the objective function
as well as all functional constraints may be non-linear functions of design vari-
ables in a non-linear optimization problem.
Example:
Maximize y = f (x1 , x2 ) = x21 + x32 (2.3)
subject to
x41 + x22 ≤ 629,
x31 + x32 ≤ 133
and
x1 , x2 ≥ 0.

2. Based on the existence of any functional constraint, optimization problems are clas-
sified into two groups, such as un-constrained (without any functional constraint)
and constrained (when at least one functional constraint is present) optimization
problems, the examples of which are given below.

• Un-constrained optimization problem

Minimize y = f (x1 , x2 ) = (x1 − 5)2 + (x2 − 2)2 (2.4)

where
Introduction to Optimization 11

x1 , x2 ≥ 0.
• Constrained optimization problem

Minimize y = f (x1 , x2 ) = (x1 − 5)2 + (x2 − 2)2 (2.5)

subject to
gi (x1 , x2 ) ≤ ci , i = 1, 2, . . . , n
and
x1 , x2 ≥ 0.

3. Depending on the type of design variables, optimization problems are clustered into
three groups, namely integer programming, real-valued programming and mixed-
integer programming problems, which are briefly discussed below.

• Integer programming problem – An optimization problem is said to be an

integer programming problem, if all the design variables take only integer values.
Example:
Maximize y = f (x1 , x2 ) = 2x1 + x2 (2.6)

subject to
x1 + x2 ≤ 3,
5x1 + 2x2 ≤ 9
and
x1 , x2 ≥ 0,
x1 , x2 are the integers.
• Real-valued programming problem – An optimization problem is known as
a real-valued programming problem, if all the design variables are bound to take
only real values.
Example:
Maximize y = f (x1 , x2 ) = 2x1 + x2 (2.7)

subject to
x1 + x2 ≤ 3.2,
5x1 + 2x2 ≤ 10.0
and
x1 , x2 ≥ 0.0,
x1 , x2 are the real variables.
12 SOFT COMPUTING:Fundamentals and Applications

• Mixed-integer programming problem – It is an optimization problem, in

which some of the variables are integers and the remaining variables take real
values.
Example:
Maximize y = f (x1 , x2 , x3 ) = x1 + 3x2 + 4x3 (2.8)
subject to
2x1 + 5x2 + x3 ≤ 20.5,
x1 + x2 + x3 ≤ 7.5,
3x1 + x2 + 2x3 ≤ 16.4
and
x1 , x2 , x3 ≥ 0,
x1 , x2 are the integers and x3 is a real variable.

4. Depending on the nature of design variables, optimization problems are classified into
two groups, such as static and dynamic, as discussed below.

• Static optimization problem – Let us consider a beam (either cantilever or

simply-supported) of constant cross-section throughout its length, which is sub-
jected to some forms of loading. Our aim is to determine its optimal cross-section
to satisfy a few conditions. It is an example of static optimization problem, as
its cross-section does not vary along its length.
• Dynamic optimization problem – In the above beam, if the cross-section
varies along its length, it becomes a dynamic optimization problem.

2.1.3 Principle of Optimization

To explain the principle of optimization, let us consider a constrained optimization problem
as given below.
Minimize y = f (x1 , x2 ) = (x1 − a)2 + (x2 − b)2 (2.9)
subject to

gi (x1 , x2 ) ≤ ci , i = 1, 2, . . . , n

and

x1 ≥ xmin
1 ,
x2 ≥ xmin
2 ,

where a and b are the constants and n indicates the number of functional constraints.
Fig. 2.3 shows the plots of the constraints. Depending on the nature of these constraints, a
feasible zone has been identified. Any point lying in the feasible zone is a probable candidate
Introduction to Optimization 13

x Side constraint
2

Functional constraints

Feasible zone

A Optimal
solution
Functional constraints

Side constraint
(a,b)
x
(0,0) 1
y
1
y
2
y
3
y
4
y
5
Figure 2.3: A schematic view explaining the principle of optimization.

for the optimal solution. The points residing inside the feasible zone are called free points,
whereas the points lying on boundary of the feasible zone are known as bound points.
Thus, an optimal solution can be either a free point or a bound point contained in the feasible
zone. To determine the optimal solution, the following procedure is adopted. For different
fixed values of y (say y1 , y2 , . . . ), we draw contour plots of the objective function. The point
(say A), at which one of the contour plots just touches the feasible zone, is declared as the
optimal point and the corresponding optimized function value is determined. Thus, the
optimal solution of the constrained optimization problem is denoted by A, whereas that of
the corresponding un-constrained optimization problem is represented by the point (a, b),
the center of the contour plots.

2.1.4 Duality Principle

Let us consider a unimodal function of a single variable denoted by y = f (x), which is to
be minimized. The problem may be stated as follows:

Minimize y = f (x) (2.10)

subject to

x ≥ 0.0.
14 SOFT COMPUTING:Fundamentals and Applications

Fig. 2.4 shows the plot of the function. Let us also assume that the function reaches its
minimum value, corresponding to a value of x = x∗ . The above minimization problem can

y f(x)

x*
x

−f(x)

Figure 2.4: A function plot used to explain the duality principle of optimization.

be posed as a maximum problem as given below.

Maximize − f (x) (2.11)

subject to

x ≥ 0.0.

It is important to mention that through this conversion of minimization problem into a

maximization problem, the value of x at which the optimum occurs will remain the same
as x∗ . It is known as the duality principle of optimization.

2.2 Traditional Methods of Optimization

A huge literature is available on traditional methods of optimization. Various methods are
in use (refer to Fig. 2.5) and the selection of an appropriate one is problem-dependent.
For example, we use a group of methods for solving the linear programming problems,
whereas for non-linear problems, we take the help of some other methods. In a non-linear
optimization problem, the objective function may be composed of either a single variable
or a number of variables. A single variable non-linear optimization problem can be solved
using analytical, numerical methods. The analytical method works based on the principle
of differential calculus. Numerical methods include both the elimination and interpolation
methods. Multi-variable optimization problems may be either constrained or unconstrained
in nature. Unconstrained optimization problems can be tackled utilizing either direct search
or gradient-based methods. On the other hand, there are some direct and indirect methods
to solve the constrained optimization problems.
Traditional Methods of Optimization 15

Conventional Optimization Methods

Linear programming methods Non−linear programming methods Specialized algorithms

1. Graphical method 1. Integer programming
2. Simplex method 2. Geometric programming
3. Dynamic programming

Single variable problems Multi−variable problems

Analytical method Numerical methods

Elimination methods Interpolation methods

1. Unrestricted search 1. Quadratic interpolation method
2. Exhaustive search 2. Cubic interpolation method
3. Dichotomous search 3. Direct root method
4. Fibonacci method
5. Golden section method
Unconstrained optimization algorithms Constrained optimization algorithms

Direct search methods Gradient−based methods Direct methods Indirect method

1. Random search methods 1. Steepest descent method 1. The complex method 1. Penalty function
2. Univariate method 2. Conjugate gradient method 2. Cutting plane method method
3. Pattern search method 3. Quasi−Newton method 3. Methods of feasible directions
4. Rosenbrock’s method 4. Variable metric method

Figure 2.5: Classification of traditional methods of optimization.

16 SOFT COMPUTING:Fundamentals and Applications

A detailed discussion on all these traditional optimization methods is beyond the scope of
this book. Interested readers may refer to the books [9, 10] for the above purpose. Although
the scope is limited, a few traditional methods are explained below, in detail.

2.2.1 Exhaustive Search Method

Let us consider a unimodal function y (that is, it has only one peak) of a single variable x
defined in the range of (xmin , xmax ), where xmin and xmax are the lower and upper limits
of the variable x, respectively. Our aim is to maximize the function y = f (x) in the above
range. The problem may be mathematically stated as follows:

Maximize y = f (x) (2.12)

subject to

xmin ≤ x ≤ xmax .

x min x x x x x x x max
I1 I2 I3 I4 I5 I(n−1)
x

Figure 2.6: A schematic view used to explain the principle of an exhaustive search method.

Fig. 2.6 shows the plot of the function to be optimized. Depending on the desired accuracy
in the solution, let us suppose that the range of the variable, that is, (xmax −xmin ) is divided
into n equal parts. Thus, the number of equi-spaced intermediate points lying in this range
becomes equal to (n − 1). The small change in x, that is, Δx is given by the expression
(xmax −xmin )
n . The following steps are used to locate the maximum value of the function:

• Step 1: We set

x1 = xmin
x2 = x1 + Δx = xI1
x3 = x2 + Δx = xI2
Traditional Methods of Optimization 17

Therefore, the values of x1 , x2 and x3 are in ascending order.

• Step 2: We calculate the function values, that is, f (x1 ), f (x2 ), f (x3 ) and check for
the maximum point.
If f (x1 ) ≤ f (x2 ) ≥ f (x3 ), the maximum point lies in the range of (x1 , x3 ). We
terminate the program,
Else

x1 = x2 (previous)
x2 = x3 (previous)
x3 = x2 (present) + Δx

• Step 3: We check whether x3 exceeds xmax .

If x3 does not exceed xmax , we go to Step 2,
Else we say that the maximum does not lie in the range of (xmin , xmax ).

A Numerical Example:
Maximize y = f (x) = 2x2 − x3 (2.13)
subject to

0.0 ≤ x ≤ 2.0.

Solution:
Fig. 2.7 shows the plot of this function.
Given: xmin = 0.0 and xmax = 2.0.
1.2

0.8

0.6
y

0.4

0.2

0
0 0.5 1 1.5 2
x 1.333

Figure 2.7: Plot of the function y = 2x2 − x3 .

• Step 1: Let us suppose that the range (xmax − xmin ) is divided into n = 10 equal
parts. We calculate the incremental change in x value, that is, Δx as follows:
18 SOFT COMPUTING:Fundamentals and Applications

xmax −xmin 2.0−0.0

Δx = n = 10 = 0.2

We set

x1 = xmin = 0.0
x2 = x1 + Δx = 0.0 + 0.2 = 0.2
x3 = x2 + Δx = 0.2 + 0.2 = 0.4

• Step 2: We determine the function values at x = x1 , x2 and x3 like the following.

f (x1 ) = f (0.0) = 0.0

f (x2 ) = f (0.2) = 0.072
f (x3 ) = f (0.4) = 0.256

Therefore,

f (x1 ) < f (x2 ) < f (x3 ).

Thus, the maximum does not lie in the interval (0.0, 0.4).

• Step 3: We set

x1 = x2 (previous) = 0.2
x2 = x3 (previous) = 0.4
x3 = x2 (present) + Δx = 0.4 + 0.2 = 0.6

• Step 4: We calculate the function values as follows:

f (x1 ) = f (0.2) = 0.072

f (x2 ) = f (0.4) = 0.256
f (x3 ) = f (0.6) = 0.504

Therefore,

f (x1 ) < f (x2 ) < f (x3 ).

Thus, the maximum does not lie in the interval (0.2, 0.6).

• Step 5: We set

x1 = x2 (previous) = 0.4
x2 = x3 (previous) = 0.6
x3 = x2 (present) + Δx = 0.6 + 0.2 = 0.8
Traditional Methods of Optimization 19

• Step 6: We calculate the function values as follows:

f (x1 ) = f (0.4) = 0.256

f (x2 ) = f (0.6) = 0.504
f (x3 ) = f (0.8) = 0.768

Therefore,

f (x1 ) < f (x2 ) < f (x3 ).

Thus, the maximum does not lie in the interval (0.4, 0.8).

• Step 7: We set

x1 = x2 (previous) = 0.6
x2 = x3 (previous) = 0.8
x3 = x2 (present) + Δx = 0.8 + 0.2 = 1.0

• Step 8: We calculate the function values as follows:

f (x1 ) = f (0.6) = 0.504

f (x2 ) = f (0.8) = 0.768
f (x3 ) = f (1.0) = 1.0

Therefore,

f (x1 ) < f (x2 ) < f (x3 ).

Thus, the maximum does not lie in the interval (0.6, 1.0).

• Step 9: We set

x1 = x2 (previous) = 0.8
x2 = x3 (previous) = 1.0
x3 = x2 (present) + Δx = 1.0 + 0.2 = 1.2

• Step 10: We calculate the function values as follows:

f (x1 ) = f (0.8) = 0.768

f (x2 ) = f (1.0) = 1.0
f (x3 ) = f (1.2) = 1.152

Therefore,
20 SOFT COMPUTING:Fundamentals and Applications

f (x1 ) < f (x2 ) < f (x3 ).

Thus, the maximum does not lie in the interval (0.8, 1.2).

• Step 11: We set

x1 = x2 (previous) = 1.0
x2 = x3 (previous) = 1.2
x3 = x2 (present) + Δx = 1.2 + 0.2 = 1.4

• Step 12: We calculate the function values as follows:

f (x1 ) = f (1.0) = 1.0

f (x2 ) = f (1.2) = 1.152
f (x3 ) = f (1.4) = 1.176

Therefore,

f (x1 ) < f (x2 ) < f (x3 ).

Thus, the maximum does not lie in the interval (1.0, 1.4).

• Step 13: We set

x1 = x2 (previous) = 1.2
x2 = x3 (previous) = 1.4
x3 = x2 (present) + Δx = 1.4 + 0.2 = 1.6

• Step 14: We calculate the function values as follows:

f (x1 ) = f (1.2) = 1.152

f (x2 ) = f (1.4) = 1.176
f (x3 ) = f (1.6) = 1.024

Therefore,

f (x1 ) < f (x2 ) > f (x3 ).

Thus, the maximum lies in the interval (1.2, 1.6).

A coarse interval of x, that is, (1.2, 1.6) is obtained using n = 10. However, a finer interval
can be obtained by setting n to a higher value.
Note: The maximum value of this function is found to be equal to 1.185 analytically and
it occurs at x = 1.333.
Traditional Methods of Optimization 21

2.2.2 Random Walk Method

Random walk method is one of the direct search methods, in which the search is carried
out using the objective function value but its derivative information is not utilized [9]. Here,
the present solution (that is, (i + 1)-th) is determined using the previous solution (that is,
i-th), according to the rule given below.

Xi+1 = Xi + λui , (2.14)

where X = (x1 , x2 , . . . , xm )T , m is the number of variables, λ indicates the step length, ui is

a unit random vector determined as ui = 2 2 1 T
1 (r1 , r2 , . . . , rn ) , where (r1 , r2 , . . . , rn )
2)2
(r1 +r2 +...+rn
are the random numbers lying between −1.0 and 1.0, such that n = m and (r12 + r22 + . . . +
1
rn2 ) 2 = R ≤ 1.0. It is to be noted that the number of random numbers, that is, n is kept
equal to the number of variables, that is, m.
The following steps are considered to determine the optimum (say minimum) solution:

• Step 1: We start with an initial solution X1 , created at random. We set initial values
to λ,  (permissible minimum value of λ) and maximum number of iterations to be
tried for improvement of the solution (that is, N ). We determine the function value
f1 = f (X1 ).

• Step 2: A set of n random numbers (lying between −1.0 and 1.0) are generated and
we calculate u1 .

• Step 3: We determine the function value using the expression given below.

f2 = f (X2 ) = f (X1 + λu1 )

• Step 4: If f2 < f1 , then we set X1 = X1 + λu1 , f1 = f2 , and repeat the Steps 2

through 4.
Else repeat Steps 2 through 4, up to the maximum number of iterations N .

• Step 5: If a better point Xi+1 is not obtained after running the program for N
iterations, we reduce λ to 0.5λ.

• Step 6: Is the modified (new) λ <  ?

If no, go to Step 2,
Else we declare X ∗ = X1 , f ∗ = f1 , and terminate the program.

A Numerical Example:
Minimize f (x1 , x2 ) = 4x21 + 3x22 − 6x1 x2 − 4x1 (2.15)
subject to
22 SOFT COMPUTING:Fundamentals and Applications

−10.0 ≤ x1 , x2 ≤ 10.0.

using random walk method.

Assume: λ = 1.2,  = 0.075, N = 100.
Solution:
Fig. 2.8 shows the plot of the function.

f(x_1,x_2)

1400
1200
1000
800
600
400
200
0
-200

10

-10
0
-5 x_2
0 -5

x_1 5
10 -10

Figure 2.8: Plot of the function f (x1 , x2 ) = 4x21 + 3x22 − 6x1 x2 − 4x1 .

Some of the iterations carried out to solve the above problem using the random walk
method are:

• Iteration 1 It consists of the following steps:

– Step 1: Let us consider an initial solution generated at random as given below.

 
0.0
X1 =
0.0
We determine the function value f1 corresponding to X1 like the following.
f1 = f (X1 ) = 0.0.
– Step 2: Let us also assume that the following two random numbers lying between
−1.0 and 1.0 have been generated:
   
r1 −0.5
=
r2 0.2
Traditional Methods of Optimization 23

1
We calculate R = (r12 + r22 ) 2 = 0.5385, which is less than 1.0. So, we consider
the above r values for determination of the unit random vector as given below.
     
1 r1 1 −0.5 −0.9285
u= 1 = 0.5385 =
(r12 +r22 ) 2 r2 0.2 0.3714
– Step 3: We determine the function value f2 as follows:
f2 = f (X2 ) = f (X1 + λu1 ) = f (−1.1142, 0.4457) = 12.9981.
 
−1.1142
As f2 > f1 , cannot be considered as X2 and we go for the next
0.4457
iteration.

• Iteration 2
 
0.0
– Step 1: We have X1 = and f1 = f (X1 ) = 0.0.
0.0
– Step 2: Let us consider another set of random numbers lying between −1.0 and
1.0 as follows:
   
r1 0.001
=
r2 0.1
1
We calculate R = (r12 + r22 ) 2 = 0.1000, which is less than 1.0. So, the above set
of r values can be used to determine the unit random vector like the following.
     
1 r1 1 0.001 0.01
u= 1 = 0.1 =
(r12 +r22 ) 2 r2 0.1 1.0
– Step 3: We determine the function value f2 as follows:
f2 = f (X2 ) = f (X1 + λu1 ) = f (0.012, 1.2) = 4.1862.
 
0.012
As f2 > f1 , will not be considered as X2 and we go for the next
1.2
iteration.

If X2 cannot be obtained after running the program for N = 100 iterations, we set λnew =
0.5 × λ. If λnew > , we repeat the iterations as explained above, else we declare optimal
X ∗ = X1 , f ∗ = f1 .

2.2.3 Steepest Descent Method

Steepest descent method is one of the gradient-based methods, in which the search direc-
tion is dependent on the gradient of the objective function to be optimized [9]. Thus, this
method is not applicable to a discontinuous function. Let us define the term: gradient of a
function, prior to explaining the principle of this method.
24 SOFT COMPUTING:Fundamentals and Applications

Gradient of a function:
Let us consider a continuous function of m variables as given below.
y = f (X) = f (x1 , x2 , . . . , xm ) (2.16)
Gradient of this function is defined as the collection of its partial derivatives with respect
to each of the m variables and it is denoted as follows:
∂f ∂f ∂f T
f = ( , ,..., ) (2.17)
∂x1 ∂x2 ∂xm
It is important to mention that the rate of change of a function is found to be the maximum
along its gradient direction. Thus, the direction of its gradient is nothing but the direction
of steepest ascent. In a steepest descent method, the search is carried out along a direction
opposite to its gradient. Fig. 2.9 shows the search directions of this algorithm at different
iterations.
x
2

Contour plots

x
1

Figure 2.9: A schematic view showing search directions of a steepest descent method.

Principle of the method:

We start with an initial solution X1 , created at random and move along the search direction,
according to the rule given below until it reaches the optimum point.
Xi+1 = Xi + λ∗i Si , (2.18)

where Xi and Xi+1 are the solution vectors at i-th and (i + 1)-th iterations, respectively,
λ∗i represents the optimal step length along the search direction Si = −  fi .
Termination Criteria:
Any one of the following two criteria can be considered as the termination criterion of the
algorithm.
 
 
1. If the rate of change of function value, that is,  f (Xi+1 )−f (Xi )
f (Xi )  becomes less than or
equal to a pre-defined small quantity 1 , the algorithm is terminated.
 
 ∂f 
2. If the derivatives  ∂x j
, j = 1, 2, . . . , m, come out to be less than or equal to a pre-
defined small quantity 2 , the program is terminated.
Traditional Methods of Optimization 25

Advantages:
It is one of the most popular traditional methods of optimization, due to the following
reasons:
1. It has a faster convergence rate.

2. The algorithm is simple and easy to understand and implement.

Limitation of the algorithm:
The search direction of this algorithm is nothing but the steepest descent direction, which
is opposite to the gradient direction. As gradient is a local property of the function (that
means, it may vary from point to point lying on the function), there is a chance of the
solutions of this algorithm for being trapped into the local minima.

A Numerical Example:
Minimize f (x1 , x2 ) = 4x21 + 3x22 − 6x1 x2 − 4x1 (2.19)
subject to
−10.0 ≤ x1 , x2 ≤ 10.0.
Fig. 2.8 shows the plot of the function. A few iterations of the steepest descent method
used to solve this problem are:
• Iteration 1:  
0.0
Let us assume the initial solution created at random as X1 = .
0.0
The function value at X1 is determined as f1 = 0.0.
Gradient of the function is obtained as follows:

 ∂f   
8x1 − 6x2 − 4
f = ∂x1 =
∂f
∂x2
−6x1 + 6x2

Now, the gradient of the objective function is found to be like the following at X1 .
 
−4
f1 = f (X1 ) =
0

Therefore, the search direction at X1 is determined as follows:

 
4
S1 = −  f 1 =
0

To determine X2 , we need to know the value of optimal step length, that is, λ∗1 , for
the estimation of which, we minimize
26 SOFT COMPUTING:Fundamentals and Applications

f (X1 + λ1 × S1 ) = f (4λ1 , 0) = 64λ21 − 16λ1

df
We determine dλ 1
and put it equal to 0.0. The optimal step length is found to be
equal to 8 , that is, λ∗1 = 18 .
1
 
1
We calculate X2 = X1 + λ∗1 S1 = 2
0
The function value at X2 is calculated as f2 = −1.0.
We determine the gradient of the function at X2 , that is,
   
0 0
f2 = f (X2 ) = =
−3 0

Therefore, X2 is not the desired optimum point and we should go for the next iteration.

• Iteration 2:
The search direction at X2 is determined as follows:
 
0
S2 = −  f 2 =
3

To determine X3 , we need to know the value of optimal step length, that is, λ∗2 , for
the estimation of which, we minimize

f (X2 + λ2 × S2 ) = f ( 12 , 3λ2 ) = 27λ22 − 9λ2 − 1

We put df
= 0.0 and get the optimal step length, that is, λ∗2 = 16 .
dλ2  
1
We calculate X3 = X2 + λ∗2 S2 = 2
1
2
The function value at X3 is determined as f3 = − 74 .
The gradient of the function at X3 is determined as follows:
   
−3 0
f3 = f (X3 ) = =
0 0

Therefore, X3 is not the desired optimum point. We should go for the next iteration.

In the similar way, some more iterations are to be carried out, until it reaches the optimum
(minimum) point. Finally, the optimal solution of this function is found to be as follows:

 
2.0
Xopt = , fopt = −4.0.
2.0
Traditional Methods of Optimization 27

2.2.4 Drawbacks of Traditional Optimization Methods

Traditional methods of optimization have the following drawbacks:

1. The final solution of an optimization problem solved using a traditional method de-
pends on the randomly chosen initial solution. If the initial solution lies in the local
basin, the final solution will get stuck at local optimum. Thus, if the user is lucky
enough to choose the initial solution lying in the global basin, the obtained optimal
solution may be global in nature (refer to Fig. 2.10).

f(x)

Local Global
basin
basin

Local minimum

Global minimum

Figure 2.10: A schematic view showing the local and global basins of an objective function.

2. For a discontinuous objective function, the gradient cannot be determined at the point
of discontinuity. Thus, the gradient-based methods cannot be used for such functions.

3. There is a chance of the solutions of a gradient-based optimization method for being

trapped into the local minima.

4. Discrete (integer) variables are difficult to handle using the traditional methods of
optimization, although there exists a separate integer programming method.

5. These methods may not be suitable for parallel computing, in which the total compu-
tation is distributed into a number of computers and they communicate one another
using an MPI (Message Passing Interface) algorithm.

6. A particular traditional method of optimization may not be suitable to solve a variety

of problems.

Thus, it is necessary to develop an optimization method, which is not only efficient but also
robust in nature.
28 SOFT COMPUTING:Fundamentals and Applications

2.3 Summary
The content of this chapter may be summarized as follows:

1. A brief introduction is given to the concept of optimization. Different terms related

to optimization, namely decision variable, objective function, functional constraint,
geometric constraint, and others, have been defined with the help of a practical ex-
ample.

2. Optimization problems have been classified in a number of ways, into different groups,
namely linear optimization problem, non-linear optimization problem, constrained op-
timization problem, un-constrained optimization problem, integer variables problem,
real variables problem, mixed-integer variables problem, static and dynamic optimiza-
tion problems, and others.

3. The principle of optimization has been explained with the help of an example.

4. There exists a number of traditional methods of optimization. A few of them, namely

Exhaustive Search Method, Random Walk Method and Steepest Descent Method,
have been explained in detail with some suitable examples.

5. The drawbacks of the traditional optimization methods are highlighted at the end of
this chapter.

2.4 Exercise
1. Define the following terms related to optimization by taking a suitable example: (i)
decision variables, (ii) objective function, (iii) functional constraints, (iv) geometric
constraints.

2. The presence of geometric constraint(s) is a must for an optimization problem but

that of functional constraint(s) is optional – justify the statement.

3. Explain the principle of optimization with a suitable example.

4. In a constrained optimization, an optimal point is either a free point or a bound point

lying in the feasible zone – justify the statement.

5. Why do the solutions of a steepest descent method get stuck at the local minima ?

6. Discuss briefly the drawbacks of traditional methods of optimization.

7. Determine the minimum/maximum/inflection point of the following functions: (i)

f (x) = x3 ; (ii) f (x) = x4 .
Exercise 29

8. In case of turning operation carried out on a Lathe, cutting parameters (such as

cutting speed v in m/min, feed t in mm/rev and depth of cut d in mm) are to
be selected in such a way that it can produce the smoothest surface after ensuring
a minimum life of the cutting tool T Lmin . Let us assume that surface roughness in
turning S (micro−m) and life of the turning tool T L (min) are given by the following
expressions:

S = 15077v −1.52 t1.004 d0.25 ,

T L = 1.475 × 109 v −4.0 t−4.29 d−4.35

Formulate it as a constrained optimization problem. The cutting parameters are

allowed to vary in the ranges given below.

30.0 ≤ v ≤ 190.0,
0.01 ≤ t ≤ 2.5,
0.5 ≤ d ≤ 4.0.

9. Minimize y = f (x) = x322 + x in the range of 0.0 < x ≤ 10.0. Use (i) analytical
approach based on differential calculus and (ii) exhaustive search method.
(Hints: Let the three values of x, say x1 , x2 and x3 are in ascending order. For a
minimization problem, if f (x1 ) ≥ f (x2 ) ≤ f (x3 ), then the minimum value lies in the
range of (x1 , x3 ).)

10. Minimize f (x1 , x2 ) = 4x21 +x22 −3x1 x

2 +6x1+12x2 in the range of −100.0 ≤ x1 , x2 ≤
0.0
100.0. Take the initial solution X1 = .
0.0
(i) Use Random Walk Method. Assume step length λ = 1.0, permissible minimum
value of λ, that is,  = 0.25 and maximum number of iterations N = 50.
(ii) Use Steepest Descent Method.