Dr.

Muhammad Shafiq
Department of Industrial Engineering
University of Engineering and Technology

RANDOM WALK METHOD

In this method, the iteration procedure is conducted based on the
relation:

Xi1 Xi ui
in which
is a predefined scalar step length
u i is a randomly generated unit vector in the i iteration

The detailed procedure for a minimization problem is as follows:

1. Start with X1 , a sufficiently large step length , a minimum
allowable step length , and a minimum number of iterations N
2. Find f1 f X1
3. Set the iteration number i = 1
4. Generate a set of n random numbers r1 , r2 ,..., rn in the range
1,1 and formulate the unit vector u

r1
r
1
2
u

2
2
2
r1 r2 ... rn
rn
It should be noted the unit vector u generated above will be employed
only if R r12 r22 ... rn2 1; otherwise, another set of random
numbers should be generated

Determine the new vector X X1 u and compute f f X

62. If f f : set X X , f f and go back to step 3. If f f ,
1
1
1
1
go to step 7
73. If i N : set i i 1 and go to step 4. Otherwise, go to step 8
51.

4. Compute the new reduced step length

. If the new step

2
length is smaller than or equal to , go to step 9. Otherwise, go
to step 4
95. Stop the procedure: X opt X1 , f opt f1

A revised version of random walk method:

Xi1 Xi i*ui
in which * can be determined by solving:

fi1 f Xi u Min f Xi i ui
*
i i

(using any one-dimensional optimization method)

Notes:
The method can work for discontinuous or nondifferentiable
functions at some points; or when the objective function
possesses several relative minima (maxima)
The method is not so efficient. However, it can be used in the
early stage of optimization to help detect the region where the
global optimum is likely to be found. Then, other more efficient
techniques can be used to find the precise optimum point
likely to be found. Then, other more efficient techniques can be
used to find the precise optimum point

Q #1
Write Dichotomous/Golden section method
procedure (conditions, iterative process,
termination condition etc.) for a minimization
problem
Q #2

STEEPEST DESCENT METHOD CAUCHY METHOD

This is a method developed based on gradient of a function, which is
defined as
f x1
f x

2
f X

f xn

Theorem: The gradient vector represents the direction of steepest

ascent
Proof:
Consider an arbitrary point X in the n-dimensional space and its
neighboring point X dX in which
dx1
dx

dX 2

dxn
The magnitude of vector dX , denoted by ds is given by:
n

dX dX ds dxi
T

i 1

The value of the objective function at X dX is f df , in which the

change in f , i.e., df , can be expressed as:

f
T
df
dxi f X dX
i 1 xi
n

If u denotes the unit vector along the direction dX , we can write

dX uds

And hence, the rate of change of the function w.r.t. the step length ds
is given by
n
df
f dxi
T dX
T

f X
f X u
ds i1 xi ds
ds

or

df
f X u cos
ds
( : the angle between the vectors f X and u )

So,

df
1.
will be maximum when 0 , i.e., when u is along f X (the
ds
df
direction of ascent, due to
0)
ds
df
2.
will be minimum when 1800 (the direction of descent, due
ds
df
to
0)
ds

Procedure of Cauchy method: (for minimization problem)

1. Start with an arbitrary point X1 . Set the iteration number i 1
2. Find the search direction S i : Si f Xi
3. Determine the optimal step length i* in the direction S i
How? By solving
f Xi i*Si Min f Xi i f Xi

and set

Xi1 Xi i*Si Xi i*f Xi

4. Test Xi1 for optimality ( f Xi 0 or use convergence

criteria). If Xi1 is optimum, stop the iteration process.
Otherwise, go to step 5
Set the new iteration number i i 1 and go to step 2

Convergence criteria
1. Change in function value in two consecutive iterations is small
f Xi1 f Xi

f Xi
2. The partial derivatives are small
f
i 1,2,..., n
xi
3. Change in the design vector in two consecutive iterations is small
Xi1 Xi

Example:
Minimize f x1 , x2 x1 x2 2 x12 2 x1 x2 x22

0
starting from X1
0
We have

1 4 x1 2 x2
f X

1 2 x1 2 x2

Iteration 1:

1
f1 f X1
1

1
S1 f1
1

Find the optimal step length 1* by minimizing:

f X1 1 S1 f 1 , 1 12 21

So,

1* 1

1
X2 X1 S
1
*
1 1

1
X 2 is not optimum due to f 2 f X2 0
1

Iteration 2:

1
S 2 f 2
1
Find the optimal step length 2* by minimizing:

f X2 2 S2 f 1 2 ,1 2 522 22 1

So,

2* 15

0.8
X3 X 2 S 2

1.2
*
2

0.2
X3 is not optimum due to f3 f X3
0
0.2

Iteration 3:

0.2
S3 f3

0.2
Find the optimal step length 3* by minimizing:

f X3 3 S3 f 0.8 0.23 ,1.2 0.23 0.0432 0.083 1.20

So,

3* 1.0

1.0
X 4 X3 S

1.4

*
3 3

0.2
X 4 is not optimum due to f 4 f X4
0
0.2

.
The optimum point:

1.0
X

1.5
*

Note:
The method of steepest descent is not really effective in most
problems because the steepest descent direction is a local
property. So, it is good for local search, not global!

 This is a modification of the steepest descent method to help improve the

convergence

Procedure: (for minimization problem)

1. Start with an arbitrary point X1 . Set the iteration number i 1
2. Set the first search direction S1 f1 f X1
3. Find the point X 2 according to the relation

X2 X1 1*S1
where 1* is the optimal step length in the direction S1
Set i 2 and go to the next step
4. Find fi f Xi and set
Si fi

fi
fi1

2
2

Si 1

5. Compute the optimum step length i* in the direction S i , and find

the new point
Xi1 Xi i*Si
6. Test Xi1 for optimality. If Xi1 is optimum, stop the iteration
process. Otherwise, go to step 7
7. Set the new iteration number i i 1 and go to step 4

Example:
Minimize f x1 , x2 x1 x2 2 x12 2 x1 x2 x22

0
starting from X1
0
We have

1 4 x1 2 x2
f X

2
x

2
x

1
2

Iteration 1:

1
f1 f X1
1

1
S1 f1
1

Find the optimal step length 1* by minimizing:

f X1 1 S1 f 1 , 1 12 21

So,

1* 1

1
X2 X1 S
1
*
1 1

1
X 2 is not optimum due to f 2 f X2 0
1

Iteration 2:
The next search direction is

S 2 f 2

f 2

f1

0
S1
2

Find the optimal step length 2* by minimizing:

f X2 2 S2 f 1,1 22 422 22 1

So,

2* 1 4

1
X3 X 2 S 2
1.5
*
2

X3 is optimum due to f3 f X3 0

It is also noted that even if we do not notice that X3 is optimum, we

cannot move from X3 due to:

f 2

2, f3

f3

0
S3 f3
S 0
2 2
f 2
0

NEWTONS METHOD
The Newton-Raphson method presented before for single-variable
optimization can be generalized for multivariable case.
Consider the quadratic approximation of f X at X Xi :
1
T
T
f X f Xi f Xi X Xi X Xi J i X Xi
2
in which J i is the Hessian matrix of f X evaluated at X Xi :

J i 2 f X X X
Setting the partial derivatives of f X equal to 0 (i.e., f X 0 ), we
i

have:
If J i

f Xi J i X Xi 0
is nonsingular, an improved approximation can be obtained as:

Xi1 Xi J i f Xi
1

Example:
Minimize f x1 , x2 x1 x2 2 x12 2 x1 x2 x22

0
starting from X1
0
We have:

1 4 x1 2 x2
f X

1 2 x1 2 x2

Hence,

1
f X1
1
1
1

4 2
1
2
2

1
1
2 2
1
1
2

1
1

X2 X1 J1 f X1 3

0
f X2 : Stop
0

It should be noted that the Newtons method will find the optimum of
a quadratic function in only one iteration. This can be proved as
follows:
Consider the general form of a quadratic function:
1 T
f X X AX BT X C
2
The optimum of f X is the solution of:

X* A1B
f X AX B 0
Using the Newtons method:
1
X2 X1 J1 f X1 X1 A 1 AX1 B A 1B (Q.E.D.)

Notes:
The Newtons method will converge if the initial point is
sufficiently close to the solution.
To prevent divergence or convergent to saddle points/relative
maxima, the method can be modified using
1
Xi1 Xi i*Si Xi i* J i f Xi
in which i* is the minimizing step length in the direction

Si J i f Xi . With this modification, the method will find the

minimum in all cases with lesser number of iterations.
1

Although the Newtons method appears to be the most powerful

minimization method, it is not very useful in practice, especially
for problems involving a complicated objective function with a
large number of variables, due to:
- It is very difficult and sometimes, impossible to determine
the elements of J i
- It requires J i at each step.
1