Daa
Daa
DIGITAL NOTES
SCHOOL OF ENGINEERING
DESIGN AND ANALYSIS OF ALGORITHMS
UNIT-I
UNIT- II
Disjoint set operations, Union and Find algorithms, Spanning trees, AND/OR graphs,
Connected components, Bi-connected components.
UNIT- III
Greedy method: General method, applications- Job sequencing with deadlines, Knapsack
problem, Minimum cost spanning trees, Single source shortest path problem.
Dynamic Programming: General method, applications- Matrix chained multiplication,
Optimal binary search trees, 0/1 Knapsack problem, All pairs shortest path problem,
Traveling sales person problem, Reliability design.
UNIT- IV
UNIT- V
Branch and Bound: General method, applications- Travelling sales person problem, 0/1
Knapsack problem- LC branch and Bound solution, FIFO branch and Bound solution.
NP-Hard and NP-Complete Problems: Basic concepts, Non deterministic algorithms, NP-
Hard and NP-Complete classes, NP-Hard problems, Cook’s theorem.
TEXT BOOKS:
REFERENCES:
Introduction
For example:
Algorithm:
Add water and milk to the kettle,
Boil it, add tea leaves,
Add sugar, and then serve it in cup.
PROPERTIES OF ALGORITHMS: There are several properties that algorithms
generallyshare. They are useful to keep in mind when algorithms are described. These
properties are:
Example
Describe an algorithm for finding the maximum (largest) value in a finite sequence of
integers.
Solution:
1. Set the temporary maximum equal to the first integer in the sequence. (The
temporarymaximum will be the largest integer examined at any stage of the procedure.)
2. Compare the next integer in the sequence to the temporary maximum, and if it is largerthan
the temporary maximum, set the temporary maximum equal to this integer.
3. Repeat the previous step if there are more integers in the sequence.
4. Stop when there are no integers left in the sequence. The temporary maximum at thispoint
is the largest integer in the sequence.
Algorithms that are definite and effective are also called computational procedures.
1. Natural language like English: When this way is choused care should be taken, we
should ensure that each & every statement is definite.
2. Graphic representation called flowchart: This method will work well when the
algorithmis small& simple.
Example
THE LINEAR SEARCH:The first algorithm that we will present is called the
linearsearch,or sequential search, algorithm. The linear search algorithm begins by
comparing xand a1.When x = a1, the solution is the location of a1, namely, 1. When x ≠ a1,
compare x with a2. Ifx = a2, the solution is the location of a2, namely, 2. When x ≠ a2,
compare x with a3. Continuethis process, comparing x successively with each termof the list
until a match is found, where thesolution is the location of that term, unless no match occurs.
If the entire list has been searchedwithout locating x, the solution is 0.
A second measure is the amount of computer memory required to implement the
algorithmwhen input values are of a specified size.
An analysisof the time required to solve a problem of a particular size involves the time
complexityof the algorithm.
An analysis of the computer memory required involves the space complexityof the
algorithm.
Considerations of space complexity are tied in with the particular data structures used
toimplement the algorithm.
Time Complexity
The limiting behaviour of the complexity as size increases is called the asymptotic time
complexity. It is the asymptotic complexity of an algorithm, which ultimately determines the
size of problems that can be solved by the algorithm.
The time T(P) taken by a program P is the sum of the compile time and the run (or execution)
time. The compile time does not depend on the instance characteristics. Also, we may assume
that a compiled program will be run several times without recompilation. Consequently, we
concern ourselves with just the run time of a program. This run time is denoted by tP(instance
characteristics).
Because many of the factors tP depends on are not known at the timea programis conceived, it
is reasonable to attempt only to estimate tP. If we knew the characteristics of the compiler to
be used, we could proceed to determine the number of additions, subtractions,
multiplications, divisions, compares, loads, stores, and soon, that would be made by the code
for P. So, we could obtain an expression for tP(n) of the form
where n denotes the instance characteristics, and ca, cs, cm, cd, and soon, respectively, denote
the time needed for an addition, subtraction, multiplication, division, and soon, and ADD,
SUB, MUL, DIV, and so on, are functions whose values are the numbers of additions,
subtractions, multiplications, divisions, and soon, that are performed when the code for P is
used on an instance with characteristic n.
Example
Describe the time complexity of Algorithm for finding the maximum element in a finite set of
integers.
Solution:
Example
Describe the time complexity of the linear search algorithm
Solution:
WORST-CASE COMPLEXITY
The type of complexity analysis done in the above Example is aworst-case analysis. By the
worst-case performance of an algorithm, we mean the largest numberof operations needed to
solve the given problem using this algorithm on input of specifiedsize. Worst-case analysis
tells us how many operations an algorithm requires to guarantee thatit will produce a
solution.
AVERAGE-CASE COMPLEXITY
Space Complexity:
The space complexity of a program is the amount of memory it needs to run to completion.
The space need by a program has the following components:
Instruction space: Instruction space is the space needed to store the compiled version of the
program instructions.
Data space: Data space is the space needed to store all constant and variable values. Data
space has two components:
Space needed by constants and simple variables in program.
Space needed by dynamically allocated objects such as arrays and class instances.
Environment stack space: The environment stack is used to save information needed to
resume execution of partially completed functions.
Instruction Space: The amount of instructions space that is needed depends on factors such
as:
The compiler used to complete the program into machine code.
The compiler options in effect at the time of compilation
The target computer.
The space requirement S(P) of any algorithm P may therefore be written as
S(P) = c+SP(Instance characteristics), Where ‘c’ is a constant.
Example
The problem instances for above Algorithm are characterized by n, the number of elements to
be summed. The space needed by n is one word, since it is of type integer. The space needed
by a is the space needed by variables of type array of floating point numbers. This is at least n
words, since a must be large enough to hold the n elements to be summed. So, we obtain
Ssum(n) ≥ (n + 3) (n for a[ ], one each for n, i, and s).
Complexity of Algorithms
The complexity of an algorithm M is the function f(n) which gives the running time and/or
storage space requirement of the algorithm in terms of the size ‘n’ of the input data. Mostly,
the storage space required by an algorithm is simply a multiple of the data size ‘n’.
Complexity shall refer to the running time of the algorithm.
The function f(n), gives the running time of an algorithm, depends not only on the size ‘n’ of
the input data but also on the particular data. The complexity function f(n) for certain cases
are:
1. Best Case: The minimum possible value of f(n) is called the best case.
3. Worst Case: The maximum value of f(n) for any key possible input.
Asymptotic notation
The efficiency of an algorithm depends on the amount of time, storage and other resources
required to execute the algorithm. The efficiency is measured with the help of asymptotic
notations.
An algorithm may not have the same performance for different types of inputs. With the
increase in the input size, the performance will change.
The study of change in performance of the algorithm with the change in the order of the input
size is defined as asymptotic analysis.
Asymptotic notations are the mathematical notations used to describe the running time of an
algorithm when the input tends towards a particular value of a limiting value.
The following notations are commonly use notations in performance analysis and used to
characterize the complexity of an algorithm:
1. Big–OH (O)
2. Big–OMEGA (Ω)
3. Big–THETA (Θ) and
4. Little–OH (o)
Let f, g: R→R. Thefunction f(n) = O(g(n)) (readas “f of n isbig oh of g of n”) iff (if and only
if) there exist positive constants c and no such that f(n) ≤ c* g(n) for all n, n ≥ no.
The constants c and no are called witnesses to the relationshipbetween f and g. Only one pair
of witnesses is needed. (One pairimplies many pairs, since one can always make no or c
larger.)
Big-O notation represents the upper bound of the running time of an algorithm. Thus, it gives
the worst-case complexity of an algorithm.
Example:Prove5n2 + 3n +1 = O(n2)
So from definition
0 ≤ f(n) ≤ c*g(n) for n ≥1
Here c = 9, n0 = 1 and g(n) = n2
Hence 5n2 + 3n +1 = O(n2) proved.
If f is O(g) and h(x) ≥ g(x) for all positive real numbers x then f is O(h).
The O-notation describes upper bounds on how fast functionsgrow.
E.g., f (x) = x2 + 3x is O(x2) but also O(x3), etc.
Note:
The function f(n) = Ω(g(n)) (read as “f of n is big omega of g of n”) iff there exist positive
constants c and no such that f(n) ≥ c* g(n) for all n, n ≥ no.
Big-OMEGA (Ω) notation represents the lower bound of the running time of an algorithm.
Thus, it gives the best case complexity of an algorithm.
Example:Prove5n2 + 3n +1 = Ω(n2)
So from definition
0 ≤ c*g(n) ≤ f(n) for n ≥1
Here c = 5, n0 = 1 and g(n) = n2
Hence 5n2 + 3n +1 = Ω(n2) proved.
Note:
Thefunction f(n) = Θ(g(n)) (readas “f of n isbig thetaof g of n”) iff there exist positive
constantsc1, c2 and nosuchthat c1* g(n) ≤ f(n) ≤ c2* g(n) for all n, n ≥ no.
Big-THETA (Θ) notation encloses the function from above and below. Since it represents the
upper and lower bound of the running time of an algorithm, it is used for analyzing the
average-case complexity of an algorithm.
Example:Prove5n2 + 3n +1 = Θ(n2)
So from definition
0 ≤ c1* g(n) ≤ f(n) ≤ c2* g(n) for n ≥5
Here c1 = 5, c2= 6, n0 =5 and g(n) = n2
Hence 5n2 + 3n +1 = Θ(n2) proved.
Note:
Little–OH (o)
A theoretical measure of the execution of an algorithm, usually the time or memory needed,
given the problem size n, which is usually the number of items. Informally, saying some
equation f(n) = o(g(n)) means f(n) becomes insignificant relative to g(n) as n approaches
infinity.
Formal Definition: f(n) = o(g(n)) means for all c > 0 there exists some n0> 0 such that
0 ≤ f(n) < c*g(n) for all n ≥ n0. The value of n0 must not depend on n, but may depend on c.
Example:Prove3n +2 = o(n2)
Suppose ‘M’ is an algorithm, and suppose ‘n’ is the size of the input data. Clearly the
complexity f(n) of M increases as n increases. It is usually the rate of increase of f(n) we want
to examine. This is usually done by comparing f(n) with some standard functions. The most
common computing times are:
O(1), O(logn), O(n), O(n log n), O(n2), O(n3), O(2n), n! and nn
General method
These subproblems must be solved, and then a method must be found to combine
subsolutions into a solution of the whole.
If the subproblems are still relatively large, then the divide-and-conquer strategy
canpossibly be reapplied.
Often the subproblems resulting from a divide-and conquer design are of the same type
as the original problem.
Now smaller and smaller subproblems of the same kind are generated until eventually
subproblems that are small enough to be solved without splitting are produced.
A control abstraction is a procedure whose flow of control is clear but whose primary
operations are specified by other procedures whose precise meanings are left undefined. The
control abstraction for divide and conquer technique is DAndC(P), where P is the problem to
be solved.
Small(P) is a Boolean-valued function that determines whether the input size is small enough
that the answer can be computed without splitting. If this is so, the function S is invoked.
Otherwise the problem P is divided into smaller subproblems. These subproblems P1,P2,..,Pk
are solved by recursive applications of DAndC. Combine is a function that determines the
solution to P using the solutions to the k subproblems.
If the size of P is n and the sizes of the k subproblems are n1, n2,…,nk, respectively, then the
computing time of DAndC is described by the recurrence relation
Where T(n) is the time for DAndC on any input of sizen and g(n) is the time to compute the
answer directly for small inputs. The function f(n) is the time for dividing P and combining
the solutions to subproblems.
Binary search,
Quick sort,
Merge sort,
Strassen’s matrix multiplication.
BINARY SEARCH
Let ai, 1 ≤ i ≤ n, bea list of elements that are sorted in nondecreasing order. Consider the
problem of determining whether a given element x is present in the list.
If x is present,we areto determinea value j such that aj = x. If x is not in the list,thenj is
to be set to zero.
Let P = (n, ai,…,al,. x) denote an arbitrary instance of this search problem(n is the
number of elements in the list, ai,…,al is the list of elements, and x is the element searched
for).
If P has more than one element, it can be divided (or reduced)into a new subproblems
as follows.
Pickan indexq (in the range [i, l]) and compare x with aq. There are three possibilities:
(1) x =aq: In this case the problem P is immediately solved.
(2) x<aq: In this case x has to be searchedfor only in the sublist ai ,ai+1,…,aq-1.
Therefore, P reduces to (q-i, ai,…,aq-1, x).
(3) x>aq: In this case the sublist to be searched is aq+1,…,al. P reduces to (l-q, aq+1,…,al, x).
After a comparison with aq, the instance remaining to be solved (if any) can be solved
by using this divide-and-conquer scheme again. If q is always chosen such that aq is the
middle element(that is, q = ⌊(n + l)/2⌋), then theresulting searchalgorithmis known as binary
search.
place them in a[1 : 14], and simulate the steps the BinSearch goes through as it searches for
different values of x. Only the variables low, high, and midneed to be traced as we simulate
the algorithm. We try the values for x: 151, -14, and 9 for two successful searches and
unsuccessful searche.
The traces of BinSearh on these three inputs are
x = 151 low high mid
1 14 7
8 14 11
12 14 13
14 14 14
found
a: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]
Elements: -15 -6 0 7 9 23 54 82 101 112 125 131 142 151
Comparisons: 3 4 2 4 3 4 1 4 3 4 2 4 3 4
There are15 possible ways that an unsuccessful search may terminate depending on the value
of x.
If x < a[l], the algorithm requires 3 element comparisons to determine that x is not present.
For all the remaining possibilities, BinSearch requires 4 element comparisons.
The time complexity for a successful search is O(log n) and for an unsuccessful search is
Θ(log n).
In conclusion we are now able to completely describe the computing time of binary search by
giving formulas that describe the best, average, and worst cases:
This algorithm is called mergesort. We assume throughout that the elements are to be
sorted in nondecreasing order.
Each set is individually sorted, and the resulting sorted sequences are merged to
produce a single sorted sequence of n elements.
Thus we have another ideal example of the divide-and-conquer strategy in which the
splitting is into two equal-sized sets and the combining operation is the merging of two sorted
sets into one.
Merge sort
Merging two sorted subarrays using auxiliary storage
Example: Consider the array of eight elements a[l:8] = (7, 3, 2, 16, 24, 4, 11, 9).
If the time for the merging operation is proportional to n, then the computing time for merge
sort is described by the recurrence relation
QUICKSORT
In quicksort, the division into two subarrays is made so that the sorted subarrays do not
need to be merged later.
The quick sort algorithm partitions the original array by rearranging it into two groups.
The first group contains those elements less than some arbitrary chosen value taken from the
set and the second group contains those elements greater than or equal to the chosen value.
The chosen value is known as the pivot element. Once the array has been rearranged in
this way with respect to the pivot, the very same partitioning is recursively applied to each of
the two subsets. When all the subsets have been partitioned and rearranged, the original array
is sorted.
The function Partition( ) makes use of two pointers ‘i’ and ‘j’ which are moved toward
each other in the following fashion:
4. Repeat the steps 1, 2 and 3 till the ‘i’ pointer crosses the ‘j’ pointer. If ‘i’ pointer
crosses ‘j’ pointer, the position for pivot is found and place pivot element in ‘j’ pointer
position.
Sorting by partitioning
Example: Let us consider an array with 8 elements 50, 30, 10, 90, 80, 20, 40, 60 to analyze
quick sort:
Solution:
At first, we select a pivot element which in our case is the first element, 50.
Similarly, j is moved towards the left until it finds a value smaller than the pivot, which we
get at index 6 that is a value of 40. At this point in figure 6, we can see we have got both i and
j values. Finally, we swap the corresponding values and get to a position shown in figure 7.
Then we again go through the i and j index searching and hence get to figure 9, where both
our pointers, i and j stand at 4th and 5th index.
Similar to the previous case, we swap the corresponding values at ith and jth positions. So
after this, we need to move the pointer i towards the right and j towards the left. Since 20<50,
so we need to increase i by 1 towards the right. As 80 > 50, we stop moving i further. As
shown in figure 10 and 11.
Then we start moving j towards the left. As 20<50, j is decreased by 1 and now stands at the
4th index as shown in figure 12. From the above condition, it is clear that i crosses j. Thus we
stop at the point where j<i. Therefore the position of j is the split point.
Hence we swap the pivot and the value at index j points at giving us the array as in figure 13.
After this, we have to apply the same method for the sub-arrays to the left and right of the
pivot element, 50. By this divide and conquer method, finally, we will get our sorted array.
Best case scenario: The best case scenario occurs when the partitions are as evenly balanced
as possible, i.e their sizes on either side of the pivot element are either are equal or are have
size difference of 1 of each other.
Case 1: The case when a size of sublist on either side of pivot becomes equal occurs when the
subarray has an odd number of elements and the pivot is right in the middle after partitioning.
Each partition will have (n–1)/2 elements.
Case 2: The size difference of 1 between the two sublists on either side of pivot happens if
the subarray has an even number, n, of elements. One partition will have n/2 elements with
the other having (n/2)-1.
In either of these cases, each partition will have at most n/2 elements, and the tree
representation of the subproblem sizes will be as below:
The best-case complexity of the quick sort algorithm is O(n log n).
If the partitioning procedure produces two regions of size n/2, quicksort runs much faster.
The recurrence is the T(n) = 2T(n/2) + O(n), has solution T(n) = O(n log n). Thus, this best-
case partitioning produces a much faster algorithm.
Worst case scenario: This happens when we encounter the most unbalanced partitions
possible, then the original call takes n time, the recursive call on n-1 elements will take (n-
1) time, the recursive call on (n-2) elements will take (n-2) time, and so on. The worst case
time complexity of Quick Sort would be O(n2).
Following is recurrence for worst case. T(n) = T(0) + T(n-1) + O(n) which is equivalent to
T(n) = T(n-1) + O(n).
Average case scenario: The average-case running time of quicksort is much closer to the
best case than to the worst case. The key to understanding why this might be true is to
understand how the balance of the partitioning is reflected in the recurrence that describes the
running time.
Suppose, for example, that the partitioning algorithm always produces a 9-to-1 proportional
split, which at first blush seems quite unbalanced. We then obtain the recurrence
T(n) = T(9n/10) + T(n/10) + O(n).
Space Complexity of Quick sort
The space complexity is calculated based on the space used in the recursion stack. The worst
case space used will be O(n). The average case space used will be of the order O(log n). The
worst case space complexity becomes O(n), when the algorithm encounters its worst case
where for getting a sorted list, we need to make n recursive calls.
Let A and B be two n x n matrices. The product matrix C = AB is also an n x n matrix whose
i, jth element is formed by taking the elements in the ith row of A and the jth column of B and
multiplying them to get
for all i and j between 1 and n. To compute C(i, j) using this formula, we need n
multiplications.
The divide-and-conquer strategy suggests another way to compute the product of two n x n
matrices. For simplicity we assume that n is a power of 2, that is, that there exists a
nonnegative integer k such that n = 2k. In case n is not a power of two, then enough rows and
columns of zeros can be added to both A and B so that the resulting dimensions are a power
of two.
Let A and B be two n × n Matrices. Imagine that A and B are each partitioned into four square
𝑛 𝑛
submatrices, each submatrix having dimensions 2 x 2. Then the product AB can be computed
by using the above formula for the product of 2 x 2 matrices: if AB is
To compute AB, we need to perform eight multiplications of n/2 x n/2 matrices and four
additions of n/2 x n/2 matrices. Since two n/2 x n/2 matrices can be added in time cn2 for
some constant c, the overall computing time T(n) of the resulting divide-and-conquer
algorithm is given by the recurrence
This recurrence can be solved in the same way as earlier recurrences to obtain T(n) = O(n3).
Hence no improvement over the conventional method has been made. Since matrix
multiplications are more expensive than matrix additions, we can attempt to reformulate the
equations for Cij so as to have fewer multiplications and possibly more additions.
Volker Strassen has discovered a way to compute the Cij's using only 7 multiplications and 18
additions or subtractions.
His method involves first computing the 7 n/2 x n/2 matrices P, Q, R, S, T, U, and V.
So, concluding that Strassen’s algorithm is asymptotically more efficient than the standard
algorithm. In practice, the overhead of managing the many small matrices does not pay off
until ‘n’ revolves the hundreds.
Example:
UNIT II
Disjoint set operations
Disjoint Sets: If Si and Sj, i ≠ j, are two sets, then there is no element that is in both Si and Sj.
For example, when n = 10, the elements can be partitioned into three disjoint sets,
S1 = {1, 7, 8, 9} S2 = {2, 5, 10}, and S3= {3, 4, 6}.
In this representation, each set is represented as a tree. Notice that for each set we have linked
the nodes from the children to the parent, rather than our usual method of linking from the
parent to the children.
Find(i): Given the element i, find the set containing i. Thus, 4 is in set S3, 1 is in S1, and 10 is
in set S2.
Find(4) → S3
Find(1) → S1
Find(10) → S2
In presenting the union and find algorithms, we ignore the set names and identify sets just by the roots
of the trees representing them.
UNION operation
The function Union(i, j) requires two trees with roots i and j be joined. S1 U S2 is obtained by
making any one of the sets as sub tree of other.
−1 −1 −1 −1 −1 −1
1 2 3 4 5 6
Since the time taken for a union is constant, the n – 1 unions can be processed in time O(n).
Find Operation
The operation of Find(i) now becomes: Determine the root of the tree containing element i.
Algorithm Find(i)
{
while (p[i] ≥ 0) do i := p[i];
return i;
}
Find(1)=−1
Find(3)=1, since its parent is 1. (i.e, root is 1).
Example: Considering
Array representation:
p[i] −1 1 1 2
i 1 2 3 5
Find(5)=2
Find(2)=1
Find(3)=1
The root node represents all the nodes in the tree. Time Complexity of ‘n’ find operations is
O(n2).
To improve the performance of union and find algorithms by avoiding the creation of
degenerate tree. To accomplish this, we use Weighting rule for Union(i, j).
Weighting rule for Union(i, j): If the number of nodes in the tree with root i is less than the
number in the tree with root j, then make j the parent of i; otherwise make i the parent of j.
The trees of above Figure are obtained. As is evident, the height of each tree with m nodes is
⌊log2m⌋ + 1.
Collapsing rule: If j is a node on the path from i to its root and p[i] ≠ root[i], then set p[j] to
root[i].
Find algorithm with collapsing rule
Example: Consider the tree created by WeightedUnion on the sequence of unions of above
Example. Now process the following eight finds: Find(8), Find(8),…,Find(8).
Solution: If SimpleFind is used, each Find(8) requires going up three parent link fields for a
total of 24 moves to process all eight finds. When CollapsingFindis used, the first Find(8)
requires going up three links and then resetting two links. Note that even though only two
parent links need to be reset, CollapsingFind will reset three(the parent of 5 is reset to 1).
Each of the remaining seven finds requires going up only one link field. The total cost is now
only 13 moves(3 going up + 3 resets + 7 remaining finds).
Spanning Trees
A spanning tree is a sub-graph of an undirected connected graph, which includes all the
vertices of the graph with a minimum possible number of edges. (or)
Let G = (V, E) be an undirected connected graph. A sub graph Gʹ = (V, Eʹ) of G is a spanning
tree of G iff Gʹ is a tree.
If all the vertices are connected in a graph, then there exists at least one spanning tree. In a
graph, there may exist more than one spanning tree.
Some of the possible spanning trees that can be created from the above graph are:
Application of Spanning Tree
Spanning tree is basically used to find a minimum path to connect all nodes in a graph.
Common applications of spanning trees are:
Civil Network Planning
Computer Network Routing Protocol
Cluster Analysis
Properties of spanning-tree
A minimum spanning tree is a spanning tree in which the sum of the weight of the edges is as
minimum as possible.
Example:
The initial graph is:
The minimum spanning tree from the above spanning trees is:
There are two basic algorithms for finding minimum-cost spanning trees
Prim’s Algorithm
Kruskal’s Algorithm
AND/OR graphs
The AND-OR GRAPH(or Tree) is useful for representing the solution of problems that
can be solved by decomposing them into a set of smaller problems, all of which must then be
solved.
One AND arc are may point to any number of successor nodes, all of which must be
solved in order for the arc to point to a solution.
Just as in an OR graph, several arcs may emerge from a single node, indicating a variety
of ways in which the original problem might be solved.
Example:
Example:
Example:
Draw an AND/OR graph for the following prepositions:
1. A
2. B
3. C
4. A ˄ B → D
5. A ˄ C → E
6. B ˄ D → F
7. F → G
8. A ˄ E → H
Graph traversals
Given a graph G = (V, E) and a vertex V in V(G) traversing can be done in two ways.
Breadth first search
Depth first search
Breadth first search:
In breadth first search we start at a vertex v and mark it as having been reached (visited). The
vertex v is at this time said to be unexplored. A vertex is said to have been explored by an
algorithm when the algorithm has visited all vertices adjacent from it. All unvisited vertices
adjacent from v are visited next. These are new unexplored vertices. Vertex v has now been
explored. The newly visited vertices haven't been explored and are put onto the end of a list
of unexplored vertices. The first vertex on this list is the next to be explored. Exploration
continues until no unexplored vertex is left. The list of unexplored vertices operates as a
queue and can be represented using any of the standard queue representations.
Pseudocode for breadth first search
Maximum Time complexity and space complexity of G(n, e), nodes are in adjacency list.
T(n, e) = θ(n + e)
S(n, e) = θ(n)
BFS example
We start from vertex 0, the BFS algorithm starts by putting it in the Visited list and putting all
its adjacent vertices in the stack.
Next, we visit the element at the front of queue i.e. 1 and go to its adjacent nodes. Since 0 has
already been visited, we visit 2 instead.
Vertex 2 has an unvisited adjacent vertex in 4, so we add that to the back of the queue and
visit 3, which is at the front of the queue.
Only 4 remains in the queue since the only adjacent node of 3 i.e. 0 is already visited. We
visit it.
Since the queue is empty, we have completed the Breadth First Traversal of the graph.
Depth first search:
A depth first search of a graph differs from a breadth first search in that the exploration of a
vertex v is suspended as soon as a new vertex is reached. At this time the exploration of the
new vertex u begins. When this new vertex has been explored, the exploration of v continues.
The search terminates when all reached vertices have been fully explored.
Maximum Time complexity and space complexity of G(n, e), nodes are in adjacency list.
T(n, e) = θ(n + e)
S(n, e) = θ(n)
DFS Example
We start from vertex 0, the DFS algorithm starts by putting it in the Visited list and putting
all its adjacent vertices in the stack.
Next, we visit the element at the top of stack i.e. 1 and go to its adjacent nodes. Since 0 has
already been visited, we visit 2 instead.
Vertex 2 has an unvisited adjacent vertex in 4, so we add that to the top of the stack and visit
it.
After we visit the last element 3, it doesn't have any unvisited adjacent nodes, so we have
completed the Depth First Traversal of the graph.
After we visit the last element 3, it doesn’t have any unvisited adjacent nodes, so we have
completed the Depth First Traversal of the graph.
Connected components
A set of nodes forms a connected component in an undirected graph if any node from the set
of nodes can reach any other node by traversing edges. The main point here is reachability.
In connected components, all the nodes are always reachable from each other.
Example
Let G = (V, E) be a graph. According to the definition, the vertices in the set V should reach
one another via a path. We’re choosing two random vertices v1 and v6:
v6 is reachable to v1 via: e4 → e7 or e3 → e5 → e7 or e1 → e2 → e6 → e7
v1 is reachable to v6 via: e7 → e4 or e7 → e5 → e3 or e7 → e6 → e2 → e1
Example
If G is a connected undirected graph, then all vertices of G will get visited on the first call to
BFS.
Example
Solution using BFS: Call BFS algorithm once, if |V (G)| = |V (T)|, then G is connected and if
|V (G)| ≠ |V (T)|, then G is disconnected, where T is the BFS tree constructed in the first call
to BFS algorithm. i.e., if number of calls to BFS is greater than one, then G is disconnected
and the number of calls to BFS gives the number of disconnected components.
If G is a connected undirected graph, then all vertices of G will get visited on the first call to
DFS.
Example
Solution using DFS: Call DFS algorithm once, if |V (G)| = |V (T)|, then G is connected and if
|V (G)| ≠ |V (T)|, then G is disconnected, where T is the DFS tree constructed in the first call
to DFS algorithm. i.e., if number of calls to DFS is greater than one, then G is disconnected
and the number of calls to DFS gives the number of disconnected components.
Bi-connected components
A vertex v in a connected graph G is an articulation point if and only if the deletion of vertex
v together with all edges incident to v disconnects the graph into two or more nonempty
components.
Example
An example graph
In the connected graph of above Figure (a) vertex 2 is an articulation point as the deletion of
vertex 2 and edges (1, 2), (2, 3), (2, 5), (2, 7), and (2, 8)leaves behind two disconnected
nonempty components (Figure(b)). Graph G of Figure (a) has only two other articulation
points: vertex 5 and vertex 3. Note that if any of the remaining vertices is deleted from G,
then exactly one component remains.
The graph of above Figure (a) is not biconnected. The graph of below Figure is biconnected.
(c) A biconnected graph
The graph of Figure (c) has only one biconnected component (i.e., the entire graph).
The biconnected components of the graph of Figure (a) are shown in the below Figure.
Example:
Example:
Backtracking
General method
The term backtracking suggests that if the current solution is not suitable, then
backtrack and try other solutions. Thus, recursion is used in this approach.
In the search for fundamental principles of algorithm design, backtracking represents
one of the most general techniques. Many problems which deal with searching for a set of
solutions or which ask for an optimal solution satisfying some constraints can be solved using
the backtracking formulation.
The solution is based on finding one or more vectors that maximize, minimize, or
satisfy a criterion function P(x1,…, xn). Form a solution and check at every step if this has any
chance of success. If the solution at any point seems not promising, ignore it. All solutions
requires a set of constraints divided into two categories: explicit and implicit constraints.
In backtracking problem, the algorithm tries to find a sequence path to the solution which has
some small checkpoints from where the problem can backtrack if no feasible solution is
found for the problem.
Here,
Green is the start point, blue is the intermediate point, red are points with no feasible solution,
dark green is end solution.
Here, when the algorithm propagates to an end to check if it is a solution or not, if it is then
returns the solution otherwise backtracks to the point one step behind it to find track to the
next point to find solution.
Backtracking is the procedure whereby, after determining that a node can lead to
nothing but dead end, we go back (backtrack) to the nodes parent and proceed with the search
on the next child.
A backtracking algorithm need not actually create a tree. Rather, it only needs to keep
track of the values in the current branch being investigated. This is the way we implement
backtracking algorithm. We say that the state space tree exists implicitly in the algorithm
because it is not actually constructed.
State space is the set of paths from root node to other nodes. State space tree is the tree
organization of the solution space. The state space trees are called static trees. This
terminology follows from the observation that the tree organizations are independent of the
problem instance being solved. For some problems it is advantageous to use different tree
organizations for different problem instance. In this case the tree organization is determined
dynamically as the solution space is being searched. Tree organizations that are problem
instance dependent are called dynamic trees.
Terminology:
Solution states are the problem states ‘S’ for which the path from the root node to ‘S’ defines
a tuple in the solution space.
Answer states are those solution states for which the path from root node to s defines a tuple
that is a member of the set of solutions.
Live node is a node that can be further generated.
E-node is a live node whose children are being generated and become a success node.
Dead node is a generated node which cannot be further generated and also does not provide a
feasible solution.
Branch and Bound refers to all state space search methods in which all children of an E-
node are generated before any other live node can become the E-node.
Depth first node generation with bounding functions is called backtracking. State
generation methods in which the E-node remains the E-node until it is dead, lead to branch
and bound methods.
Applications of Backtracking:
N-Queens Problem
Sum of subsets problem
Graph coloring
Hamiltonian cycles.
N-Queens Problem:
Consider a 4 x 4 chessboard. Let there are 4 queens. The objective is placing the 4
queens on 4 x 4 chessboard in such a way that no two queens should be placed in the same
row, same column or diagonal position.
There is one more solution, i.e. the mirror image of figure (h)
1
2
3
4
Following figure shows the complete state space 4-queens problem. But we can use
backtracking method to generate the necessary node and stop if the next node violates the
rule, i.e. if two queens are attacking.
Tree organization of the 4-queens solution space. Nodes are numbered as in depth first
search.
Portion of the tree of above figure that is generated during backtracking
Let us number the rows and columns of the chessboard 1 through 8.
The queens can also be numbered 1through 8. Since each queen must be on a different
row, we can without loss of generality assume queen i is to be placed on row i.
This realization reduces the size of the solution space from 88 tuples to 8! tuples. We
see later how to formulate the second constraint in terms of the xi. Expressed as an 8-tuple,
the solution in above Figure is (4, 6, 8, 2, 7, 1, 3, 5).
Subset sum problem is the problem of finding a subset such that the sum of elements
equals a given number. The backtracking approach generates all permutations in the worst
case but in general, performs better than the recursive approach towards subset sum problem.
For example, consider the list of numbers = [1, 2, 3, 4]. If the target = 7, there are two subsets
that achieve this sum: {3, 4} and {1, 2, 4}. If target = 11, there are no solutions.
A subset A of n positive integers and a value sum(d) is given, find whether or not there
exists any subset of the given set, the sum of whose elements is equal to the given value of
sum.
ALGORITHM:
Example: S = {3, 5, 6, 7} and d = 15, Find the sum of subsets by using backtracking.
Graph coloring
Graph coloring problem involves assigning colors to certain elements of a graph subject to
certain restrictions and constraints. This has found applications in numerous fields in
computer science.
Vertex coloring is the most commonly encountered graph coloring problem. The problem
states that given m colors, determine a way of coloring the vertices of a graph such that no
two adjacent vertices are assigned same color.
For example:
Geographical maps: There can be cases when no two adjacent cities/states can be assigned
same color in the maps of countries or states. In this case, only four colors would be
sufficient to color any map.
Note: It is possible to color all the vertices with the given colors then we have to output the
colored result, otherwise output ‘no solution possible.
In this approach, we color a single vertex and then move to its adjacent (connected)
vertex to color it with different color.
After coloring, we again move to another adjacent vertex that is uncolored and repeat
the process until all vertices of the given graph are colored.
In case, we find a vertex that has all adjacent vertices colored and no color is left to
make it color different, we backtrack and change the color of the last colored vertices and
again proceed further.
If by backtracking, we come back to the same vertex from where we started and all
colors were tried on it, then it means the given number of colors(i.e. m) is insufficient to color
the given graph and we require more colors.
Steps To color graph using the Backtracking Algorithm:
Different colors:
– Confirm whether it is valid to color the current vertex with the current color
(by checking whether any of its adjacent vertices are colored with the same
color).
– If yes then color it and otherwise try a different color.
– Check if all vertices are colored or not.
– If not then move to the next adjacent uncolored vertex.
If no other color is available then backtrack (i.e. un-color last colored vertex).
In other words if a Hamiltonian cycle begins at some vertex v1 ϵ G and the vertices of G
are visited in the order v1, v2, …, vn+1, then the edges (vi, vi+1) are in E, 1 ≤ i ≤ n, and the vi are
distinct except for v1 and vn+1, which are equal.
Hamiltonian circuit is a graph cycle (i.e., closed loop) through a graph that visits each node
exactly once.
The graph G1 in the above figure contains the Hamiltonian cycle 1, 2, 8, 7, 6, 5, 4, 3, 1. The
graph G2 in the above figure contains no Hamiltonian cycle.
There is no known easy way to determine whether a given graph contains a Hamiltonian
cycle. We now look at a backtracking algorithm that finds all the Hamiltonian cycles in a
graph. The graph may be directed or undirected. Only distinct cycles are output.
Generating a next vertex
Solution:
The backtracking approach uses a state-space tree to check if there exists a Hamiltonian cycle
in the graph.
Step 1: Tour is started from vertex 1. There is no path from 5 to 1. So it’s the dead-end state.
Step 2: Backtrack to the node from where the new path can be explored, that is 3 here
Step 3: New path also leads to a dead end so backtrack and explore all possible paths
Step 4: Next path is also leading to a dead-end so keep backtracking until we get some node
that can generate a new path, i.e. vertex 2 here
Step 5: One path leads to Hamiltonian cycle, next leads to a dead end so backtrack and
explore all possible paths at each vertex
Step 6: Total two Hamiltonian cycles are detected in a given graph
Example: Find the Hamiltonian cycle by using the backtracking approach for a given graph.
Solution:
The backtracking approach uses a state-space tree to check if there exists a Hamiltonian cycle
in the graph. Black nodes indicate the Hamiltonian cycle.
UNIT III
GREEDY METHOD
GENERAL METHOD
The greedy method is one of the strategies like Divide and conquer used to solve the
problems. This method is used for solving optimization problems.
Most of these problems have n inputs and require us to obtain a subset that satisfies
some constraints.
We need to find a feasible solution that either maximizes or minimizes a given objective
function.
The greedy method suggests that one can devise an algorithm that works in stages,
considering one input at a time.
At each stage, a decision is made regarding whether a particular input is in an optimal
solution. This is done by considering the inputs in an order determined by some selection
procedure.
If the inclusion of the next input into the partially constructed optimal solution will
result in an infeasible solution, then this input is not added to the partial solution.
Most of these, however, will result in algorithms that generate suboptimal solutions.
This version of the greedy technique is called the subset paradigm.
Greedy algorithm is a problem-solving strategy that makes locally optimal decisions at each
stage in the hopes of achieving a globally optimum solution.
However, we can determine if the algorithm can be used with any problem if the problem has
the following properties:
By following the steps given below, you will be able to formulate a greedy solution for the
given problem statement:
Step 1: In a given problem, find the best substructure or subproblem.
Step 2: Determine what the solution will include (e.g., largest sum, shortest path).
Step 3: Create an iterative process for going over all subproblems and creating an
optimum solution.
Procedure Greedy describes the essential way that a greedy based algorithm will look,
once a particular problem is chosen and the functions select, feasible and union are properly
implemented.
The function select selects an input from ‘a’, removes it and assigns its value to ‘x’.
Feasible is a Boolean valued function, which determines if ‘x’ can be included into the
solution vector.
The function Union combines ‘x’ with solution and updates the objective function.
Job sequencing with deadlines
We are given a set of n jobs. Associated with job i is an integer deadline di ≥ 0 and a
profit pi > 0
For any job i the profit pi is earned iff the job is completed by its deadline
To complete a job, one has to process the job on a machine for one unit of time
A feasible solution for this problem is a subset J of jobs such that each job in this subset
can be completed by its deadline
The value of a feasible solution J is the sum of the profits of the jobs in J, or ∑𝑖∈𝐽 𝑝𝑖
Greedy algorithm for sequencing unit time jobs with deadlines and profits
Knapsack problem
Given a set of items, each with a weight and a value, determine a subset of items to
include in a collection so that the total weight is less than or equal to a given limit and the
total value is as large as possible.
Problem Scenario
A thief is robbing a store and can carry a maximal weight of W into his
knapsack. There are n items available in the store and weight of ith item is wi and its
profit is pi. What items should the thief take?
In this context, the items should be selected in such a way that the thief will
carry those items for which he will gain maximum profit. Hence, the objective of the
thief is to maximize the profit.
Object i has a weight wi and the knapsack has a capacity m. If a fraction xi, 0 ≤ xi ≤ 1, of
object i is placed into the knapsack, then a profit of pixi earned
The objective is to obtain a filling of the knapsack that maximizes the total profit earned
Since the knapsack capacity is m, we require the total weight of all chosen objects to be
at most m.
1. First, we try to fill the knapsack by selecting the objects in some order:
2. Select the object with the maximum profit first (p = 25). So, x1 = 1 and profit earned is 25.
Now, only 2 units of space is left, select the object with next largest profit (p = 24). So, x2 =
2/15
Sort the objects in order of the non-increasing order of the ratio pi / xi. Select the object with
the maximum pi / xi ratio, so, x2 = 1 and profit earned is 24. Now, only 5 units of space is left,
select the object with next largest pi / xi ratio, so x3 =1 / 2 and the profit earned is 7.5.
The objects are to be sorted into non-decreasing order of pi / wi ratio. But if we disregard the
time to initially sort the objects, the algorithm requires only O(n) time.
Spanning Trees
A spanning tree is a sub-graph of an undirected connected graph, which includes all the
vertices of the graph with a minimum possible number of edges. (or)
Let G = (V, E) be an undirected connected graph. A sub graph Gʹ = (V, Eʹ) of G is a spanning
tree of G iff Gʹ is a tree.
If all the vertices are connected in a graph, then there exists at least one spanning tree. In a
graph, there may exist more than one spanning tree.
Some of the possible spanning trees that can be created from the above graph are:
Application of Spanning Tree
Spanning tree is basically used to find a minimum path to connect all nodes in a graph.
Common applications of spanning trees are:
Civil Network Planning
Computer Network Routing Protocol
Cluster Analysis
Properties of spanning-tree
A minimum spanning tree is a spanning tree in which the sum of the weight of the edges is as
minimum as possible.
Example:
The initial graph is:
The minimum spanning tree from the above spanning trees is:
There are two basic algorithms for finding minimum-cost spanning trees
Prim’s Algorithm
Kruskal’s Algorithm
Note:
Both algorithms differ in their methodology, but both eventually end up with the
MST(Minimum Spanning Tree).
Kruskal's algorithm uses edges, and Prim’s algorithm uses vertex connections in
determining the MST.
Prim’s Algorithm:
A greedy method to obtain a minimum-cost spanning tree builds this tree edge by edge
A given graph can have many spanning trees. From these many spanning trees, we have
to select a least cost one. This tree is called as minimal cost spanning tree
Minimal cost spanning tree is a connected undirected graph G in which each edge is
labeled with a number (edge labels may signify lengths, weights other than costs). Minimal
cost spanning tree is a spanning tree for which the sum of the edge labels is as small as
possible. The slight modification of the spanning tree algorithm yields a very simple
algorithm for finding an MST
In the spanning tree algorithm, any vertex not in the tree but connected to it by an edge
can be added. To find a Minimal cost spanning tree, we must be selective - we must always
add a new vertex for which the cost of the new edge is as small as possible
This simple modified algorithm of spanning tree is called prim's algorithm for finding a
Minimal cost spanning tree
Example:
Considering the following graph, find the minimal spanning tree using Prim’s algorithm.
Solution:
Stages in Prim's algorithm
Edge Cost
(1,6) 10
Edge Cost
(5,6) 25
Edge Cost
(4,5) 22
Edge Cost
(3,4) 12
Edge Cost
(2,3) 16
Edge Cost
(2,7) 14
The time required by the Prim’s algorithm is directly proportional to the no. of vertices. If a
graph ‘G’ has ‘n’ vertices then the time required by prim’s algorithm is O(n2).
Kruskal’s Algorithm:
This is a greedy algorithm. A greedy algorithm chooses some local optimum (i.e.
picking an edge with the least weight in a MST).
Considering the following graph, find the minimal spanning tree using Kruskal's algorithm.
Solution:
Stages in Kruskal's algorithm
Edge Cost
(1,6) 10
Edge Cost
(3,4) 12
Edge Cost
(2,7) 14
Edge Cost
(2,3) 16
Edge Cost
(4,5) 22
Edge Cost
(5,6) 25
Kruskal's algorithm
Running time or Analysis:
If the no. of edges in the graph is given by |E| then the time for Kruskal’s algorithm is given
by O(|E|.log |E|).
In the previously studied graphs, the edge labels are called as costs, but here we think
them as lengths. In a labeled graph, the length of the path is defined to be the sum of the
lengths of its edges.
In the single source, all destinations, shortest path problem, we must find a shortest path
from a given source vertex to each of the vertices (called destinations) in the graph to which
there is a path.
Dijkstra’s Algorithm is a graph algorithm for finding the shortest path from a source
node to all other nodes in a graph(single source shortest path). It is a type of greedy
algorithm. It only works on weighted graphs with positive weights.
Dijkstra’s algorithm is similar to prim's algorithm for finding minimal spanning trees.
Dijkstra’s algorithm takes a labeled graph and a pair of vertices P and Q, and finds the
shortest path between then (or one of the shortest paths) if there is more than one. The
principle of optimality is the basis for Dijkstra’s algorithms.
Example:
Considering the following graph, find the single source shortest path using Dijkstra’s
algorithm.
Greedy algorithm to generate shortest paths
It has a time complexity of O(V2) using the adjacency matrix representation of graph. The
time complexity can be reduced to O((V + E)logV) using adjacency list representation of
graph, where E is the number of edges in the graph and V is the number of vertices in the
graph.
UNIT IV
DYNAMIC PROGRAMMING
GENERAL METHOD
Dynamic programming is an algorithm design method that can be used when the
solution to a problem can be viewed as the result of a sequence of decisions (or) Dynamic
programming is a technique that breaks the problems into sub-problems, and saves the result
for future purposes so that we do not need to compute the result again. The subproblems are
optimized to optimize the overall solution is known as optimal substructure property
One way to solve problems for which it is not possible to make a sequence of stepwise
decisions leading to an optimal decision sequence is to try all possible decision sequences
We could enumerate all decision sequences and then pick out the best
The essential difference between the greedy method and dynamic programming is that
in the greedy method only one decision sequence is ever generated
The use of these tabulated values makes it natural to recast the recursive equations into
an iterative algorithm
Top-Down
Start solving the given problem by breaking it down. If you see that the problem has been
solved already, then just return the saved answer. If it has not been solved, solve it and save
the answer. This is usually easy to think of and very intuitive. This is referred to
as Memoization.
Bottom-Up
Analyze the problem and see the order in which the sub-problems are solved and start solving
from the trivial subproblem, up towards the given problem. In this process, it is guaranteed
that the subproblems are solved before solving the problem. This is referred to as Tabulation.
Consider an example of the Fibonacci series
int fib(int n)
{
if (n <= 1)
return n;
return fib(n – 2) + fib(n – 1);
}
int fib(int n)
{
if (n <= 1)
return n;
F[0] = 0; F[1] = 1;
for(int i = 2; i <= n; i++)
{
F[i] = F[i – 2] + F[i – 1];
}
return F[n];
}
MATRIX CHAIN MULTIPLICATION
We can evaluate the above expression using the standard algorithm for multiplying pairs
of matrices as a subroutine once we have parenthesized it to resolve all ambiguities in how the
matrices are multiplied together.
Matrix multiplication is associative, and so all parenthesizations yield the same product.
For example, if the chain of matrices is (A1, A2, A3, A4) then we can fully parenthesize the
product (A1A2A3A4) in five distinct ways:
1: (A1(A2(A3A4))) ,
2: (A1((A2A3)A4)),
3: ((A1A2)(A3A4)),
4: ((A1(A2A3))A4),
5: (((A1A2)A3)A4).
We can multiply two matrices A and B only if they are compatible. The number of
columns of A must equal the number of rows of B. If A is a p x q matrix and B is a q x r
matrix, the resulting matrix C is a p x r matrix.
If we have three matrices (A1, A2, A3) and its cost is (10x100), (100x5), (5x500) respectively.
So we can calculate the cost of scalar multiplication is 10*100*5=5000 if ((A1A2)A3),
10*5*500=25000 if (A1(A2A3)), and so on cost calculation.
Note that in the matrix-chain multiplication problem, we are not actually multiplying
matrices. Our goal is only to determine an order for multiplying matrices that has the lowest
cost.
So the problem is we can perform a many time of cost multiplication and repeatedly the
calculation is performing. So this general method is very time consuming and tedious. So we
can apply dynamic programming for solve this kind of problem.
When we used the Dynamic programming technique we shall follow some steps:
Characterize the structure of an optimal solution
Recursively define the value of an optimal solution
Compute the value of an optimal solution
Construct an optimal solution from computed information
We have matrices of any of order; our goal is find optimal cost multiplication of matrices.
When we solve the this kind of problem using DP step 2 we can get
(or)
Example: We are given the sequence {4, 10, 3, 12, 20, and 7}. The matrices have size 4 x 10,
10 x 3, 3 x 12, 12 x 20, 20 x 7. We need to compute M[i, j], 0 ≤ i, j ≤ 5. We know M[i, i] = 0
for all i.
Let us proceed with working away from the diagonal. We compute the optimal solution for the
product of 2 matrices.
We have to sort out all the combination but the minimum output combination is taken into
consideration.
We initialize the diagonal element with equal i, j value with ‘0’.
After that second diagonal is sorted out and we get all the values corresponded to it
Now the third diagonal will be solved out in the same way.
There are two cases by which we can solve this multiplication: ( M1 x M2) + M3, M1+ (M2 x
M3)
After solving both cases we choose the case in which minimum output is there.
M [1, 3] =264
As Comparing both output 264 is minimum in both cases so we insert 264 in table and ( M1 x
M2) + M3 this combination is chosen for the output making.
There are two cases by which we can solve this multiplication: (M2x M3)+M4, M2+(M3 x
M4)
After solving both cases we choose the case in which minimum output is there.
M [2, 4] = 1320
As Comparing both output 1320 is minimum in both cases so we insert 1320 in table and
M2+(M3 x M4) this combination is chosen for the output making.
There are two cases by which we can solve this multiplication: ( M3 x M4) + M5, M3+ (
M4xM5)
After solving both cases we choose the case in which minimum output is there.
M [3, 5] = 1140
As Comparing both output 1140 is minimum in both cases so we insert 1140 in table and (M3
x M4) + M5 this combination is chosen for the output making.
After solving these cases we choose the case in which minimum output is there
M [1, 4] =1080
As comparing the output of different cases then ‘1080’ is minimum output, so we insert 1080
in the table and (M1 xM2) x (M3 x M4) combination is taken out in output making
There are three cases by which we can solve this multiplication:
1. (M2 x M3 x M4)x M5
2. M2 x( M3 x M4 x M5)
3. (M2 x M3)x ( M4 x M5)
After solving these cases we choose the case in which minimum output is there
M [2, 5] =1350
As comparing the output of different cases then ‘1350’ is minimum output, so we insert 1350
in the table and M2 x( M3 x M4xM5)combination is taken out in output making.
After solving these cases we choose the case in which minimum output is there
M [1, 5] =1344
As comparing the output of different cases then ‘1344’ is minimum output, so we insert 1344
in the table and M1 x M2 x (M3 x M4 x M5)combination is taken out in output making.
Final Output is:
Algorithm of matrix chain multiplication:
Analysis: There are three nested loops. Each loop executes a maximum n times.
1. l, length, O (n) iterations.
2. i, start, O (n) iterations.
3. k, split point, O (n) iterations
In the all pairs shortest path problem, we are to find a shortest path between every pair
of vertices in a directed graph G. That is, for every pair of vertices (i, j), we are to find a
shortest path from i to j as well as one from j to i. These two paths are the same when G is
undirected.
When no edge has a negative length, the all-pairs shortest path problem may be solved
by using Dijkstra’s greedy single source algorithm n times, once with each of the n vertices
as the source vertex.
The all pairs shortest path problem is to determine a matrix A such that A (i, j) is the
length of a shortest path from i to j. The matrix A can be obtained by solving n single-source
problems using the algorithm shortest Paths. Since each application of this procedure requires
O (n2) time, the matrix A can be obtained in O (n3) time.
The dynamic programming solution, called Floyd’s algorithm, runs in O (n3) time.
Floyd’s algorithm works even when the graph has negative length edges (provided there are
no negative length cycles).
The cost of the graph is the length or cost of each edges and cost(i, i) = 0
If there is an edge between i and j then cost of (i, j) = length/cost of the edge from i to j
and if there is no edge then (i, j) = ∞
Need to calculate the shortest path/cost between any two nodes using intermediate
nodes
Example
Road Networking
Network Routing
Flight Reservations
Driving Directions
Optimal binary search trees
Optimal Binary Search Tree extends the concept of Binary search tree. Binary Search
Tree (BST) is a nonlinear data structure which is used in many scientific applications for
reducing the search time. In BST, left child is smaller than root and right child is greater than
root. This arrangement simplifies the search procedure
What is an optimal binary search tree, and how are they different than normal
binary search trees. The cost of searching a node in a tree plays a very big role, and it
is decided by the frequency and a key value of the node. Often, our objective is to
reduce the cost of searching (find minimum cost of BST), and that is done through
an optimal binary search tree.
Example
Let us assume the binary tree has the following keys: 10, 20, 30, 40, 50, 60, 70.
The maximum time required to search a node is equal to the minimum height of the tree,
equal to log.n.
Now we will see how many binary search trees can be made from the given number of keys.
For example: 10, 20, 30 are the keys, and the following are the binary search trees that can be
made out from these keys.
When we use the above formula, then it is found that total 5 number of trees can be created.
The cost required for searching an element depends on the comparisons to be made to search
an element. Now, we will calculate the average cost of time of the above binary search trees.
In the above tree, total number of 3 comparisons can be made. The average number of
comparisons can be made as:
In the above tree, the average number of comparisons that can be made as:
In the above tree, the average number of comparisons that can be made as:
In the above tree, the total number of comparisons can be made as 3. Therefore, the average
number of comparisons that can be made as:
In the above tree, the total number of comparisons can be made as 3. Therefore, the average
number of comparisons that can be made as:
In the third case, the number of comparisons is less because the height of the tree is less, so
it's a balanced binary search tree.
Till now, we read about the height-balanced binary search tree. To find the optimal binary
search tree, we will determine the frequency of searching a key.
Let's assume that frequencies associated with the keys 10, 20, 30 are 3, 2, 5.
The above trees have different frequencies. The tree with the lowest frequency would be
considered the optimal binary search tree. The tree with the frequency 17 is the lowest, so it
would be considered as the optimal binary search tree.
Dynamic Approach
Consider the below table, which contains the keys and frequencies.
When i=0 and j=2, then keys 10 and 20. There are two possible trees that can be made out
from these two keys shown below:
When i=1 and j=3, then keys 20 and 30. There are two possible trees that can be made out
from these two keys shown below:
When i=2 and j=4, we will consider the keys at 3 and 4, i.e., 30 and 40. There are two
possible trees that can be made out from these two keys shown as below:
The following are the trees that can be made out from these two keys shown below:
In the first tree, 10 is the root node, 20 is the right child of node 10, and 30 is the right child
of node 20.
Cost would be: 1*4 + 2*2 + 3*6 = 26.
In the second tree, 10 is the root node, 30 is the right child of node 10, and 20 is the left child
of node 20.
Cost would be: 1*4 + 2*6 + 3*2 = 22.
In the third tree, 20 is the root node, 30 is the right child of node 20, and 10 is the left child of
node 20.
Cost would be: 1*2 + 4*2 + 6*2 = 22.
In the third tree, 30 is the root node, 20 is the left child of node 30, and 10 is the left child of
node 20.
Cost would be: 1*6 + 2*2 + 3*4 = 22.
In the above tree, 30 is the root node, 10 is the left child of node 30 and 20 is the right child
of node 10.
Cost would be: 1*6 + 2*4 + 3*2 = 20
Therefore, the minimum cost is 20 which is the 3rd root. So, c[0,3] is equal to 20.
When i=1 and j=4 then we will consider the keys 20, 30, 40
In this case, we will consider four keys, i.e., 10, 20, 30 and 40. The frequencies of 10, 20, 30
and 40 are 4, 2, 6 and 3 respectively.
w[0, 4] = 4 + 2 + 6 + 3 = 15
In the above cases, we have observed that 26 is the minimum cost; therefore, c[0,4] is equal
to 26.
Let G = (V, E) be a directed graph with edge costs Cij. The variable Cij is defined such that Cij
> 0 for all i and j and Cij = ∞ if <i, j> ∉ E. Let |V| = n and assume n > 1. A tour of G is a
directed simple cycle that includes every vertex in V. The cost of a tour is the sum of the cost
of the edges on the tour. The traveling salesperson problem is to find a tour of minimum cost.
The tour is to be a simple path that starts and ends at vertex 1.
Let g(i, S) be the length of shortest path starting at vertex i, going through all vertices in S,
and terminating at vertex 1. The function g(1, V – {1}) is the length of an optimal salesperson
tour. From the principal of optimality it follows that:
The Equation can be solved for g(1, V – {1}) if we know g(k, V – {1, k}) for all choices of k.
Clearly g(i, Ø) = Ci1, 1 ≤ i ≤ n. Hence, we can use equation 2 to obtain g(i, S) for all S of
size1. Then we can obtain g(i, S) for S with |S|= 2, and soon. When |S| < n - 1, the values of i
and S for which g(i, S) is needed are such that i ≠ 1, 1 ∉ S, and i ∉ S.
Example:
For the following graph find minimum cost tour for the traveling salesperson problem:
Consider the directed graph in Figure 1. The edge lengths are given by matrix c in Figure 2.
Let us start the tour from vertex 1:
Time Complexity
There are at most O(n*2n) subproblems, and each one takes linear time to solve. The total
running time is therefore O(n2*2n).
A real-world application that calculates the route of the Travelling Salesperson Problem
using the current traffic intensity information from Google Maps is prepared.
It can be concluded that on application of TSP algorithm with dynamic programming is able
to produce optimal route tour to serve a customer including the shortest or minimum length
of travel time or optimal travel time.
Here knapsack is like a container or a bag. Suppose we have given some items which have
some weights or profits. We have to put some items in the knapsack (so that the value of
objects in the knapsack is optimized) in such a way total value produces a maximum profit.
For example, the weight of the container is 20 kg. We have to select the items in such a way
that the sum of the weight of items should be either smaller than or equal to the weight of the
container, and the profit should be maximum.
The 0/1 knapsack problem means that the items are either completely or no items are filled in
a knapsack.
For example, we have two items having weights 2kg and 3kg, respectively. If we pick
the 2kg item then we cannot pick 1kg item from the 2kg item (item is not divisible); we have
to pick the 2kg item completely. This is a 0/1 knapsack problem in which either we pick the
item completely or we will not pick that item. The 0/1 knapsack problem is solved by the
dynamic programming.
Example:
Consider the problem having weights and profits are:
Weights: {3, 4, 6, 5}
Profits: {2, 3, 1, 4}
The weight of the knapsack is 8 kg
The number of items is 4
How this problem can be solved by using the Dynamic programming approach?
wi = {3, 4, 5, 6}
pi = {2, 3, 4, 1}
The first row and the first column would be 0 as there is no item for w = 0
When i = 1, w = 1
w1 = 3; since we have only one item in the set having weight 3, but the capacity of the
knapsack is 1. We cannot fill the item of 3kg in the knapsack of capacity 1 kg so add 0 at
M[1][1] shown as below:
When i = 1, w = 2
w1 = 3; since we have only one item in the set having weight 3, but the capacity of the
knapsack is 2. We cannot fill the item of 3kg in the knapsack of capacity 2 kg so add 0 at
M[1][2] shown as below:
When i = 1, w = 3
w1 = 3; Since we have only one item in the set having weight equal to 3, and weight of the
knapsack is also 3; therefore, we can fill the knapsack with an item of weight equal to 3. We
put profit corresponding to the weight 3, i.e., 2 at M[1][3] shown as below:
When i = 1, w = 4
w1 = 3; Since we have only one item in the set having weight equal to 3, and weight of the
knapsack is 4; therefore, we can fill the knapsack with an item of weight equal to 3. We put
profit corresponding to the weight 3, i.e., 2 at M[1][4] shown as below:
When i = 1, w = 5
w1 = 3; Since we have only one item in the set having weight equal to 3, and weight of the
knapsack is 5; therefore, we can fill the knapsack with an item of weight equal to 3. We put
profit corresponding to the weight 3, i.e., 2 at M[1][5] shown as below:
When i = 1, w = 6
w1 = 3; Since we have only one item in the set having weight equal to 3, and weight of the
knapsack is 5; therefore, we can fill the knapsack with an item of weight equal to 3. We put
profit corresponding to the weight 3, i.e., 2 at M[1][6] shown as below:
When i = 1, w = 7
w1 = 3; Since we have only one item in the set having weight equal to 3, and weight of the
knapsack is 5; therefore, we can fill the knapsack with an item of weight equal to 3. We put
profit corresponding to the weight 3, i.e., 2 at M[1][7] shown as below:
When i = 1, w = 8
w1 = 3; Since we have only one item in the set having weight equal to 3, and weight of the
knapsack is 5; therefore, we can fill the knapsack with an item of weight equal to 3. We put
profit corresponding to the weight 3, i.e., 2 at M[1][8] shown as below:
When i = 2, w = 1
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have only one item in the
set having weight equal to 4, and the weight of the knapsack is 1. We cannot put the item of
weight 4 in a knapsack, so we add 0 at M[2][1] shown as below:
When i = 2, w = 2
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have only one item in the
set having weight equal to 4, and the weight of the knapsack is 2. We cannot put the item of
weight 4 in a knapsack, so we add 0 at M[2][2] shown as below:
When i = 2, w = 3
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have two items in the set
having weights 3 and 4, and the weight of the knapsack is 3. We can put the item of weight 3
in a knapsack, so we add 2 at M[2][3] shown as below:
When i = 2, w = 4
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have two items in the set
having weights 3 and 4, and the weight of the knapsack is 4. We can put item of weight 4 in a
knapsack as the profit corresponding to weight 4 is more than the item having weight 3, so
we add 3 at M[2][4] shown as below:
When i = 2, w = 5
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have two items in the set
having weights 3 and 4, and the weight of the knapsack is 5. We can put item of weight 4 in a
knapsack and the profit corresponding to weight is 3, so we add 3 at M[2][5] shown as below:
When i = 2, w = 6
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have two items in the set
having weights 3 and 4, and the weight of the knapsack is 6. We can put item of weight 4 in a
knapsack and the profit corresponding to weight is 3, so we add 3 at M[2][6] shown as below:
When i = 2, W = 7
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have two items in the set
having weights 3 and 4, and the weight of the knapsack is 7. We can put item of weight 4 and
3 in a knapsack and the profits corresponding to weights are 2 and 3; therefore, the total profit
is 5, so we add 5 at M[2][7] shown as below:
When i = 2, w = 8
The weight corresponding to the value 2 is 4, i.e., w2 = 4. Since we have two items in the set
having weights 3 and 4, and the weight of the knapsack is 7. We can put item of weight 4 and
3 in a knapsack and the profits corresponding to weights are 2 and 3; therefore, the total profit
is 5, so we add 5 at M[2][7] shown as below:
When i = 3, w = 1
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
having weights 3, 4, and 5, and the weight of the knapsack is 1. We cannot put neither of the
items in a knapsack, so we add 0 at M[3][1] shown as below:
When i = 3, w = 2
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
having weight 3, 4, and 5, and the weight of the knapsack is 1. We cannot put neither of the
items in a knapsack, so we add 0 at M[3][2] shown as below:
When i = 3, w = 3
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
of weight 3, 4, and 5 respectively and weight of the knapsack is 3. The item with a weight 3
can be put in the knapsack and the profit corresponding to the item is 2, so we add 2 at
M[3][3] shown as below:
When i = 3, w = 4
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
of weight 3, 4, and 5 respectively, and weight of the knapsack is 4. We can keep the item of
either weight 3 or 4; the profit (3) corresponding to the weight 4 is more than the profit
corresponding to the weight 3 so we add 3 at M[3][4] shown as below:
When i = 3, w = 5
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
of weight 3, 4, and 5 respectively, and weight of the knapsack is 5. We can keep the item of
either weight 3, 4 or 5; the profit (3) corresponding to the weight 4 is more than the profits
corresponding to the weight 3 and 5 so we add 3 at M[3][5] shown as below:
When i = 3, w = 6
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
of weight 3, 4, and 5 respectively, and weight of the knapsack is 6. We can keep the item of
either weight 3, 4 or 5; the profit (3) corresponding to the weight 4 is more than the profits
corresponding to the weight 3 and 5 so we add 3 at M[3][6] shown as below:
When i = 3, w = 7
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
of weight 3, 4, and 5 respectively, and weight of the knapsack is 7. In this case, we can keep
both the items of weight 3 and 4, the sum of the profit would be equal to (2 + 3), i.e., 5, so we
add 5 at M[3][7] shown as below:
When i = 3, w = 8
The weight corresponding to the value 3 is 5, i.e., w3 = 5. Since we have three items in the set
of weight 3, 4, and 5 respectively, and the weight of the knapsack is 8. In this case, we can
keep both the items of weight 3 and 4, the sum of the profit would be equal to (2 + 3), i.e., 5,
so we add 5 at M[3][8] shown as below:
Now the value of 'i' gets incremented and becomes 4.
When i = 4, w = 1
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 1. The weight of all
the items is more than the weight of the knapsack, so we cannot add any item in the
knapsack; Therefore, we add 0 at M[4][1] shown as below:
When i = 4, w = 2
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 2. The weight of all
the items is more than the weight of the knapsack, so we cannot add any item in the
knapsack; Therefore, we add 0 at M[4][2] shown as below:
When i = 4, w = 3
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 3. The item with a
weight 3 can be put in the knapsack and the profit corresponding to the weight 4 is 2, so we
will add 2 at M[4][3] shown as below:
When i = 4, w = 4
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 4. The item with a
weight 4 can be put in the knapsack and the profit corresponding to the weight 4 is 3, so we
will add 3 at M[4][4] shown as below:
When i = 4, w = 5
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 5. The item with a
weight 4 can be put in the knapsack and the profit corresponding to the weight 4 is 3, so we
will add 3 at M[4][5] shown as below:
When i = 4, w = 6
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 6. In this case, we can
put the items in the knapsack either of weight 3, 4, 5 or 6 but the profit, i.e., 4 corresponding
to the weight 6 is highest among all the items; therefore, we add 4 at M[4][6] shown as
below:
When i = 4, w = 7
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 7. Here, if we add
two items of weights 3 and 4 then it will produce the maximum profit, i.e., (2 + 3) equals to
5, so we add 5 at M[4][7] shown as below:
When i = 4, W = 8
The weight corresponding to the value 4 is 6, i.e., w4 = 6. Since we have four items in the set
of weights 3, 4, 5, and 6 respectively, and the weight of the knapsack is 8. Here, if we add
two items of weights 3 and 4 then it will produce the maximum profit, i.e., (2 + 3) equals to
5, so we add 5 at M[4][8] shown as below:
As we can observe in the above table that 5 is the maximum profit among all the entries. The pointer
points to the last row and the last column having 5 value. Now we will compare 5 value with the
previous row; if the previous row, i.e., i = 3 contains the same value 5 then the pointer will shift
upwards. Since the previous row contains the value 5 so the pointer will be shifted upwards as shown
in the below table:
Again, we will compare the value 5 from the above row, i.e., i = 2. Since the above row
contains the value 5 so the pointer will again be shifted upwards as shown in the below table:
Again, we will compare the value 5 from the above row, i.e., i = 1. Since the above row does
not contain the same value so we will consider the row i = 1, and the weight corresponding to
the row is 4. Therefore, we have selected the weight 4 and we have rejected the weights 5 and
6 shown below:
x = { 1, 0, 0}
The profit corresponding to the weight is 3. Therefore, the remaining profit is (5 - 3) equals to
2. Now we will compare this value 2 with the row i = 2. Since the row (i = 1) contains the
value 2; therefore, the pointer shifted upwards shown below:
Again we compare the value 2 with a above row, i.e., i = 1. Since the row i = 0 does not
contain the value 2, so row i = 1 will be selected and the weight corresponding to the i = 1 is
3 shown below:
x = {1, 1, 0, 0}
The profit corresponding to the weight is 2. Therefore, the remaining profit is 0. We compare
0 value with the above row. Since the above row contains a 0 value but the profit
corresponding to this row is 0. In this problem, two weights are selected, i.e., 3 and 4 to
maximize the profit.
RELIABILITY DESIGN
Let ri be the reliability of device Di (that is ri is the probability that device i will
function properly) then the reliability of the entire system is Πri.
Even if the individual devices are very reliable (the ri’s are very close to one), the
reliability of the system may not be very good.
For example, if n = 10 and ri = 0.99, 1 < i < 10, then Πri = .904. Hence, it is desirable
to duplicate devices. Multiply copies of the same device type are connected in parallel.
Multiple copies of the same device type are connected in parallel through the use of
switching circuits.
If stage i contains mi copies of device Di. Then the probability that all mi have a
malfunction is
In any practical situation, the stage reliability is a little less than because
the switching circuits themselves are not fully reliable.
Also, failures of copies of the same device may not be fully independent(e.g., if failure
is due to design defect)
An optimal solution m1, m2,…,mn is the result of a sequence of decisions, one decision
for each mi
Design a three stage system with device types D1, D2 and D3. The costs are $30, $15 and
$20 respectively. The Cost of the system is to be no more than $105. The reliability of each
device is 0.9, 0.8 and 0.5 respectively.
Solution:
Since, we can assume each ci > 0, each mi must be in the range 1 ≤ mi ≤ ui. Where:
General method:
Branch and Bound is another method to systematically search a solution space. Just
like backtracking, we will use bounding functions to avoid generating subtrees that
do not contain an answer node. However branch and Bound differs from backtracking
in two important manners:
1. It has a branching function, which can be a depth first search, breadth first
search or based on bounding function.
2. It has a bounding function, which goes far beyond the feasibility test as a
mean to prune efficiently the search tree.
Branch and Bound refers to all state space search methods in which all children of
the E-node are generated before any other live node becomes the E-node
Branch and Bound is the generalization of both graph search strategies, BFS and D-
search.
A BFS like state space search is called as FIFO (First in first out) search
as the list of live nodes in a first in first out list (or queue).
A D search like state space search is called as LIFO (Last in first out)
search as the list of live nodes in a last in first out (or stack).
Definition 1: Live node is a node that has been generated but whose children have
not yet been generated.
Definition 2: E-node is a live node whose children are currently being explored. In
other words, an E-node is a node currently being expanded.
Definition 3: Dead node is a generated node that is not to be expanded or explored
any further. All children of a dead node have already been expanded.
Definition 4: Branch-an-bound refers to all state space search methods in which all
children of an E-node are generated before any other live node can
become the E-node.
Definition 5: The adjective "heuristic", means" related to improving problem solving
performance". As a noun it is also used in regard to "any method or trick
used to improve the efficiency of a problem solving problem". But
imperfect methods are not necessarily heuristic or vice versa. "A heuristic
(heuristic rule, heuristic method) is a rule of thumb, strategy, trick
simplification or any other kind of device which drastically limits search
for solutions in large problem spaces. Heuristics do not guarantee optimal
solutions, they do not guarantee any solution at all. A useful heuristic
offers solutions which are good enough most of thetime.
87
Least Cost (LC) search:
In both LIFO and FIFO Branch and Bound the selection rule for the next E-node in
rigid and blind. The selection rule for the next E-node does not give any preference
to a node that has a very good chance of getting the search to an answer node
quickly.
The search for an answer node can be speeded by using an “intelligent” ranking
function c( ) for live nodes. The next E-node is selected on the basis of this ranking
function. The node x is assigned a rank using:
c( x ) = f(h(x)) + g( x )
h(x) is the cost of reaching x from the root and f(.) is any non-decreasing
function.
A search strategy that uses a cost function c( x ) = f(h(x) + g( x ) to select the next
E-node would always choose for its next E-node a live node with least c(.) is called a
LC–search (Least Cost search)
BFS and D-search are special cases of LC-search. If g( x ) = 0 and f(h(x)) = level of
node x, then an LC search generates nodes by levels. This is eventually the same as
a BFS. If f(h(x)) = 0 and g( x ) > g( y ) whenever y is a child of x, then the search is
essentially a D-search.
We associate a cost c(x) with each node x in the state space tree. It is not possible to
easily compute the function c(x). So we compute a estimate c( x ) of c(x).
Let t be a state space tree and c() a cost function for the nodes in t. If x is a node in
t, then c(x) is the minimum cost of any answer node in the subtree with root x. Thus,
c(t) is the cost of a minimum-cost answer node in t.
A heuristic c(.) is used to estimate c(). This heuristic should be easy to compute and
generally has the property that if x is either an answer node or a leaf node, then
c(x) = c( x ) .
LC-search uses c to find an answer node. The algorithm uses two functions Least() and
Add() to delete and add a live node from or to the list of live nodes, respectively.
Least() finds a live node with least c(). This node is deleted from the list of live nodes
and returned.
88
Add(x) adds the new live node x to the list of live nodes. The list of live nodes be
implemented as a min-heap.
Algorithm LCSearch outputs the path from the answer node it finds to the root node
t. This is easy to do if with each node x that becomes live, we associate a field parent
which gives the parent of node x. When the answer node g is found, the path from g
to t can be determined by following a sequence of parent values starting from the
current E-node (which is the parent of g) and ending at node t.
Listnode = record
{
Listnode * next, *parent; float cost;
}
Algorithm LCSearch(t)
{ //Search t for an answer node
if *t is an answer node then output *t and return;
E := t; //E-node.
initialize the list of live nodes to be empty;
repeat
{
for each child x of E do
{
if x is an answer node then output the path from x to t and return;
Add (x); //x is a new live node.
(x parent) := E; // pointer for path to root
}
if there are no more live nodes then
{
write (“No answer node”);
return;
}
E := Least();
} until (false);
}
The root node is the first, E-node. During the execution of LC search, this list
contains all live nodes except the E-node. Initially this list should be empty.
Examine all the children of the E-node, if one of the children is an answer node, then
the algorithm outputs the path from x to t and terminates. If the child of E is not an
answer node, then it becomes a live node. It is added to the list of live nodes and its
parent field set to E. When all the children of E have been generated, E becomes a
dead node. This happens only if none of E’s children is an answer node. Continue the
search further until no live nodes found. Otherwise, Least(), by definition, correctly
chooses the next E-node and the search continues from here.
LC search terminates only when either an answer node is found or the entire state
space tree has been generated and searched.
Bounding:
A branch and bound method searches a state space tree using any search
mechanism in which all the children of the E-node are generated before another node
becomes the E-node. We assume that each answer node x has a cost c(x) associated
with it and that a minimum-cost answer node is to be found. Three common search
strategies are FIFO, LIFO, and LC. The three search methods differ only in the
selection rule used to obtain the next E-node.
89
A good bounding helps to prune efficiently the tree, leading to a faster exploration of
the solution space.
A cost function c(.) such that c( x ) < c(x) is used to provide lower bounds on
solutions obtainable from any node x. If upper is an upper bound on the cost of a
minimum-cost solution, then all live nodes x with c(x) > c( x ) > upper. The starting
value for upper can be obtained by some heuristic or can be set to .
As long as the initial value for upper is not less than the cost of a minimum-cost
answer node, the above rules to kill live nodes will not result in the killing of a live
node that can reach a minimum-cost answer node. Each time a new answer node is
found, the value of upper can be updated.
To formulate the search for an optimal solution for a least-cost answer node in a
state space tree, it is necessary to define the cost function c(.), such that c(x) is
minimum for all nodes representing an optimal solution. The easiest way to do this is
to use the objective function itself for c(.).
For nodes representing feasible solutions, c(x) is the value of the objective
function for that feasible solution.
For nodes representing partial solutions, c(x) is the cost of the minimum-cost
node in the subtree with root x.
Since, c(x) is generally hard to compute, the branch-and-bound algorithm will use an
estimate c( x ) such that c( x ) < c(x) for all x.
A FIFO branch-and-bound algorithm for the job sequencing problem can begin with
upper = as an upper bound on the cost of a minimum-cost answer node.
Starting with node 1 as the E-node and using the variable tuple size formulation of
Figure 8.4, nodes 2, 3, 4, and 5 are generated. Then u(2) = 19, u(3) = 14, u(4) =
18, and u(5) = 21.
The variable upper is updated to 14 when node 3 is generated. Since c (4) and
c(5) are greater than upper, nodes 4 and 5 get killed. Only nodes 2 and 3 remain
alive.
Node 2 becomes the next E-node. Its children, nodes 6, 7 and 8 are generated.
Then u(6) = 9 and so upper is updated to 9. The cost c(7) = 10 > upper and node 7
gets killed. Node 8 is infeasible and so it is killed.
Next, node 3 becomes the E-node. Nodes 9 and 10 are now generated. Then u(9) =
8 and so upper becomes 8. The cost c(10) = 11 > upper, and this node is killed.
90
The next E-node is node 6. Both its children are infeasible. Node 9’s only child is also
infeasible. The minimum-cost answer node is node 9. It has a cost of 8.
An LC Branch-and-Bound search of the tree of Figure 8.4 will begin with upper =
and node 1 as the first E-node.
Node 2 is the next E-node as c(2) = 0 and c(3) = 5. Nodes 6, 7 and 8 are generated
and upper is updated to 9 when node 6 is generated. So, node 7 is killed as c(7) = 10
> upper. Node 8 is infeasible and so killed. The only live nodes now are nodes 3 and
6.
Node 6 is the next E-node as c(6) = 0 < c(3) . Both its children are infeasible.
Node 3 becomes the next E-node. When node 9 is generated, upper is updated to 8
as u(9) = 8. So, node 10 with c(10) = 11 is killed on generation.
Node 9 becomes the next E-node. Its only child is infeasible. No live nodes remain.
The search terminates with node 9 representing the minimum-cost answer node.
2 3
The path = 1 3 9 = 5 + 3 = 8
By using dynamic programming algorithm we can solve the problem with time
complexity of O(n22n) for worst case. This can be solved by branch and bound
technique using efficient bounding function. The time complexity of traveling sale
person problem using LC branch and bound is O(n22n) which shows that there is no
change or reduction of complexity than previous method.
We start at a particular node and visit all nodes exactly once and come back to initial
node with minimum cost.
Let G = (V, E) is a connected graph. Let C(i, J) be the cost of edge <i, j>. cij = if
<i, j> E and let |V| = n, the number of vertices. Every tour starts at vertex 1 and
ends at the same vertex. So, the solution space is given by S = {1, , 1 | is a
91
permutation of (2, 3, . . . , n)} and |S| = (n – 1)!. The size of S can be reduced by
restricting S so that (1, i 1, i2, . . . . in-1, 1) S iff <ij, ij+1> E, 0 < j < n - 1 and
i0 = in =1.
1. Reduce the given cost matrix. A matrix is reduced if every row and column is
reduced. A row (column) is said to be reduced if it contain at least one zero
and all-remaining entries are non-negative. This can be done as follows:
a) Row reduction: Take the minimum element from first row, subtract it
from all elements of first row, next take minimum element from the
second row and subtract it from second row. Similarly apply the same
procedure for all rows.
b) Find the sum of elements, which were subtracted from rows.
c) Apply column reductions for the matrix obtained after row reduction.
e) Obtain the cumulative sum of row wise reduction and column wise
reduction.
2. Calculate the reduced cost matrix for every node R. Let A is the reduced cost
matrix for node R. Let S be a child of R such that the tree edge (R, S)
corresponds to including edge <i, j> in the tour. If S is not a leaf node, then
the reduced cost matrix for S may be obtained as follows:
b) Set A (j, 1) to .
c) Reduce all rows and columns in the resulting matrix except for rows
and column containing only . Let r is the total amount subtracted to
reduce the matrix.
92
Example:
Find the LC branch and bound solution for the traveling sale person problem whose
cost matrix is as follows:
20 30 10 11
15 16 4 2
The cost matrix is 3 5 2 4
18 3
19 6
16 4
7 16
Step 1: Find the reduced cost matrix.
10 20 0 1
13
14 2 0
The resulting row wise reduced cost matrix = 1 3 0 0
3 15 0
16
12 0 3 12
Deduct 1 (which is the minimum) from all values in the 1st column.
Deduct 3 (which is the minimum) from all values in the 3rd column.
10 17 0 1
12
11 2 0
The resulting column wise reduced cost matrix (A) = 0 3 0 2
3 12 0
15
11 0 0 12
Cumulative reduced sum = row wise reduction + column wise reduction sum.
= 21 + 4 = 25.
This is the cost of a root i.e., node 1, because this is the initially reduced cost matrix.
93
Starting from node 1, we can next visit 2, 3, 4 and 5 vertices. So, consider to explore
the paths (1, 2), (1, 3), (1, 4) and (1, 5).
i=2 i = 4 i= 5
i=3
2 3 4 5
Step 2:
Change all entries of row 1 and column 2 of A to and also set A(2, 1) to .
11 2
0
0 0 2
12 0
15
11 0 12
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
11 2
0
Then the resultant matrix is 0 0 2
12 0
15
11 0 12
Row reduction sum = 0 + 0 + 0 + 0 = 0
Column reduction sum = 0 + 0 + 0 + 0 = 0
Cumulative reduction (r) = 0 + 0 = 0
Change all entries of row 1 and column 3 of A to and also set A(3, 1) to .
94
12 2 0
3 0 2
0
15 3
11 0 12
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
1 2 0
Then the resultant matrix is 3 0 2
3 0
4
0 0 12
Change all entries of row 1 and column 4 of A to and also set A(4, 1) to .
12
11 0
0 3 2
3 12 0
11 0 0
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
12
11 0
Then the resultant matrix is 0 3 2
3 12 0
11 0 0
95
Therefore, as cS cR A 1, 4 r
c S = 25 + 0 + 0 = 25
Change all entries of row 1 and column 5 of A to and also set A(5, 1) to .
12
11 2
0 3 0
12
15 3
0 0 12
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
10 9 0
Then the resultant matrix is 0 3 0
12 0 9
0 0 12
i=2 i = 4 i= 5
i=3
35 2 53 3 25 4 31 5
i = 2 i= 5
i=3
6 7 8
The cost of the paths between (1, 2) = 35, (1, 3) = 53, (1, 4) = 25 and (1, 5) = 31.
The cost of the path between (1, 4) is minimum. Hence the matrix obtained for path
(1, 4) is considered as reduced cost matrix.
96
12
11 0
A = 0 3 2
3 12 0
11 0 0
The new possible paths are (4, 2), (4, 3) and (4, 5).
Change all entries of row 4 and column 2 of A to and also set A(2, 1) to .
11 0
0 2
11 0
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
11 0
Then the resultant matrix is 0 2
11 0
Change all entries of row 4 and column 3 of A to and also set A(3, 1) to .
12
0
3 2
11 0
97
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
1 0
Then the resultant matrix is 1 0
0 0
Change all entries of row 4 and column 5 of A to and also set A(5, 1) to .
12
11
0 3
0 0
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
1 0
Then the resultant matrix is 0 3
0 0
98
The tree organization up to this point is as follows:
U =
1 L = 25
i=2 i = 4 i= 5
i=3
35 2 53 3 25 4 31 5
i = 2 i= 5
i=3
28 6 7 8
36
50
i=3
i=5
9 10
The cost of the paths between (4, 2) = 28, (4, 3) = 50 and (4, 5) = 36. The cost of
the path between (4, 2) is minimum. Hence the matrix obtained for path (4, 2) is
considered as reduced cost matrix.
11 0
A = 0 2
11 0
Change all entries of row 2 and column 3 of A to and also set A(3, 1) to .
2
11
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
99
Then the resultant matrix is 0
0
Change all entries of row 2 and column 5 of A to and also set A(5, 1) to .
0
0
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
Then the resultant matrix is 0
0
Row reduction sum = 0
Column reduction sum = 0
Cumulative reduction (r) = 0 + 0 = 0
100
U =
1 L = 25
i=2 i = 4 i= 5
i=3
35 2 53 3 25 4 31 5
i = 2 i= 5
i=3
28 6 7 8
36
50
i=3
i=5
52 9 10 28
i= 3
11
The cost of the paths between (2, 3) = 52 and (2, 5) = 28. The cost of the path
between (2, 5) is minimum. Hence the matrix obtained for path (2, 5) is considered
as reduced cost matrix.
A = 0
0
Change all entries of row 5 and column 3 of A to and also set A(3, 1) to .
Apply row and column reduction for the rows and columns whose rows and
columns are not completely .
Then the resultant matrix is
Row reduction sum = 0
Column reduction sum = 0
Cumulative reduction (r) = 0 + 0 = 0
101
U =
1 L = 25
i=2 i = 4 i= 5
i=3
35 2 53 3 25 4 31 5
i = 2 i= 5
i=3
28 6 7 8
36
50
i=3
i=5
52 9 10 28
i=3
11 28
1 4 2 5 3 1
Consider the instance: M = 15, n = 4, (P1, P2, P3, P4) = (10, 10, 12, 18) and
(w1, w2, w3, w4) = ( 2, 4, 6, 9).
0/1 knapsack problem can be solved by using branch and bound technique. In this
problem we will calculate lower bound and upper bound for each node.
Profit = P1 + P2 + P3 = 10 + 10 + 12
So, Upper bound = 32
To calculate lower bound we can place w4 in knapsack since fractions are allowed in
calculation of lower bound.
3
Lower bound = 10 + 10 + 12 + ( X 18) = 32 + 6 = 38
9
Knapsack problem is maximization problem but branch and bound technique is
applicable for only minimization problems. In order to convert maximization problem
into minimization problem we have to take negative sign for upper bound and lower
bound.
We choose the path, which has minimum difference of upper bound and lower bound.
If the difference is equal then we choose the path by comparing upper bounds and
we discard node with maximum upper bound.
102
U = - 32
1 L = -38
x1 = 1 x1 = 0
U = - 32 U = - 22
2 3
L = -38 L = -32
Now we will calculate upper bound and lower bound for nodes 2, 3.
For node 2, x1= 1, means we should place first item in the knapsack.
3
L = 10 + 10 + 12 + x 18 = 32 + 6 = 38, make it as -38
9
For node 3, x1 = 0, means we should not place first item in the knapsack.
Next, we will calculate difference of upper bound and lower bound for nodes 2, 3
U = - 32
1 L = -38
x1 = 1 x1 = 0
U = - 32 U = - 22
2 3
L = - 38 L = -32
x2 = 1 x2 = 0
U = - 32 U = - 22
4 5
L = - 38 L = -36
Now we will calculate lower bound and upper bound of node 4 and 5. Calculate
difference of lower and upper bound of nodes 4 and 5.
103
U = - 32
1 L = -38
x1 = 1 x1 = 0
U = - 32 U = - 22
2 3
L = -38 L = -32
x2 = 1 x2 = 0
U = -32 U = - 22
4 5
L = -38 L = -36
x3 = 1 x3 = 0
U = -32 U = -38
6 7
L = -38 L = -38
Now we will calculate lower bound and upper bound of node 8 and 9. Calculate
difference of lower and upper bound of nodes 8 and 9.
U = - 32
1 L = -38
x1 = 1 x1 = 0
U = - 32 U = - 22
2 3
L = -38 L = -32
x2 = 1 x2 = 0
U = - 32 U = - 22
4 5
L = -38 L = -36
x3 = 1 x3 = 0
U = - 32 U = - 38
6 7
L = -38 L = -38
x4 = 1 x4 = 0
U = - 38 U = - 20
8 9
L = -38 L = -20
Now we will calculate lower bound and upper bound of node 4 and 5. Calculate
difference of lower and upper bound of nodes 4 and 5.
Here the difference is same, so compare upper bounds of nodes 8 and 9. Discard the
node, which has maximum upper bound. Choose node 8, discard node 9 since, it has
maximum upper bound.
X1 = 1
X2 = 1
X3 = 0
104
X4 = 1
The solution for 0/1 Knapsack problem is (x1, x2, x3, x4) = (1, 1, 0, 1)
Pi xi = 10 x 1 + 10 x 1 + 12 x 0 + 18 x 1
= 10 + 10 + 18 = 38.
Portion of state space tree using FIFO Branch and Bound for above problem:
As follows:
105
NP-Hard and NP-Complete problems
Deterministic and non-deterministic algorithms
Deterministic: The algorithm in which every operation is uniquely defined is called
deterministic algorithm.
Non-Deterministic: The algorithm in which the operations are not uniquely defined but
are limited to specific set of possibilities for every operation, such an algorithm is called
non-deterministic algorithm.
The non-deterministic algorithms use the following functions:
1. Choice: Arbitrarily chooses one of the element from given set.
2. Failure: Indicates an unsuccessful completion
3. Success: Indicates a successful completion
A non-deterministic algorithm terminates unsuccessfully if and only if there exists no
set of choices leading to a success signal. Whenever, there is a set of choices that leads to
a successful completion, then one such set of choices is selected and the algorithm
terminates successfully.
In case the successful completion is not possible, then the complexity is O(1). In case of
successful signal completion then the time required is the minimum number of steps
needed to reach a successful completion of O(n) where n is the number of inputs.
The problems that are solved in polynomial time are called tractable problems and the
problems that require super polynomial time are called non-tractable problems. All
deterministic polynomial time algorithms are tractable and the non-deterministic
polynomials are intractable.
106
Satisfiability Problem:
The satisfiability is a boolean formula that can be constructed using the
following literals and operations.
1. A literal is either a variable or its negation of the variable.
2. The literals are connected with operators ˅, ˄͢, ⇒ , ⇔
3. Parenthesis
Example:
107
Reducability:
A problem Q1 can be reduced to Q2 if any instance of Q1 can be easily rephrased as an
instance of Q2. If the solution to the problem Q2 provides a solution to the problem Q1,
then these are said to be reducable problems.
Let L1 and L2 are the two problems. L1 is reduced to L2 iff there is a way to solve L1 by
a deterministic polynomial time algorithm using a deterministic algorithm that solves L2
in polynomial time and is denoted by L1α L2.
If we have a polynomial time algorithm for L2 then we can solve L1 in polynomial time.
Two problems L1 and L2 are said to be polynomially equivalent iff L1α L2 and L2 α L1.
Example: Let P1 be the problem of selection and P2 be the problem of sorting. Let the
input have n numbers. If the numbers are sorted in array A[ ] the ith smallest element of
the input can be obtained as A[i]. Thus P1 reduces to P2 in O(1) time.
Decision Problem:
Any problem for which the answer is either yes or no is called decision problem. The
algorithm for decision problem is called decision algorithm.
Example: Max clique problem, sum of subsets problem.
Optimization Problem: Any problem that involves the identification of an optimal value
(maximum or minimum) is called optimization problem.
Example: Knapsack problem, travelling salesperson problem.
In decision problem, the output statement is implicit and no explicit statements are
permitted.
The output from a decision problem is uniquely defined by the input parameters and
algorithm specification.
Many optimization problems can be reduced by decision problems with the property that
a decision problem can be solved in polynomial time iff the corresponding optimization
problem can be solved in polynomial time. If the decision problem cannot be solved in
polynomial time then the optimization problem cannot be solved in polynomial time.
108
Class P:
P: the class of decision problems that are solvable in O(p(n)) time, where p(n) is a
polynomial of problem’s input size n
Examples:
• searching
• element uniqueness
• graph connectivity
• graph acyclicity
• primality testing
Class NP
NP (nondeterministic polynomial): class of decision problems whose proposed
solutions can be verified in polynomial time = solvable by a nondeterministic
polynomial algorithm
A nondeterministic polynomial algorithm is an abstract two-stage procedure that:
• generates a random string purported to solve the problem
• checks whether this solution is correct in polynomial time
By definition, it solves the problem if it’s capable of generating and verifying a
solution on one of its tries
Example: CNF satisfiability
Problem: Is a boolean expression in its conjunctive normal form (CNF) satisfiable, i.e.,
are there values of its variables that makes it true? This problem is in NP.
Nondeterministic algorithm:
• Guess truth assignment
• Substitute the values into the CNF formula to see if it evaluates to true
109
NP HARD AND NP COMPLETE
Polynomial Time algorithms
Problems whose solutions times are bounded by polynomials of small degree are called
polynomial time algorithms
Example: Linear search, quick sort, all pairs shortest path etc.
Non- Polynomial time algorithms
Problems whose solutions times are bounded by non-polynomials are called non-
polynomial time algorithms
Examples: Travelling salesman problem, 0/1 knapsack problem etc
It is impossible to develop the algorithms whose time complexity is polynomial for
non-polynomial time problems, because the computing times of non-polynomial are
greater than polynomial. A problem that can be solved in polynomial time in one model
can also be solved in polynomial time.
NP-Hard and NP-Complete Problem:
Let P denote the set of all decision problems solvable by deterministic algorithm in
polynomial time. NP denotes set of decision problems solvable by nondeterministic
algorithms in polynomial time. Since, deterministic algorithms are a special case of
nondeterministic algorithms, P ⊆ NP. The nondeterministic polynomial time problems
can be classified into two classes. They are
1. NP Hard and
2. NP Complete
NP-Hard: A problem L is NP-Hard iff satisfiability reduces to L i.e., any
nondeterministic polynomial time problem is satisfiable and reducable then the problem
is said to be NP-Hard.
Example: Halting Problem, Flow shop scheduling problem
A problem that is NP-Complete has the property that it can be solved in polynomial time
iff all other NP-Complete problems can also be solved in polynomial time. (NP=P)
110
If an NP-hard problem can be solved in polynomial time, then all NP- complete problems
can be solved in polynomial time. All NP-Complete problems are NP-hard, but some NP-
hard problems are not known to be NP- Complete.
Normally the decision problems are NP-complete but the optimization problems are NP-
Hard.
However if problem L1 is a decision problem and L2 is an optimization problem, then it is
possible that L1α L2.
Example: Knapsack decision problem can be reduced to knapsack
optimization problem.
There are some NP-hard problems that are not NP-Complete.
Let P, NP, NP-hard, NP-Complete are the sets of all possible decision problems that are
solvable in polynomial time by using deterministic algorithms, non-deterministic
algorithms, NP-Hard and NP-complete respectively. Then the relationship between P,
NP, NP-hard, NP-Complete can be expressed using Venn diagram as:
Problem conversion
A decision problem D1 can be converted into a decision problem D2 if there is an
algorithm which takes as input an arbitrary instance I1 of D1 and delivers as output an
instance I2 of D2such that I2 is a positive instance of D2 if and only if I1 is a positive
instance of D1. If D1 can be converted into D2, and we have an algorithm which solves
D2, then we thereby have an algorithm which solves D1. To solve an instance I of D1,
we first use the conversion algorithm to generate an instance I0 of D2, and then use the
algorithm for solving D2 to determine whether or not I0 is a positive instance of D2. If it
is, then we know that I is a positive instance of D1, and if it is not, then we know that I is
a negative instance of D1. Either way, we have solved D1 for that instance. Moreover, in
this case, we can say that the computational complexity of D1 is at most the sum of the
computational complexities of D2 and the conversion algorithm. If the conversion
algorithm has polynomial complexity, we say that D1 is at most polynomially harder than
D2. It means that the amount of computational work we have to do to solve D1, over and
111
above whatever is required to solve D2, is polynomial in the size of the problem instance.
In such a case the conversion algorithm provides us with a feasible way of solving D1,
given that we know how to solve D2.
Given a problem X, prove it is in NP-Complete.
1. Prove X is in NP.
2. Select problem Y that is known to be in NP-Complete.
3. Define a polynomial time reduction from Y to X.
4. Prove that given an instance of Y, Y has a solution iff X has a solution.
Cook’s theorem:
Cook’s Theorem implies that any NP problem is at most polynomially harder than SAT.
This means that if we find a way of solving SAT in polynomial time, we will then be in a
position to solve any NP problem in polynomial time. This would have huge practical
repercussions, since many frequently encountered problems which are so far believed to
be intractable are NP. This special property of SAT is called NP-completeness. A
decision problem is NP-complete if it has the property that any NP problem can be
converted into it in polynomial time. SAT was the first NP-complete problem to be
recognized as such (the theory of NP-completeness having come into existence with the
proof of Cook’s Theorem), but it is by no means the only one. There are now literally
thousands of problems, cropping up in many different areas of computing, which have
been proved to be NP- complete.
In order to prove that an NP problem is NP-complete, all that is needed is to show that
SAT can be converted into it in polynomial time. The reason for this is that the sequential
composition of two polynomial-time algorithms is itself a polynomial-time algorithm,
since the sum of two polynomials is itself a polynomial.
Suppose SAT can be converted to problem D in polynomial time. Now take any NP
problem D0. We know we can convert it into SAT in polynomial time, and we know we
can convert SAT into D in polynomial time. The result of these two conversions is a
polynomial-time conversion of D0 into
D. since D0 was an arbitrary NP problem, it follows that D is NP-complete
112