2024

Design & Analysis of

Algorithms

COURSE CODE CSI-504

NOTES BY PROF. ADEEL SHAHZAD

0345-7376153 | [email protected]
Design & Analysis of Algorithms CSI-504

Introduction to Design and Analysis of Algorithms

### Introduction to Design and Analysis of Algorithms

The design and analysis of algorithms is a fundamental area in computer science, focusing on the
creation of efficient and effective algorithms to solve computational problems. This field
involves two main aspects: designing algorithms that provide solutions to specific problems and
analyzing their efficiency and effectiveness.

#### 1. What is an Algorithm?

An algorithm is a step-by-step procedure or a set of rules to be followed in calculations or

problem-solving operations, usually by a computer. Algorithms are essential for performing
tasks in an automated way and are used in various applications from simple calculations to
complex data processing.

#### 2. Importance of Algorithm Design

- **Efficiency**: Efficient algorithms make optimal use of resources such as time and memory.
For example, sorting algorithms like QuickSort and MergeSort are designed to sort data quickly
and efficiently.

- **Scalability**: Well-designed algorithms can handle large inputs effectively. For instance,
search engines use sophisticated algorithms to handle vast amounts of data.

- **Correctness**: A correctly designed algorithm ensures accurate and reliable results.

- **Optimization**: Optimization algorithms seek the best solution among many possible ones,
crucial in fields like operations research and artificial intelligence.

#### 3. Key Concepts in Algorithm Design

- **Problem Definition**: Clearly defining the problem is the first step in designing an
algorithm. This involves understanding the inputs, desired outputs, and constraints.

- **Algorithmic Paradigms**: These are general approaches to solving problems. Common

paradigms include:

Page | 1
Design & Analysis of Algorithms CSI-504

- **Divide and Conquer**: Breaking a problem into smaller sub-problems, solving each one,
and combining the results (e.g., MergeSort).

- **Dynamic Programming**: Solving complex problems by breaking them down into

simpler subproblems and storing the results of these subproblems to avoid redundant work (e.g.,
Fibonacci sequence calculation).

- **Greedy Algorithms**: Making the locally optimal choice at each step with the hope of
finding a global optimum (e.g., Dijkstra's shortest path algorithm).

- **Backtracking**: Trying out all possible solutions and abandoning a path as soon as it is
determined that it cannot lead to a viable solution (e.g., solving mazes).

#### 4. Analysis of Algorithms

- **Time Complexity**: Measures the amount of time an algorithm takes to complete as a

function of the length of the input. Common notations include:

- **Big O (O)**: Upper bound on time, representing the worst-case scenario.

- **Omega (Ω)**: Lower bound on time, representing the best-case scenario.

- **Theta (Θ)**: Tight bound on time, representing the average-case scenario.

- **Space Complexity**: Measures the amount of memory an algorithm uses as a function of

the length of the input.

- **Correctness**: Ensuring the algorithm produces the correct output for all valid inputs.

- **Optimality**: Analyzing whether an algorithm is the best possible in terms of time and
space complexity.

#### 5. Practical Applications

Algorithms are pervasive in modern technology. Here are a few applications:

- **Search Engines**: Use algorithms to index and retrieve information efficiently.

- **Cryptography**: Relies on algorithms for secure communication.

Page | 2
Design & Analysis of Algorithms CSI-504

- **Machine Learning**: Algorithms form the backbone of learning models and data
processing.

- **Network Routing**: Algorithms determine the best path for data to travel across networks.

### Conclusion

The study of design and analysis of algorithms is essential for creating effective computational
solutions. Understanding and applying the principles of algorithm design can lead to more
efficient, scalable, and reliable software, which is critical in today’s data-driven world. This field
is not only about finding solutions but also about finding the most optimal and efficient
solutions.

Asymptotic Notations

Asymptotic notations are used to describe the limiting behavior of a function when the argument

tends towards a particular value or infinity. These notations help analyze and compare the

efficiency of algorithms, particularly in terms of time and space complexity.

asymptotic notation like a tool that help us to compare how different program

1. Big O Notation (O)

Big O notation describes an upper bound on the time complexity of an algorithm. It provides the

worst-case scenario, ensuring that the algorithm will not exceed a certain amount of time for

sufficiently large inputs.

Formal Definition:

Page | 3
Design & Analysis of Algorithms CSI-504

2. Big Omega Notation (Ω)

Big Omega notation provides a lower bound on the time complexity of an algorithm. It
represents the best-case scenario or the minimum time required by the algorithm for sufficiently
large inputs.

Page | 4
Design & Analysis of Algorithms CSI-504

3. Big Theta Notation (Θ)

Big Theta notation provides a tight bound on the time complexity of an algorithm. It combines
both the upper and lower bounds, giving a precise asymptotic behavior of the algorithm.

4. Little o Notation (o)

Little o notation describes an upper bound that is not asymptotically tight. It represents the
upper bound of the algorithm’s growth rate in a loose manner.

Page | 5
Design & Analysis of Algorithms CSI-504

5. Little Omega Notation (ω)

Little omega notation describes a lower bound that is not asymptotically tight. It represents the
lower bound of the algorithm’s growth rate in a loose manner.

Summary

Understanding asymptotic notations is essential for analyzing and comparing the efficiency of
algorithms, especially for large input sizes. These notations provide a way to describe the upper,
lower, and tight bounds of the growth rates of algorithms, helping to determine their performance
and scalability.

Recursion and Recurrence Relations

Recursion

Recursion is a programming technique where a function calls itself in order to solve smaller
instances of the same problem. This technique is commonly used to break down complex
problems into simpler sub-problems, making them easier to solve. Recursion involves two main
parts: the base case and the recursive case.

Key Components of Recursion:

1. Base Case: The condition under which the recursive function stops calling itself. It
prevents infinite recursion and eventually terminates the recursive calls.

Page | 6
Design & Analysis of Algorithms CSI-504

2. Recursive Case: The part of the function where it calls itself with a modified argument,
moving towards the base case.

Example of a Recursive Function:

Recurrence Relations

Recurrence relations are equations or inequalities that describe a function in terms of its value at
smaller inputs. They are used to analyze the time complexity of recursive algorithms.

Formulation of Recurrence Relations:

To derive a recurrence relation, identify how the solution to the problem depends on the
solutions to smaller sub-problems.

Example of Recurrence Relations:

Consider the recursive definition of the factorial function:

1 & \text{if } n = 0 \\

n \cdot T(n-1) & \text{if } n > 0

Page | 7
Design & Analysis of Algorithms CSI-504

\end{cases} \]

Here, \( T(n) \) denotes the time complexity of computing \( n! \).

### Solving Recurrence Relations

There are several methods to solve recurrence relations, including:

1. **Substitution Method**:

- Guess the form of the solution.

- Use mathematical induction to prove that the guess is correct.

2. **Recursion Tree Method**:

- Visualize the recurrence as a tree.

- Sum the costs of all levels to get the total cost.

3. **Master Theorem**:

- Provides a direct way to solve recurrences of the form \( T(n) = aT\left(\frac{n}{b}\right) + f(n)
\).

### Example Solved Using Different Methods

#### Example Recurrence Relation:

\[ T(n) = 2T\left(\frac{n}{2}\right) + n \]

**1. Substitution Method:**

- Guess: \( T(n) = O(n \log n) \)

- Induction Hypothesis: Assume \( T(k) \leq c k \log k \) for all \( k < n \).

Page | 8
Design & Analysis of Algorithms CSI-504

- Induction Step:

\[

T(n) = 2T\left(\frac{n}{2}\right) + n \\

\leq 2 \left(c \frac{n}{2} \log \frac{n}{2} \right) + n \\

= cn \log \frac{n}{2} + n \\

= cn (\log n - 1) + n \\

= cn \log n - cn + n \\

= cn \log n - (c - 1)n \\

\leq cn \log n \text{ for sufficiently large } n \text{ and appropriate } c

\]

- Conclusion: \( T(n) = O(n \log n) \).

**2. Recursion Tree Method:**

- The root of the tree has cost \( n \).

- Each level splits into two subproblems of size \( \frac{n}{2} \).

- Each level \( i \) has \( 2^i \) nodes, each contributing \( \frac{n}{2^i} \) work.

- Total work at level \( i \) is \( n \).

- Total levels: \( \log n \).

- Total cost: \( n \log n \).

Page | 9
Design & Analysis of Algorithms CSI-504

**3. Master Theorem:**

- Compare \( f(n) = n \) with \( n^{\log_b a} \):

\[

a = 2, b = 2 \implies n^{\log_2 2} = n

\]

- \( f(n) = O(n^{\log_b a}) \):

\[

T(n) = O(n \log n) \text{ by Master Theorem Case 2}

\]

Conclusion

Recursion and recurrence relations are essential concepts in computer science, especially for
designing and analyzing algorithms. Recursion simplifies the process of breaking down
problems, while recurrence relations help in understanding the time complexity of these
recursive solutions. By mastering these concepts, you can create more efficient algorithms and
understand their performance characteristics.

Divide and Conquer Approach

Divide and Conquer Approach

The divide and conquer approach is a powerful algorithmic paradigm based on multi-branched
recursion. It involves breaking down a problem into smaller subproblems that are easier to solve,
solving each subproblem recursively, and then combining the solutions to get the final result.
This method is particularly effective for problems that can be decomposed into similar
subproblems of smaller size.
Page | 10
Design & Analysis of Algorithms CSI-504

Steps in Divide and Conquer:

1. Divide: Split the problem into a number of smaller subproblems of the same type.
2. Conquer: Solve the subproblems recursively. If the subproblems are small enough, solve
them directly.
3. Combine: Merge the solutions of the subproblems to form the solution to the original
problem.

Examples of Divide and Conquer Algorithms:

1. Merge Sort:
o Divide: Split the array into two halves.
o Conquer: Recursively sort each half.
o Combine: Merge the two sorted halves to produce the sorted array.

Quick Sort:

 Divide: Choose a pivot element and partition the array into elements less than the pivot
and elements greater than the pivot.
 Conquer: Recursively sort the subarrays.
 Combine: Concatenate the sorted subarrays and the pivot.

Binary Search:

 Divide: Split the array into two halves.

 Conquer: Determine which half contains the target value and recursively search in that
half.
 Combine: The result is the position of the target value if found, otherwise indicate that
the target is not in the array.

Analysis of Divide and Conquer Algorithms:

The time complexity of divide and conquer algorithms is often analyzed using recurrence
relations.

Page | 11
Design & Analysis of Algorithms CSI-504

 Merge Sort:

Advantages of Divide and Conquer:

 Efficiency: Can significantly reduce the time complexity of problems, making them more
efficient to solve.
 Parallelism: Subproblems can often be solved independently and in parallel, taking
advantage of multi-core processors.
 Simplicity: Breaks complex problems into simpler subproblems, making the algorithm
easier to understand and implement.

Disadvantages of Divide and Conquer:

 Overhead: Recursive calls and merging steps can introduce overhead.

 Space Complexity: Can require additional memory for storing subproblems, especially
for algorithms like merge sort.

Conclusion

The divide and conquer approach is a powerful strategy for solving complex problems
efficiently. By breaking down problems into smaller, more manageable subproblems, solving
them recursively, and combining their solutions, many classical algorithms achieve optimal

Page | 12
Design & Analysis of Algorithms CSI-504

performance and scalability. Understanding and applying this technique is essential for designing
efficient algorithms in computer science.

Define Sorting

Sorting

Sorting is the process of arranging data in a specific order, typically in ascending or descending
order. Sorting is fundamental in computer science and is used in a variety of applications,
including searching, database query optimization, and data analysis.

Importance of Sorting

1. Efficient Searching: Many searching algorithms, such as binary search, require sorted
data.
2. Data Organization: Sorting helps organize data, making it easier to understand and
analyze.
3. Optimization: Sorting can optimize other algorithms, such as those that perform merge
operations.
4. Data Duplication: Sorted data makes it easier to identify and remove duplicates.

Types of Sorting

Sorting algorithms can be classified into two main categories based on their method of operation:

1. Internal Sorting: Sorting that is performed entirely in main memory.

2. External Sorting: Sorting that involves data residing on external storage, used when data
is too large to fit into memory.

Page | 13
Design & Analysis of Algorithms CSI-504

Common Sorting Algorithms

1. Bubble Sort

Bubble Sort is a simple comparison-based sorting algorithm. It repeatedly steps through the list,
compares adjacent elements, and swaps them if they are in the wrong order.

2. Selection Sort

Selection Sort repeatedly finds the minimum element from the unsorted portion and moves it to
the beginning.

Page | 14
Design & Analysis of Algorithms CSI-504

Algorithm:

Time Complexity:

3. Insertion Sort

Insertion Sort builds the final sorted array one item at a time, with the assumption that the first
part of the array is already sorted.

4. Merge Sort

Merge Sort is a divide-and-conquer algorithm. It divides the array into two halves, recursively
sorts them, and then merges the sorted halves.

5. Quick Sort

Quick Sort is another divide-and-conquer algorithm. It picks a pivot element, partitions the array
around the pivot, and then recursively sorts the partitions.

Page | 15
Design & Analysis of Algorithms CSI-504

6. Heap Sort

Heap Sort converts the array into a heap data structure, then repeatedly extracts the maximum
element from the heap and rebuilds the heap.

Stability in Sorting

A sorting algorithm is said to be stable if it preserves the relative order of equal elements.
Stability is important when sorting records based on multiple keys.

Example: Consider sorting the following list of tuples first by the first element, and then by the
second element:

Page | 16
Design & Analysis of Algorithms CSI-504

Conclusion

Sorting is a critical operation in computer science with numerous applications. Understanding

the different sorting algorithms, their time complexities, and their properties is essential for
choosing the right algorithm for a specific problem. Whether you need a stable sort, an in-place
sort, or a sort that performs well on average, there is a sorting algorithm that fits the requirement.

What is Heaps

A heap is a specialized tree-based data structure that satisfies the heap property. There are two
types of heaps: max-heaps and min-heaps. In a max-heap, for any given node, the value of the
node is greater than or equal to the values of its children, with the highest value at the root.
Conversely, in a min-heap, the value of the node is less than or equal to the values of its children,
with the lowest value at the root.

Types of Heaps

1. Max-Heap

In a max-heap, each parent node is greater than or equal to its children.

Properties:

 The largest element is at the root.

 Each node is greater than or equal to its children.

Page | 17
Design & Analysis of Algorithms CSI-504

Example:

2. Min-Heap

In a min-heap, each parent node is less than or equal to its children.

Properties:

 The smallest element is at the root.

 Each node is less than or equal to its children.

Example:

Binary Heaps

A binary heap is a complete binary tree, which means all levels of the tree are fully filled except
possibly for the last level, which is filled from left to right.

Page | 18
Design & Analysis of Algorithms CSI-504

Array Representation of Binary Heaps

A binary heap can be efficiently implemented using an array. For a node at index i:

 The left child is at index 2*i + 1.

 The right child is at index 2*i + 2.
 The parent is at index (i-1) // 2.

Example: For the max-heap:

Operations on Heaps

1. Insertion

To insert a new element into a heap:

1. Add the new element to the end of the heap (preserving the complete tree property).
2. "Heapify" the tree by comparing the added element with its parent and swapping if
necessary, until the heap property is restored.

2. Deletion

To delete the root element (max element in a max-heap or min element in a min-heap):

1. Replace the root with the last element in the heap.

2. Remove the last element.
3. "Heapify" the tree by comparing the new root with its children and swapping if
necessary, until the heap property is restored.

Page | 19
Design & Analysis of Algorithms CSI-504

Building a Heap

Building a heap from an unsorted array can be done efficiently using the "heapify" operation
from the bottom up.

Applications of Heaps

1. Heap sort: An efficient comparison-based sorting algorithm.

2. Priority Queues: Implemented using heaps to efficiently support insertion and extraction
of the highest (or lowest) priority element.
3. Graph Algorithms: Such as Dijkstra's shortest path algorithm, which uses a priority
queue.
4. Order Statistics: Finding the k-th smallest (or largest) element in an array.

Heap sort

Heapsort is a comparison-based sorting algorithm that uses a binary heap data structure. It has a
time complexity of O(nlog⁡n)O(n \log n)O(nlogn) and is not a stable sort.

Algorithm:

1. Build a max-heap from the input array.

2. Swap the root of the heap with the last element.
3. Reduce the heap size by one and heapify the root element.
4. Repeat steps 2-3 until the heap is empty.

Conclusion

Heaps are versatile data structures with a wide range of applications, from sorting and priority
queues to graph algorithms. Understanding their properties, operations, and implementations is
essential for efficient algorithm design and optimization in computer science.

Page | 20
Design & Analysis of Algorithms CSI-504

What is Hashing

Hashing

Hashing is a technique used to map data of arbitrary size to fixed-size values. The values
returned by a hash function are called hash values, hash codes, digests, or simply hashes.
Hashing is widely used in computer science for efficient data retrieval, storage, and management.

Hash Functions

A hash function is a function that takes an input (or "key") and returns a fixed-size string of
bytes. The output, typically a number, is usually a "hash code" that is used for quick data
retrieval.

Properties of a Good Hash Function:

1. Deterministic: The same input will always produce the same output.
2. Fast Computation: The hash function should be able to return the hash code quickly.
3. Uniform Distribution: Hash values should be uniformly distributed to minimize
collisions.
4. Minimal Collisions: Different inputs should produce different hash codes as much as
possible.
5. Avalanche Effect: A small change in the input should produce a significantly different
hash code.

Hash Tables

A hash table (or hash map) is a data structure that implements an associative array abstract data
type, a structure that can map keys to values. A hash table uses a hash function to compute an
index into an array of buckets or slots, from which the desired value can be found.

Page | 21
Design & Analysis of Algorithms CSI-504

Basic Operations:

1. Insert: Add a key-value pair to the hash table.

2. Delete: Remove the key-value pair from the hash table.
3. Search: Find the value associated with a given key.

Collision Resolution Techniques

Collisions occur when two keys hash to the same index. There are several techniques to handle
collisions:

1. Chaining

Chaining uses linked lists to handle collisions. Each slot in the hash table points to a linked list of
entries that hash to the same index.

2. Open Addressing

Open addressing resolves collisions by finding another open slot within the hash table. Several
probing techniques are used:

Page | 22
Design & Analysis of Algorithms CSI-504

Rehashing

Rehashing is the process of increasing the size of the hash table and re-inserting all the existing
elements using the new hash function. This is typically done to maintain a low load factor, which
is the ratio of the number of elements to the number of slots in the hash table.

Applications of Hashing

1. Databases: Indexing and quick retrieval of data.

2. Caching: Storing the results of expensive computations for fast retrieval.
3. Cryptography: Hash functions are used in various encryption algorithms and digital
signatures.
4. Data Structures: Implementing sets, dictionaries, and maps.
5. Checksum Functions: Detecting errors in data storage and transmission.

Hashing Algorithms

1. MD5 (Message Digest Algorithm 5): Produces a 128-bit hash value, widely used but
prone to collisions.
2. SHA (Secure Hash Algorithms): A family of cryptographic hash functions with varying
digest sizes (SHA-1, SHA-256, SHA-512).
3. CRC (Cyclic Redundancy Check): Used for error-checking in networks and storage
devices.

Page | 23
Design & Analysis of Algorithms CSI-504

Hashing in Programming Languages

Many modern programming languages provide built-in support for hash tables, often called
dictionaries or maps.

Conclusion

Hashing is a crucial concept in computer science, providing efficient methods for data retrieval
and management. Understanding the principles of hash functions, collision resolution techniques,
and the implementation of hash tables is essential for designing and optimizing algorithms and
data structures in various applications.

Greedy Approach

Greedy Approach

The greedy approach is a method for solving optimization problems by making a sequence of
choices, each of which looks best at the moment. The basic idea is to build up a solution piece by
piece, always choosing the next piece that offers the most immediate benefit. The greedy
approach is characterized by the following steps:

1. Make a Greedy Choice: Select the best option available at the moment.
2. Reduce the Problem: After making the choice, reduce the problem to a smaller subproblem.
3. Repeat: Apply the greedy choice and reduction steps recursively or iteratively until the problem
is solved.

Properties of the Greedy Approach

A greedy algorithm does not always yield the globally optimal solution for all problems. It works
if the problem exhibits the following properties:

Page | 24
Design & Analysis of Algorithms CSI-504

1. Greedy Choice Property: A globally optimal solution can be arrived at by selecting the locally
optimal choice.
2. Optimal Substructure: An optimal solution to the problem contains optimal solutions to
subproblems.

Steps in Designing a Greedy Algorithm

1. Define the Problem: Clearly define the problem and the set of choices.
2. Choose the Greedy Strategy: Determine the greedy strategy by identifying the choice that seems
best at each step.
3. Prove the Greedy-Choice Property: Prove that a sequence of locally optimal choices leads to a
globally optimal solution.
4. Prove Optimal Substructure: Prove that the optimal solution of the problem contains optimal
solutions to its subproblems.
5. Develop the Algorithm: Implement the algorithm based on the greedy strategy.
6. Analyze the Algorithm: Evaluate the time complexity and correctness of the algorithm.

Examples of Greedy Algorithms

1. Activity Selection Problem

Given a set of activities with start and finish times, the goal is to select the maximum number of
activities that don't overlap.

Algorithm:

1. Sort activities by their finish times.

2. Select the first activity and add it to the solution.
3. For each subsequent activity, if its start time is greater than or equal to the finish time of the last
selected activity, select it.

2. Fractional Knapsack Problem

Given weights and values of n items and a knapsack of capacity W, the goal is to maximize the
value in the knapsack. Items can be broken into smaller parts.

Page | 25
Design & Analysis of Algorithms CSI-504

Algorithm:

1. Calculate the value-to-weight ratio for each item.

2. Sort items by this ratio in descending order.
3. Take items with the highest ratios first, adding as much of each item as possible until the
knapsack is full.

3. Huffman Coding

Huffman Coding is a lossless data compression algorithm. It assigns variable-length codes to

input characters, with shorter codes assigned to more frequent characters.

Algorithm:

1. Create a priority queue (min-heap) of all characters, with their frequencies as the key.
2. Extract the two nodes with the smallest frequencies.
3. Create a new internal node with these two nodes as children and the sum of their
frequencies as the new frequency.
4. Insert the new node into the priority queue.
5. Repeat steps 2-4 until there is only one node left, which is the root of the Huffman tree .

Advantages and Disadvantages of the Greedy Approach

Advantages:

1. Simplicity: Greedy algorithms are usually simple and easy to understand.

2. Efficiency: They can be very efficient in terms of time and space complexity.
3. Locally Optimal Solutions: Often provide good approximations to the optimal solution.

Disadvantages:

1. Not Always Optimal: Greedy algorithms do not always produce the globally optimal solution.
2. Problem-Specific: The success of a greedy algorithm is highly dependent on the problem.

Page | 26
Design & Analysis of Algorithms CSI-504

Conclusion

The greedy approach is a powerful method for solving optimization problems where making a
series of local, optimal choices leads to a global optimum. While it is not universally applicable,
when the greedy-choice property and optimal substructure are present, greedy algorithms can be
highly effective and efficient.

Dynamic Programming

Dynamic Programming

Dynamic Programming (DP) is a method for solving complex problems by breaking them down
into simpler subproblems. It is particularly useful for optimization problems where the solution
can be composed of solutions to subproblems. DP is both a mathematical optimization method
and a computer programming method.

Key Concepts in Dynamic Programming

1. Optimal Substructure: A problem exhibits optimal substructure if an optimal solution to the

problem contains optimal solutions to its subproblems.
2. Overlapping Subproblems: A problem has overlapping subproblems if it can be broken down
into subproblems which are reused several times.

Approaches to Dynamic Programming

There are two main approaches to implementing DP:

1. Top-Down Approach (Memorization):

o In the top-down approach, we solve the problem recursively and store the results of
subproblems to avoid redundant calculations.
o This approach uses recursion and a hash table (or an array) to store the results of
subproblems.

Page | 27
Design & Analysis of Algorithms CSI-504

2. Bottom-Up Approach (Tabulation):

o In the bottom-up approach, we solve the subproblems first and use their solutions to build
up solutions to larger problems.
o This approach uses iteration and typically a table (array) to store the results.

Steps to Solve a Problem Using Dynamic Programming

1. Characterize the Structure of an Optimal Solution:

o Identify how the solution to the problem can be constructed from solutions to
subproblems.
2. Define the Value of an Optimal Solution Recursively:
o Express the solution in terms of solutions to smaller subproblems.
3. Compute the Value of an Optimal Solution (Memorization or Tabulation):
o Use either a top-down or bottom-up approach to compute the solution.
4. Construct an Optimal Solution (if required):
o Once the optimal value is computed, if the problem requires the solution itself (not just
the value), trace back the computed values to construct the solution.

Examples of Dynamic Programming

1. Fibonacci Sequence

The Fibonacci sequence is a classic example of a problem that can be solved with DP.

2. Longest Common Subsequence (LCS)

Given two sequences, find the length of the longest subsequence present in both of them.

3. Knapsack Problem

Given a set of items, each with a weight and a value, determine the number of each item to
include in a collection so that the total weight is less than or equal to a given limit and the total
value is as large as possible.

Page | 28
Design & Analysis of Algorithms CSI-504

Conclusion

Dynamic Programming is a powerful technique for solving optimization and search problems
that exhibit optimal substructure and overlapping subproblems. By breaking a problem down
into simpler subproblems and solving each subproblem only once, DP can significantly improve
the efficiency of algorithms. Understanding both the top-down and bottom-up approaches and
recognizing when a problem can be effectively solved using DP are crucial skills in computer
science and algorithm design.

Graph Algorithms

Graph Algorithms

Graph algorithms are a set of instructions designed to solve problems related to graph theory,
which is a study of graphs used to model pairwise relations between objects. Graphs are used in a
variety of fields, including computer science, network analysis, logistics, and social science.

Basic Terminology

 Graph (G): A set of vertices (V) and a set of edges (E) connecting pairs of vertices.
 Vertex (Node): A fundamental unit of a graph.
 Edge: A connection between two vertices.
 Directed Graph: A graph where edges have a direction.
 Undirected Graph: A graph where edges do not have a direction.
 Weighted Graph: A graph where edges have weights (values).
 Path: A sequence of edges that connects a sequence of vertices.
 Cycle: A path that starts and ends at the same vertex without repeating any edge or
vertex.

Page | 29
Design & Analysis of Algorithms CSI-504

Types of Graph Algorithms

1. Traversal Algorithms
2. Shortest Path Algorithms
3. Minimum Spanning Tree Algorithms
4. Flow Algorithms
5. Cycle Detection Algorithms

1. Traversal Algorithms

Traversal algorithms are used to visit all the vertices in a graph. They are fundamental for
understanding the structure and properties of the graph.

Breadth-First Search (BFS)

BFS explores the graph level by level, starting from a given source vertex. It uses a queue to
keep track of the vertices to be explored next.

Depth-First Search (DFS)

DFS explores as far as possible along each branch before backtracking. It uses a stack (or
recursion) to keep track of the vertices to be explored next.

2. Shortest Path Algorithms

Shortest path algorithms find the shortest path from a source vertex to one or more other vertices.

Dijkstra’s Algorithm

Dijkstra’s algorithm finds the shortest paths from a source vertex to all other vertices in a
weighted graph with non-negative weights.

Page | 30
Design & Analysis of Algorithms CSI-504

Bellman-Ford Algorithm

The Bellman-Ford algorithm computes shortest paths from a single source vertex to all other
vertices in a weighted graph. It can handle negative weights.

3. Minimum Spanning Tree (MST) Algorithms

MST algorithms find a subset of the edges that connects all vertices in the graph with the
minimum possible total edge weight.

Prim’s Algorithm

Prim’s algorithm builds the MST by starting from an arbitrary vertex and adding the cheapest
edge from the tree to a vertex not yet in the tree.

Kruskal’s Algorithm

Kruskal’s algorithm builds the MST by sorting all edges and adding them one by one, as long as
they don’t form a cycle.

4. Flow Algorithms

Flow algorithms solve problems related to finding the maximum flow in a flow network.

Ford-Fulkerson Algorithm

The Ford-Fulkerson algorithm computes the maximum flow in a flow network using the concept
of augmenting paths.

5. Cycle Detection Algorithms

Cycle detection algorithms are used to determine if a cycle exists in a graph.

Page | 31
Design & Analysis of Algorithms CSI-504

Conclusion

Graph algorithms are essential for solving a wide range of problems in computer science and
beyond. Understanding different types of graph algorithms and their applications allows for
efficient problem-solving and optimization in areas such as network design, scheduling, and
resource allocation. Whether dealing with traversal, shortest paths, minimum spanning trees,
flow networks, or cycle detection, mastering graph algorithms is fundamental for algorithm
design and analysis.

What is Network Flow

Network flow is a concept in graph theory that models the movement of resources or entities
through a network of nodes and edges. It is a mathematical abstraction used to study and solve
optimization problems related to the efficient distribution or transportation of goods, data, or any
measurable quantity across a network.

Key Concepts in Network Flow:

 Source and Sink:

 Source: A special vertex from which the flow begins.

 Sink: A special vertex where the flow ends.

Page | 32
Design & Analysis of Algorithms CSI-504

Flow:

 Flow on an Edge: The amount of flow passing through an edge.

 Net Flow at a Vertex: The difference between total incoming and total outgoing flow.

Types of Network Flow Problems

1. Maximum Flow Problem: Find the maximum amount of flow that can be sent from the
source to the sink in a network.
2. Minimum Cut Problem: Find the minimum capacity cut (separating the source from the
sink) in a network.

Algorithms for Network Flow

1. Ford-Fulkerson Algorithm

The Ford-Fulkerson algorithm is a method for computing the maximum flow in a flow network
using the concept of augmenting paths.

 Steps:
1. Start with an initial flow of 0.
2. While there exists an augmenting path from the source to the sink:
 Find the maximum possible flow along the path (bottleneck capacity).
 Augment the flow along the path.
3. The maximum flow is the sum of all augmenting paths.

Page | 33
Design & Analysis of Algorithms CSI-504

2. Edmonds-Karp Algorithm

Edmonds-Karp is an implementation of the Ford-Fulkerson method using BFS to find the

augmenting path, ensuring O(VE)2 time complexity.

3. Push-Relabel Algorithm (e.g., Preflow-Push)

Push-relabel algorithms improve on Ford-Fulkerson by maintaining a preflow, where flow can

be pushed along edges that allow it.

3. Push-Relabel Algorithm (e.g., Preflow-Push)

Push-relabel algorithms improve on Ford-Fulkerson by maintaining a preflow, where flow can

be pushed along edges that allow it.

Applications of Network Flow

1. Transportation Networks: Optimizing the flow of goods or traffic through a network of

roads, railways, or pipelines.
2. Telecommunications: Efficient routing of data packets through computer networks.
3. Water Supply Networks: Maximizing water distribution through pipelines to meet
demand.
4. Max-Flow Min-Cut Theorem: The maximum flow in a network is equal to the
minimum capacity of a cut separating the source and sink.

Conclusion

Network flow problems are fundamental in operations research, computer science, and
engineering. They provide a structured approach to modeling and solving optimization problems
related to the movement of resources in various real-world scenarios. By understanding the
concepts and algorithms associated with network flow, one can effectively design and implement
solutions for a wide range of practical problems involving network optimization.

Page | 34
Design & Analysis of Algorithms CSI-504

What is Disjoint Set

Disjoint Set Data Structure

The Disjoint Set (also known as Union-Find or Merge-Find) data structure is used to efficiently
manage a partitioning of a set into disjoint (non-overlapping) subsets. It supports two main
operations: union and find. Disjoint sets are particularly useful for problems involving dynamic
connectivity, such as finding connected components in a graph or determining if two elements
belong to the same set.

Key Operations

1. Find: Determines the representative of the set containing a particular element. It returns
an identifier (often called a leader or root) that represents the set containing the element.
2. Union: Merges two sets into a single set. It connects two subsets into a larger subset.

Components

 Parent Array (or Parent Pointer Array): A simple representation where each element
points to its parent. The root of a set can be identified by the fact that it points to itself
(self-loop).
 Rank (or Depth): Used to keep the tree flat during union operations to maintain
efficiency. The rank of a node is an upper bound on the height of its tree.

Operations

1. Find Operation (with Path Compression)

The find operation is used to determine the root of the set containing a particular element. Path
compression technique is applied during the find operation to flatten the structure of the tree,
which improves the efficiency of subsequent operations.

Page | 35
Design & Analysis of Algorithms CSI-504

2. Union Operation (with Union by Rank/Size)

The union operation merges two sets into a single set. It uses either the rank or size of the trees
(sets) to decide which one should be the parent when merging.

Applications

 Dynamic Graph Connectivity: Used to efficiently determine if two nodes are in the
same connected component or to merge connected components.
 Kruskal's Minimum Spanning Tree Algorithm: Uses disjoint sets to efficiently
manage and merge sets of vertices.
 Cycle Detection in Graphs: Disjoint sets can detect cycles efficiently during graph
traversal.

Implementation Considerations

 Path Compression: Ensures that the trees representing the sets remain shallow,
improving the efficiency of future operations.
 Union by Rank/Size: Helps keep the trees balanced, preventing the trees from becoming
skewed and maintaining efficient find operations.

Page | 36
Design & Analysis of Algorithms CSI-504

Conclusion

The Disjoint Set data structure is crucial for solving problems related to dynamic connectivity
efficiently. By leveraging path compression and union by rank/size strategies, it provides an
efficient way to manage and merge disjoint sets. Understanding and implementing Disjoint Set
operations is essential for algorithms that require managing partitions of a set, such as graph
algorithms and dynamic connectivity problems.

What is Polynomial and Matrix Calculations

Polynomial Calculations

Polynomials are mathematical expressions consisting of variables (usually denoted as xxx) raised
to non-negative integer powers, multiplied by coefficients. They are fundamental in various
fields of mathematics, including algebra, calculus, and numerical analysis.

Page | 37
Design & Analysis of Algorithms CSI-504

1. Addition and Subtraction: Adding or subtracting polynomials of the same degree

involves adding or subtracting corresponding coefficients.
2. Multiplication: Multiplying polynomials involves distributing each term of one
polynomial across all terms of the other polynomial and then combining like terms.
3. Differentiation: Finding the derivative of a polynomial with respect to xxx.
4. Integration: Finding the indefinite or definite integral of a polynomial with respect to
xxx.

Matrix Calculations

Matrices are rectangular arrays of numbers (or elements) arranged in rows and columns. Matrix
calculations are fundamental in various fields such as linear algebra, physics, computer graphics,
and engineering.

Representation

A matrix AAA with mmm rows and nnn columns is typically represented as:

a_{11} & a_{12} & \ldots & a_{1n} \\

a_{21} & a_{22} & \ldots & a_{2n} \\

\vdots & \vdots & \ddots & \vdots \\

a_{m1} & a_{m2} & \ldots & a_{mn}

\end{bmatrix} \]

where \( a_{ij} \) denotes the element in the \( i \)-th row and \( j \)-th column.

Page | 38
Design & Analysis of Algorithms CSI-504

#### Operations

1. **Matrix Addition and Subtraction**: Adding or subtracting matrices of the same

dimensions involves adding or subtracting corresponding elements.

2. **Matrix Multiplication**: Multiplying two matrices \( A \) and \( B \) results in a matrix \(

C \), where each element \( c_{ij} \) is computed as the dot product of the \( i \)-th row of \( A \)

and the \( j \)-th column of \( B \).

3. **Scalar Multiplication**: Multiplying a matrix by a scalar involves multiplying each

element of the matrix by the scalar.

4. **Matrix Transposition**: Transposing a matrix involves swapping its rows and columns.

5. **Matrix Inversion**: Finding the inverse of a square matrix \( A \), denoted as \( A^{-1} \),

such that \( A A^{-1} = I \), where \( I \) is the identity matrix.

6. **Matrix Determinant**: The determinant of a square matrix \( A \), denoted as \( \det(A) \),

is a scalar value that can be computed using various methods and represents certain properties of

the matrix.

### Applications

- **Polynomial Calculations**: Used in mathematical modeling, interpolation, curve fitting,

and solving differential equations.

Page | 39
Design & Analysis of Algorithms CSI-504

- **Matrix Calculations**: Essential in solving systems of linear equations, computing

transformations in computer graphics, analyzing networks, and solving optimization problems.

### Conclusion

Both polynomial and matrix calculations are fundamental tools in mathematics and various

applied fields. They provide powerful methods for representing and manipulating data and

relationships, allowing for efficient computation and analysis of complex systems and problems.

Understanding these concepts and operations is essential for advanced studies in mathematics,

engineering, computer science, and many other disciplines.

Define String Matching

### String Matching

String matching refers to the process of finding occurrences of a pattern (a substring) within a

text (a larger string). It is a fundamental problem in computer science and has applications in

various fields such as information retrieval, data processing, text processing, and bioinformatics.

### Key Concepts

1. **Text (T)**: The larger string in which we want to search for occurrences of a pattern.

2. **Pattern (P)**: The substring that we are searching for within the text.

Page | 40
Design & Analysis of Algorithms CSI-504

### Methods for String Matching

#### 1. **Naive String Matching**

The naive approach involves checking every position in the text to see if the pattern matches the

substring starting at that position.

- **Algorithm**: Compare the pattern with each substring of the text starting from each

position \( i \).

- **Complexity**: \( O((n - m + 1) \cdot m) \), where \( n \) is the length of the text and \( m \)

is the length of the pattern.

#### 2. **Knuth-Morris-Pratt (KMP) Algorithm**

The KMP algorithm improves efficiency by using information from previous comparisons to

skip unnecessary comparisons.

- **Preprocessing**: Construct a partial match table (prefix function or "pi" table) that allows

the algorithm to skip characters of the text when a mismatch occurs.

- **Complexity**: \( O(n + m) \) for preprocessing and matching, where \( n \) is the length of

the text and \( m \) is the length of the pattern.

#### 3. **Boyer-Moore Algorithm**

The Boyer-Moore algorithm preprocesses the pattern and then uses a heuristic to skip

unnecessary comparisons in the text based on the last occurrence of characters in the pattern.

Page | 41
Design & Analysis of Algorithms CSI-504

- **Preprocessing**: Construct tables (bad character table and good suffix table) to determine

shifts when mismatches occur.

- **Complexity**: Average case \( O(n + m) \), worst-case \( O(n \cdot m) \), where \( n \) is the

length of the text and \( m \) is the length of the pattern.

### Applications

- **Searching**: Finding occurrences of keywords or phrases within a document or database.

- **Text Editing**: Implementing features like find and replace in text editors.

- **Data Mining**: Analyzing large datasets to identify patterns or sequences of interest.

### Conclusion

String matching is a fundamental problem in computer science with numerous practical

applications. Efficient algorithms such as KMP and Boyer-Moore significantly improve the

performance of string matching compared to naive approaches. Understanding these algorithms

and their implementations is crucial for developing efficient search and pattern recognition

systems in various domains.

Page | 42
Design & Analysis of Algorithms CSI-504

NP Complete Problems

**NP-Complete Problems**

NP-complete problems are a class of computational problems within the field of complexity

theory. They are characterized by being both in the complexity class NP (nondeterministic

polynomial time) and being at least as hard as the hardest problems in NP. NP-complete

problems have profound implications in computer science, particularly in algorithm design and

cryptography.

### Key Concepts

1. **NP (Nondeterministic Polynomial Time)**: A class of decision problems where the

solutions can be verified in polynomial time. This means if there is a proposed solution, it can be

verified as correct in polynomial time.

2. **NP-hard**: A class of problems that are at least as hard as the hardest problems in NP.

Unlike NP-complete problems, NP-hard problems may not necessarily be in NP themselves.

### Definition of NP-Complete

A problem \( L \) is NP-complete if:

- \( L \) is in NP, meaning a proposed solution to \( L \) can be verified in polynomial time.

Page | 43
Design & Analysis of Algorithms CSI-504

- Every problem in NP can be reduced to \( L \) in polynomial time (this means that if there is a

polynomial-time algorithm for \( L \), there would be a polynomial-time algorithm for all

problems in NP).

### Implications and Characteristics

- **Complexity**: Solving NP-complete problems optimally (finding an exact solution) is

believed to be computationally intractable. The time required to solve these problems grows

exponentially with problem size.

- **Reduction**: NP-complete problems can be used to prove the hardness of other problems

by reducing them to an NP-complete problem.

- **Examples**: Some well-known NP-complete problems include the Traveling Salesman

Problem (TSP), the Knapsack Problem, the Boolean Satisfiability Problem (SAT), and the

Vertex Cover Problem.

### Cook's Theorem

In 1971, Stephen Cook formulated Cook's theorem, which states that the Boolean Satisfiability

Problem (SAT) is NP-complete. This was a significant breakthrough because it provided the first

example of an NP-complete problem, showing that SAT is at least as hard as any problem in NP.

Page | 44
Design & Analysis of Algorithms CSI-504

### P vs NP Problem

The P vs NP problem asks whether every problem whose solution can be quickly verified by a

computer can also be quickly solved by a computer. NP-complete problems play a central role in

this question because if any NP-complete problem were found to have a polynomial-time

solution (P = NP), then all problems in NP would also have polynomial-time solutions.

### Conclusion

NP-complete problems are a crucial concept in computer science and theoretical mathematics.

They represent some of the most challenging computational problems, and their study has led to

the development of heuristic algorithms and approximation techniques to find near-optimal

solutions for practical purposes. The study of NP-completeness continues to be a vibrant area of

research with implications for cryptography, algorithm design, optimization, and beyond.

Page | 45
Design & Analysis of Algorithms CSI-504

Approximation Algorithms

### Approximation Algorithms

Approximation algorithms are algorithms designed to find near-optimal solutions for

optimization problems quickly. These problems are often NP-hard or NP-complete, meaning

finding an exact optimal solution is computationally impractical for large inputs. Approximation

algorithms aim to provide solutions that are close to the optimal solution, within a guaranteed

factor of the optimal value.

### Key Concepts

1. **Optimization Problems**: Problems where the goal is to find the best (optimal) solution

from a set of feasible solutions, typically minimizing or maximizing some objective function.

2. **Approximation Ratio (Approximation Factor)**: A factor \( \alpha \) such that an

approximation algorithm guarantees that its solution \( S \) is within \( \alpha \) times the optimal

solution \( S^* \). For minimization problems:

\[ \text{maximization problem: } \frac{f(S)}{f(S^*)} \leq \alpha \]

\[ \text{minimization problem: } \frac{f(S^*)}{f(S)} \leq \alpha \]

where \( f(S) \) is the cost (or value) of solution \( S \) and \( f(S^*) \) is the cost of the optimal

solution \( S^* \).

Page | 46
Design & Analysis of Algorithms CSI-504

3. **Polynomial Time**: Approximation algorithms run in polynomial time relative to the size

of the input.

### Types of Approximation Algorithms

#### 1. **Greedy Algorithms**

Greedy algorithms make a series of locally optimal choices at each step, hoping that the

sequence of local optima leads to a globally optimal solution. They are simple to implement and

often provide reasonable approximations for certain problems like Minimum Spanning Tree

(e.g., Prim's algorithm) and Set Cover.

#### 2. **Heuristic Algorithms**

Heuristic algorithms are practical methods that sacrifice optimality for efficiency. They are often

used when exact solutions are impractical due to computational complexity. Examples include

local search algorithms and metaheuristic algorithms like genetic algorithms and simulated

annealing.

#### 3. **Randomized Algorithms**

Randomized algorithms use randomness (random choices or probabilistic analysis) to improve

the probability of finding good solutions. Examples include randomized rounding for Linear

Programming (LP) problems and Monte Carlo methods for certain optimization problems.

Page | 47
Design & Analysis of Algorithms CSI-504

#### 4. **Iterative Improvement Algorithms**

Iterative improvement algorithms start with an initial feasible solution and iteratively improve it

until no further improvements are possible. Examples include approximation algorithms for the

Traveling Salesman Problem (TSP) and certain scheduling problems.

### Applications

Approximation algorithms are widely used in various domains, including:

- **Network Design**: Approximating optimal routing or network layout.

- **Scheduling**: Approximating efficient schedules for tasks or jobs.

- **Clustering**: Approximating optimal groupings or clusters of data points.

- **Facility Location**: Approximating optimal locations for facilities to minimize costs or

maximize coverage.

### Example: Traveling Salesman Problem (TSP)

The TSP is a classic optimization problem where a salesman must visit each of a given set of

cities exactly once and return to the starting city in such a way that the total distance traveled is

minimized. While finding the exact optimal tour is NP-hard, approximation algorithms such as

the Christofides algorithm provide solutions within a factor of \( \frac{3}{2} \) times the optimal

tour length.

Page | 48
Design & Analysis of Algorithms CSI-504

### Conclusion

Approximation algorithms play a crucial role in addressing complex optimization problems

where finding exact solutions is computationally impractical. They provide a balance between

computational feasibility and solution quality, making them invaluable in practical applications

across various disciplines in computer science, operations research, economics, and beyond. The

study and development of approximation algorithms continue to be an active area of research,

contributing to advancements in algorithm design and optimization theory.

Page | 49
Design & Analysis of Algorithms CSI-504

Best of luck Students………………….!!!!!

Page | 50