COSC 3320 SU2021-MidTerm

Dynamic programming: Maxsum - Pseudo Code The Optimal Solution

1: function MaxsumDPMod(A) 2: M[1] ← A[1] 3: Mᶦⁿ[1] ← A[1] 4: Mᵒᵘᵗ[i] ← −∞ 5: S[1] ← in 6: for i ← 2 to n do 7: Mᵒᵘᵗ[i] ← M[i − 1] 8: Mᶦⁿ[i] ← max{Mᶦⁿ[i − 1] + A[i],A[i]} 9: M[i] ← max(Mᶦⁿ[i],Mᵒᵘᵗ[i]) 10: if M[i] = Mᶦⁿ[i] then 11: S[i] ← in 12: else 13: S[i] ← out 14: if Mᶦⁿ[i] = Mᶦⁿ[i − 1] + A[i] then 15: Sᶦⁿ[i] ← True 16: else 17: Sᶦⁿ[i] ← False 18: return M[n]

Dynamic programming: Maxsum - Pseudo Code Naive Recursive Implementation of Maxsum

1: function MaxsumRec(A) 2: return max(MᶦⁿRec(A),MᵒᵘᵗRec(A)) 3: function MᶦⁿRec(A) //Maxsum including the ith element 4: if i = 1 then 5: return A[1] 6: else 7: return max(MᶦⁿRec(A[1, . . . , i − 1]) + A[i]),A[i]) 8: function MᵒᵘᵗRec(A) //Maxsum excluding the ith element 9: if i = 1 then 10: return −∞ 11: else 12: return MaxsumRec(A[1, . . . , i − 1])

Closest Pair of Points: Given a set S of n > 1 points, output a pair of points that are closest among all pairs.

• Compute distances between every pair of points • Output the pair with minimum distance • Costly operation - • comparing coordinates between two given points • Quadratic run time • How to solve this faster? • Dimensions • One dimension, with sorting - O(n log n) • Higher dimensions?? • 2 dimensions • Each point in strip needs to be compared with only a constant (at most 16) # of other points in the strip!

MergeSort

• Divide and Conquer strategy • Split the array into two equal halves • Recursively sort the two subarrays independently • Merge the two sorted subarrays • Uses an auxiliary array

Greedy Activity Selector - Correctness

• Do not always produce optimal solutions • Always finds an optimal solution to an instance of the activity-selection problem • This algorithm produces solutions of maximum size • Set of activities, S = {1, 2,..., n} and activities are ordered by finish time • Activity 1 has the earliest finish time • Optimal solution exists if we begin with a greedy choice - Activity 1 • Problem reduced - once activity 1 is greedily chosen! • Next problem is to find an optimal solution over those activities in S that are compatible with activity 1 • After each greedy choice, the remaining problem is of the same form as the original problem • By induction on the number of choices made, greedy choices at every step gives an optimal solution

Section Sort Analysis

• Examine the code • For each i, there is one exchange and n-1-i comparisons • (n-1) + (n-2) + ... + 2 + 1 + 0 = n(n-1)/2 = ~n²/2 comparisons • Total of n exchanges • To sort an array of length n, Selection Sort uses • ~n²/2 comparisons - quadratic • n exchanges

Selection Sort Algorithm

• Find the smallest item in the array • Exchange it with the first entry • Find the next smallest entry and exchange it with the second entry • Repeat till the entire array is sorted

Linear Time Selection Algorithm

• Follow the partitioning method of QuickSort • Choose the pivot carefully! • Find a pivot by recursively solving another selection problem • Partition set S into groups of 5 elements • Find the median of each of these groups - About n/5 medians. • "median of medians" is a good pivot

Dynamic programming: Matrix Chain Multiplication Find a way to parenthesize the product A₁A₂ . . .Aₙ which minimizes the number of multiplications.

• For two matrices with dimensions m x n and n x p, total number of element-wise multiplication is mnp • Associative property • (AB)C = A(BC) • Does order matter? • Consider this example: • A₁ - 10 X 1000 • A₂ - 1000 X 100 • A₃ - 100 X 200 • In what order we multiply these matrices? • (A₁A₂)A₃ or A₁(A₂A₃) ?? • Cost calculations • (A₁A₂)A₃ → cost? 10X1000x100 + 10X100X200 = 1200000 • A₁(A₂A₃) → cost? 1000X100x200 + 10X1000X200 = 22000000

Find the Maximum - Analysis using Recurrence

• Function T(n) denotes the number of comparisons for an array of size n • Recurrence: T(n) = 2T(n/2) + 1 • Base case: T(1) = 0

Primality Checking

• Given a natural number, n, return True if it is prime or False if it is composite. • Central to number theory and algorithms

Greedy Activity Selector

• Given set of activities, S = {1, 2,..., n} that use a resource • The resource used for only one activity at a time • Each activity start time sᵢ and finish time fᵢ, where sᵢ ≤ fᵢ • Activities i and j are compatible if their time intervals do not overlap • Output: a maximum-size set of mutually compatible activities

Matrix Multiplication

• Given two n x n matrices, A & B • Compute C = A · B • Takes O(n) operations • Algorithm using standard multiplication takes O(n³) time • Adding two matrices takes O(n²) time • How to exploit the above aspect for better time? • addition is cheaper than multiplication • Compute r, s, t and u using 7 matrix multiplications • Matrix additions and subtractions • Recurrence • Assumptions: n is a power of 2; n is fairly large; matrices are not sparse

Dynamic programming: Aligning two Sequences

• Given two string sequences, s and t, output the best alignment between the two sequences • Aligning two characters that are same = 1 • Aligning two different characters = 0 • Aligning a letter with a space = -1 • Score of the alignment = Sum of scores • Best alignment - maximum score - Similarity • How to solve this problem? • Look at all possible alignments of two input strings • Compute the score for each alignment • Take the maximum of the scores • If you have 2n and n length strings, this becomes a problem of at least C(2n, 2) ≥ 2ⁿ • Find best alignment of substrings s[1...i] and t[1...j] where 1 ≤ i ≤ m and 1 ≤ j ≤ n •Only allow the right end of the string to vary

Features of a greedy strategy

• Greedy-choice property • Must prove the greedy choice at each step yields a globally optimal solution • Greedy choice made at the first step reduces the problem to a similar, smaller problem (optimal substructure) • Induction shows that a greedy choice can be applied at every step • Optimal substructure • Similar to DP • Optimal solution to a given problem contains within it optimal solutions to subproblems • Optimal solution A in the activity-selection problem begins with activity 1 • Set of activities A' = A - {1} is an optimal solution to S' = {i ϵ S: sᵢ ≥ f₁}

Runtime of MergeSort

• How many comparisons does it take? • If the two subarrays are of size k, to merge into X not more than 2k comparisons • If the two subarrays are of size k1 and k2, to merge into X - k1+k2 comparisons • Running time: With T(n) as the recurrence • Base case: T(1) = 0 T(n) ≤ 2T(n / 2) + n → T(n) = O(n log n)

Select Algorithm - Runtime

• How many elements in S are larger than p? • By induction, runtime is T(n) ≤ cn, for constant c > 0 • O(n)

Primality Checking - Correctness analysis

• If n is prime, line 3 will always be false, causing line 4 to never execute, line 5 will execute returning True. • If n is composite, line 3 will be true, causing line 4 to execute and return False.

Binary Heap

• Implements a binary Tree with following properties. 1) It's a complete tree (All levels are completely filled except possibly the last level and the last level has all keys as left as possible). This property of Binary Heap makes them suitable to be stored in an array. 2) A Binary Heap is either Min Heap or Max Heap. In a Min Binary Heap, the key at root must be minimum among all keys present in Binary Heap. The same property must be recursively true for all nodes in Binary Tree. Max Binary Heap is similar to MinHeap.

Simple Sorting: Algorithm

• Input • An array A of some length n • Output • Sorts the array in place

(Recursive Algorithm Example) Find Maximum: Given an array S of n elements, return the maximum value in S.

• Input • Array S, indices a and b • Output • Maximum value in S between a and b • Split the problem • Recursively solve subproblems • Combine solutions

Primality Checking - Understand the problem

• Input - A natural number n • Output - True (if n is prime) or False (if n is composite)

Runtimes: Algorithms with numerical inputs

• Input type (numerical) versus input size • Complexity is a function of the input size - number of bits to encode a numerical input • Runtime of O(n) (e.g., Primality Checking) is exponential based on number of bits of the input n • Pseudo-polynomial time algorithms - polynomial in input value, but not input size

Closest Pair of Points - Pseudo Code

• Input: A set S of n points in 2 dimensions stored in X and Y • Output: The closest pair of points in S 1: function ClosestPair(X, Y ) 2: if |X| ≤ 3 then 3: return Brute-Force Solution by comparing all pairs 4: else 5: m ← Median(X) //Partition X around the median 6: X₁ ← {(x, y) ∈ X | x ≤ m} 7: X₂ ← {(x, y) ∈ X | x > m} 8: (a, b) ← ClosestPair(X₁, Y ) 9: (c, d) ← ClosestPair(X₂, Y ) 10: if dist(a, b) ≤ dist(c, d) then 11: δ ← dist(a, b) 12: ClosestPair ← (a, b) 13: else 14: δ ← dist(c, d) 15: ClosestPair ← (c, d) 16: Q ← {(x, y) ∈ Y | |x − m| ≤ δ} //Points in the strip //sorted by y-coordinate. 17: for each p ∈ Q with index i do 18: for each q ∈ Q[i + 1..i + 8] do //Compare p with the next 8 points in the order 19: if dist(p, q) < δ then 20: δ ← dist(p, q) 21: ClosestPair ← (p, q) 22: return ClosestPair

Linear Time Selection Algorithm - Pseudo Code

• Input: An array S and the desired order statistic, k • Output: The value of the kᵗʰ order statistic 1: function Select(S, k) 2: if |S| = 1 then 3: return S[1] 4: Partition S into n/5 groups of 5 elements each and a leftover group of up to 4 elements. 5: Find the median of each group 6: R ← the set of these n/5 medians 7: p ← Select(R, |R|/ 2) 8: S₁ ← {x ∈ S | x < p} 9: S2 ← {x ∈ S | x > p} 10: if |S₁| = k − 1 then 11: return p 12: else if |S1| > k − 1 then 13: return Select(S₁, k) 14: else 15: return Select(S₂, k − |S₁| − 1)

Dynamic programming: Aligning two Sequences - Pseudo Code (Bottom-up) Algorithm O(mn)

• Input: sequences s and t of lengths m and n • Output: sim(s, t) 1: function SimScoreDP(s, t) 2: for i ← 0 to m do 3: a[i, 0] ← − i 4: for j ← 0 to n do 5: a[0, j] ← − j 6: for i ← 1 to m do 7: for j ← 1 to n do 8: both ← a[i − 1, j − 1] + score(s[i], t[j]) 9: left ← a[i, j − 1] + score(−, t[j]) 10: right ← a[i − 1, j] + score(s[i],−) 11: a[i, j] ← max(both, left, right) 12: return a[m, n]

Section Sort Runtime

• Insensitive to the input • Finding smallest item in the first pass does not help in finding the smallest item in the next pass • Time is the same if the array is already sorted or it is randomly ordered!

MIN Heap

• Insert(P, k): Insert key k (along with its data) into priority queue P. • Min(P): Return smallest key (along with its data) in P. • ExtractMin(P): Return and remove the smallest key (along with its data) from P. • DecreaseKey(P, k, v): Decrease the value of key k to value v.

n-Queens Problem - Faster solution

• Linear time - proportional to n • Analytical solution without searching • Generalize the solution from the 4-Queens problem • No solution for n=2 and n=3 • n=5 has a solution as you can place the 5th queen in the diagonal (last square of the 5th column) • Solutions exist for n=6 and n=7, but not for n=8 • All values of n that leave a reminder (0, 1, 4 and 5) when divided by 6 • Also, for values with reminders 2 and 3 when divided by 6 • n-Queens problem has a solution and can be found quickly • To find all the solutions is difficult - exponential time in n

Simple Sorting: Algorithm's Correctness

• Loop invariant of the outer for loop • After the ith iteration, ith smallest element is in the ith index of the array • Never moved from that position again

Features of a Greedy Strategy

• Make a sequence of choices • At each decision point, choice made is the best at that point • Does not always produce optimal solution • Sometimes it does! • How can you tell if greedy algorithm is the way to go for a given optimization problem? • Greedy-choice property • Make locally optimal / greedy choice to arrive at a globally optimal solution • Choice made may depend on choices made so far, but not on any future choices / on solutions to subproblem subproblems • Usually top-down, iteratively making one greedy choice after another

Data movement

• Minimal data movement • Each of the n exchanges swap the values of 2 entries in the array • Linear function of the array length

Data Structure

• Most algorithms involve organizing data involved in the computation • By products or end products of algorithms • Simple algorithms can give rise to complicated data structures • Complicated algorithms use simple data structures. • Examples of data structures

Simple Sorting: Running Time

• Number of comparisons (line 4) • Total number of comparisons: = n(n-1)/2 = n2/2 - n/2 • Dominating term determines the run time - quadratic, O(n²)

Dynamic programming: Matrix Chain Multiplication Correctness

• Optimal substructure property • optimal solution to a problem consists of optimal solution to its subproblems • computing m[i,j] depends on m[i,k] and m[k+1,j], for all i ≤ k < j • Do not consider suboptimal solutions to subproblems!

Dynamic Programming - Summary

• Optimal substructure property • Overlapping subproblems • Recursive formulation • Iterative algorithm (bottom-up) • Recursive algorithm with memoization/table lookup (top-down)

Dynamic programming: Matrix Chain Multiplication Recurrence

• P(n) - number of different ways of parenthesizing a product of n matrices • P(n) is exponential in n, P(n) = Ω(2ⁿ)

Recursive algorithm

• To implement divide and conquer strategy • Solve a problem • By breaking it into smaller size subproblems of the same type (as the original problem) • Repeatedly solve the subproblems • Combine the solutions of the subproblems to get the solution for the original problem • Recursive function • Function that invokes itself • Termination condition

Mathematical Induction

• To prove mathematical theorems • To prove correctness of algorithms • To analyze performance of algorithms (runtime) • To design algorithms • To show a mathematical statement is true for all cases, e.g., for all natural numbers, n≥0 • Base Step • Induction Step • Induction Hypothesis

Find the Maximum - Running Time

• Total number of costly operations • What is the costly operation in this algorithm? • Total number of recursive calls • Size of the recursion tree • Number of nodes of the tree → 2n • Proportional to size of the array → O(n)

When to use DP?

• Two ingredients in an optimization problem • Optimal substructure • Overlapping subproblems

Select Algorithm - Correctness

Proof by Induction • Base case: When |S| = 1 is trivially true • Induction hypothesis: • Assume it is true for all |S| < n • If |S| = n, • the Select algorithm chooses a pivot, p • partitions S into two subarrays, L and R • All elements of L are less than p and all elements of R are greater than p

Insertion Sort in Java

Public Class Insertion { public static void sort(Comparable[] a) { int N = a.length; for (int i = 0; i < N; i++) for (int j = i; j > 0; j --) if (less(a[j], a[j - 1])) exchange(a, i, j - 1); else break; } }

Selection Sort - In Java

Public Class Selection { public static void sort(Comparable[] a) { int N = a.length; for (int i = 0; i < N; i++) { int min = i; for (int j = i + 1; j < N; j++) min = j; exchange(a, i, min); } } }

Shellsort Code Snippet

Public Class Shell { public static void sort(Comparable[] a) { int N = a.length; int h = 1; while (h < N/3) h = 3 * h + 1; //1,4,13,40,121 while (h >= 1) { // h-sort the array for (int i = h; i < N; i++) { for (int j = i; j >= h; j -= h) exchange(a, j, j - h); } h = h / 3; }

Select Algorithm - Correctness: 3 possible cases

1. |L| = k − 1, then p is clearly our target element 2. |L| > k − 1, then there are at least k elements less than p, hence our target element lives in L • Output is Select(L, k) 3. |L| < k − 1, then the elements of L and the pivot, |L| + 1 elements, are all less than the target • the target is the element with rank k − |L| − 1 in R • Output is Select(R, k − |L| − 1)

Building a Heap - Pseudo Code

1: function BuildHeap(A) 2: H ← A 3: for i ← [floor(n/2)] to 1 do 4: Heapify(H, i) 5: return H

Dynamic programming: 0/1 Knapsack - Pseudo Code FPTAS Algorithm

1: function FPTASKnapsack(X, m, ε) 2: P ← max{profit(x) | x ∈ X} 3: K ← εP/n 4: X' ← ∅ //Initialize an empty array X' 5: for w, p ∈ X do //Iterate thru weights & profits of X 6: p' ← [floor(p/K)] 7: append (w, p') to X' 8: return Knapsack(X', m)

Heapify - Pseudo Code

1: function Heapify(A, i) 2: l ← left(i) 3: r ← right(i) 4: if l ≤ |A| and A[i] > A[l] then 5: smallest ← l 6: else 7: smallest ← i 8: if r ≤ |A| and A[smallest] > A[r] then 9: smallest ← r 10: if smallest ≠ i then 11: Swap A[i] and A[smallest] 12: Heapify(A, smallest)

Primality Checking - Write the pseudocode

1: function IsPrime(n) 2: for i ← 2 to n − 1 do 3: if i divides n then 4: return False 5: return True

Dynamic programming: Matrix Chain Multiplication - Pseudo Code Iterative/DP Algorithm O(n³)

1: function MatrixCostDP(p) 2: for i ← 1 to n do 3: M[i, i] ← 0 4: for l ← 2 to n do 5: for l ← 1 to n − i + 1 do 6: j ← i + l + 1 7: m[i, j] ← ∞ 8: for k ← i to j − 1 do 9: q ← m[i, k] + m[k + 1][j] + pi−1pkpj 10: if q < m[i, j] then 11: m[i, j] ← q 12: return m[1, n]

Dynamic programming: Matrix Chain Multiplication - Pseudo Code Recursive/Memoization

1: function MatrixCostRec(p) 2: return MatrixCostHelper(p, 1, n) 3: function MatrixCostHelper(p, i, j) 4: if m[i, j] = 1 then 5: if i = j then 6: M[i, j] ← 0 7: else 8: for k ← i to j − 1 do 9: left ← MatrixCostHelper(p, i, k) 10: right ← MatrixCostHelper(p, k + 1, j) 11: total ← left + right + pi−1pkpj 12: if total < m[i, j] then 13: m[i, j] ← total 14: return m[i, j]

Find Maximum - Write the pseudocode

1: function Max(S) 2: max S[1] 3: for i ← 2 to n do 4: if max < S[i] then 5: max S[i] 6: return max

Dynamic programming: Maxsum - Pseudo Code Iterative Implementation of Maxsum

1: function MaxsumDP(A) 2: M[1] ← A[1] 3: Mᶦⁿ[1] ← A[1] 4: Mᵒᵘᵗ[1] ← −∞1 5: for i ← 2 to n do 6: Mᵒᵘᵗ[i] ← M[i − 1] 7: Mᶦⁿ[i] ← max(Mᶦⁿ[i − 1] + A[i],A[i]) 8: M[i] ← max(Mᶦⁿ[i],Mᵒᵘᵗ[i]) 9: return M[n]

Dynamic programming: Maxsum - Pseudo Code Efficient implementation of the recursive formulation

1: function MaxsumRecLookup(A) 2: for i ← 1 to n do 3: M[i] ← Mᶦⁿ[i] ← Mᵒᵘᵗ[i] ← −∞ //initializ of arrays 4: Mᶦⁿ[1] ← A[1] 5: return max(MᶦⁿRecLookup(n), MᵒᵘᵗRecLookup(n)) 6: function MᶦⁿRecLookup(i) 7: if Mᶦⁿ[i] = −∞ then //Value not yet computed 8: include ← MᶦⁿRecLookup(A[1, . . . , i − 1]) + A[i]) 9: Mᶦⁿ[i] ← max(include,A[i]) 10: return Mᶦⁿ[i] 11: function MᵒᵘᵗRecLookup(i) 12: if Mᵒᵘᵗ[i] = −∞ then //Value not yet computed 13: Mᵒᵘᵗ[i] ← MaxsumRecLookup(A[1, . . . , i − 1]) 14: return Mᵒᵘᵗ[i]

QuickSort - Pseudo Code

1: function QuickSort(S) 2: if |S| ≤ 1 then 3: return S 4: else 5: p ← S[1] //p, the first element of S, //is the pivot 6: S1 ← {x 2 S − {p} | x < p} 7: S2 ← {x 2 S − {p} | x > p} //Elements in //S1 and S2 are in the same order as in S 8: return QuickSort(S1) + [p] + QuickSort(S2)

Simple Sorting - Write the pseudocode

1: procedure SimpleSort(A) 2: for i ← 1 to n do 3: for j ← i + 1 to n do 4: if A[j] < A[i] then 5: swap A[i] and A[j]

Total Order

A binary relation ≤ that satisfies: - Antisymmetry: if both v ≤ w and w ≤ v, then v = w. - Transitivity: if both v ≤ w & w ≤ x, the v ≤ x. - Totality: either v ≤ w or w ≤ v or both. EX: - Standard order for natural and real numbers. - Chronological order for dates or times. - Alphabetical order for strings.

Selection Sort

A sorting algorithm that used the "Divide and Conquer" technique.

Find the Maximum - Algorithm's Correctness

• Base step: Size of S is 1 • Output will be that single element • Induction step • Induction hypothesis: Assume the algorithm is correct for all elements < n • max1 and max2 are correct maximum values for S[a,...,m] and S[m+1,...,b] • Maximum of max1 and max2 == maximum of S

Dynamic Programming

• Break the original problem into subproblems • Combine their solutions to obtain the solution • How is this different from divide-and-conquer? • Subproblems are independent and overlapping • the same subproblem occurs in the solution of other subproblems • Main point of DP - solve each subproblem only once

Performance of Shellsort

• Breaks the quadratic runtime barrier of insertion sort • Proportional to n³/²

Dynamic programming: 0/1 Knapsack DP algorithm

• Compute Pₙ(m) from bottom up: • Base cases • P₁, P₂, ... • Time complexity - O(nm) • Pseudo polynomial runtime and algorithm • Fast when m is small • Convert to polynomial time algorithms for an approximately optimal solution

Building a Heap

The root of the binary heap is A[0] and for i > 0, the children of node A[i] are stored in A[2i + 1] and A[2i + 2], if they exist. Initially, the values in the binary tree need not be organized as a heap. The algorithm Heapify, will organize the elements to satisfy the (min) heap property starting from the leaves. The leaves trivially satisfy the heap property. As the algorithm works on nodes at higher levels, the nodes at the lower levels — and hence the subtree rooted at these nodes — already satisfy the heap property. Thus the conditions for Heapify are satisfied.

Running Time of Heapify Algorithm

If there are n elements in the heap, the height of the complete binary tree is O(log n). Hence Heapify takes O(log n) time in the worst case.

Dynamic programming: Matrix Chain Multiplication - Pseudo Code Naive Algorithm

Input: A sequence of matrix dimensions: p = p₀, p₁, ... , pₙ Output: Optimal cost to multiply Aᵢ...Aⱼ 1: function Cost(p) 2: if i = j then 3: return 0 4: m[i, j] ← ∞ 5: for k ← i to j − 1 do 6: q ← Cost(p, i, k) + Cost(p, k + 1, j) + pᵢ₋₁pₖpⱼ 7: if q < m[i, j] then 8: m[i, j] ← q 9: return m[i, j]

Design an efficient algorithm to identify a celebrity in The Celebrity Problem: A celebrity among a group of n people is a person who knows no one but is known by everyone else. Identify a celebrity by only asking questions to people of the following form: "Do you know this person?".

We start with n people. Choose 2 people, A and B, and ask whether A knows B. If yes, then A cannot be a celebrity and can be eliminated. If no, then B cannot be a celebrity and can be eliminated. This reduces the problem size by one. We repeat the same strategy among the remaining n − 1 people. After n − 1 questions, one person (say X) will remain. If there is a celebrity then it must be this person (Can you see why?). 2 To check whether this person is a celebrity, we ask whether X knows everyone else and whether everyone else knows X. The total number of questions is n − 1 + n − 1 + n − 1 = 3n − 3. An even better algorithm can be given! It can be shown that it can be done using 3n − (floor[log n]) − 3 questions and this is the best possible.

Recursion

When a function calls itself.

MergeSort - Pseudo Code

procedure MergeSortHelper(X, left, right) if left < right then middle = d(left + right)/2e MergeSortHelper(X, left, middle − 1) MergeSortHelper(X, middle, right) Merge(X, left, right) 1: procedure Merge(X, left, right) 2: i ← left //points to index in the first half 3: middle ← [ceiling((left + right)/2)] 4: j ← middle //j points to index in the //second half 5: while i < j and j ≤ right do 6: if X[i] ≤ X[j] then 7: Temp[i + j] ← X[i] //Copy from first //half in Temp 8: i ← i + 1 9: else 10: Temp[i + j] ← X[j] //Copy from //second half in Temp 11: j ← j + 1 12: if j > right then //j reaches the right end 13: for t ← 0 to middle − 1 do //Shift //elements from first half //of X to second half 14: X[right − t] ← X[middle − 1 − t] 15: for t ← 0 to i + j − 1 do 16: X[left + t] ← Temp[t]

Heap Sort

• A comparison-based sorting technique based on Binary Heap data structure. It is similar to selection sort where we first find the minimum element and place the minimum element at the beginning. We repeat the same process for the remaining elements.

Optimal Substructure

• A problem has this ingredient if • The optimal solution to the problem contains optimal solutions to the subproblems • Assume that there is a better solution to the subproblem • Show how this assumption contradicts the optimality of the solution to the original problem

Insertion Sort

• Algorithm used for sorting bridge hands • Consider items one at a time • Insert each item into its proper place amongst those already sorted • Make space to insert by moving larger items one position to the right • Items to the left of the current index are in sorted order, not necessarily in final position

Priority Queue

• An abstract data type similar to a regular queue or stack data structure in which each element additionally has a "priority" associated with it. A priority queue has the following properties: 1. Every item has a priority associated with it. 2. An element with high priority is dequeued before an element with low priority. 3. If two elements have the same priority, they are served according to their order in the queue. 4. Can be implemented using a heap.

QuickSort

• Another Divide and Conquer strategy • Implemented as qsort function in UNIX and C++ • Input is a set S with the first element p chosen as a pivot element • Partition the set based on p into 2 - S1 & S2 • All elements of S1 are less than p • All elements of S2 are greater than p • Recursively sort S1 and S2 • Output is S1 + [p] + S2

Dynamic programming: 0/1 Knapsack FPTAS - Fully Polynomial-time Approximation Scheme

• Approximation algorithm - a polynomial time algorithm • Computes a solution that is close to optimal! • Ratio bound • Knapsack problem is NP-complete • No polynomial time algorithm likely • Polynomial in input size • Runtime should be a polynomial function of • n and log(m) (number of items and capacity) • log(wᵢ) (weights of items) • log(pᵢ) (profits)

Greedy-Activity-Selector - Runtime

• Assumption: Activities are sorted by their finish times • 𝜣(n) • Activity picked in every step is a 'greedy' choice, as it leaves opportunity for remaining activities to be scheduled • Greedy choice maximizes the remaining unscheduled time

Simple Sorting: Given an array, A of n elements that can be ordered (i.e., we can write a ≤ b or b ≤ a), return an ordering A' of A such that A'[1] ≤ A'[2] ≤ . . . ≤ A'[n].

• Assumption: Only way to conclude the ordering between a and b • to compare them • by asking the question "Is a ≤ b?"

Correctness of MergeSort

• Base case: Trivial for the size of 1 • Induction hypothesis: Assume the algorithm is correct for all sets of size < n • Prove the induction step

Dynamic programming: Matrix Chain Multiplication Recursive formulation

• Base case: i = j • Recursive step: i < j • Recursive formulation correct for all l such j − i < l • Induction is on the length of the chain • Formulation correct for j - i = l • m[i,j] is the optimal cost of multiplying Aᵢ...Aⱼ

Linear Time Selection Algorithm: How to achieve linear time?

• Think of the subproblem size • Must be lower by a constant factor • Number of comparisons in each recursive level lower by a constant factor

Runtimes of algorithms

• Polynomial time - 'efficient', depends on the size of n • Exponential time - not typically preferred • Not set in stone, in practice! • If polynomial time is established, degree of polynomial can be reduced - Hence the efficiency aspect! • Preference is for smallest exponent c even among polynomial time algorithms • Hiding constants in asymptotic notation -- O(n) may be worse than O(n^2) • In the real world, important algorithms are exponential in time • Verification of these algorithms can be in polynomial time • "P versus NP" problem - • For a given problem, is there a polynomial time algorithm for computing a solution? • It is not about verification

Major goal of algorithms

• Problems in the real world • Is a given number a prime? • Find a large prime number • Find the shortest route between two towns on a road map • Best move to make in a Chess game • Searching for relevant web pages that you are looking for • Finding a word in a text file

Sorting

• Process of arranging objects to put them in some logical order

Dynamic programming: 0/1 Knapsack Recursive formulation

• Pⱼ(y) - the optimal solution for KNAP(j, y) •Base cases: • P0(y) = 0 for all y ≥ 0 • Pᵢ(y) = - ∞ if y < 0 • To compute Pⱼ(y) - the optimal solution for KNAP(j, y) • Object aⱼ is in the knapsack Pⱼ(y) = Pⱼ₋₁(y-wⱼ) + pⱼ • Object aⱼ is not in the knapsack Pⱼ(y) = Pⱼ₋₁(y) • Optimal substructure property - Pⱼ(y) should contain optimal solution to its subproblems: Pⱼ₋₁(y) and Pⱼ₋₁(y-wⱼ)

Dynamic programming: Maxsum

• Recursive formulation • Write a recurrence • Compute Min[i] and Mout[i] recursively for i > 1 • Base cases: Mᶦⁿ[i] = A[1] Mᵒᵘᵗ[i] = - ∞ Mᶦⁿ[i] = max(Mᶦⁿ[i-1] + A[i], A[i]) Mᵒᵘᵗ[i] = M[i-1] • This formulation is correct intuitively Case 1: Include A[i]. Solution Min[i-1] will include A[i-1] ensuring contiguity of elements Case 2: Exclude A[i]. The optimal solution for the subarray A[1 . . . i] à M[i] will be the optimal solution to the subarray A[1 . . . i − 1] àM[i − 1] • Total number of subproblems, K • Worst-case time to solve each subproblem, T • Upper bound on total time of DP = O(KT) • Overall time: Σ i = 1 to K (Tᵢ) • Total runtime: O(n) • Recursive formulation can be turned into • Iterative code • Recursive code • What should you watch out for in these implementations?

(Backtracking method) n-Queens Problem: Given an n × n chessboard, place n queens so that no two queens can attack each other by being on the same row, column, or diagonal.

• Reduces the number of positions to be searched • Incrementally build candidate solutions - step by step • Evaluate partial solution to see if we can be extended by one more step without breaking the problem constraints • If extension is not possible, backtrack to replace the previous step with another step • Repeat • Examine only those positions where is one queen in each column and row • Queen 1 - place in any of the n rows of first column • Queen 2 - placed in any of the n-1 remaining rows of the second column (not in the same row as Queen 1) • Queen 3 - placed in any of the n-2 remaining rows of the third column • n x (n - 1) x ... x 2 x 1 = n! positions • n! is significantly smaller than (n^2, n) • For n=4, n! = 4! = 24. • n! still grows extremely fast for larger n • Exhaustive and backtracking search methods can take exponential time. Not fast!

Dynamic programming: 0/1 Knapsack We are given a set A = {a₁, a₂, . . . , aₙ} of n objects and a knapsack. Object aᵢ has an integer weight wᵢ and the knapsack has integer capacity m, i.e., the maximum weight of all items placed in the knapsack cannot exceed m. An object aᵢ is either placed into the knapsack or entirely omitted (no fractional part allowed), i.e., xᵢ = 1 if object is placed in the knapsack, and 0 otherwise. If the object aᵢ is placed, then a profit of pᵢ is earned. The objective is to maximize the total profit earned subject to the knapsack constraint.

• Resource allocation problem with financial constraints • Combinatorics • Complexity theory • Cryptography • Applied mathematics • Solution is a 0-1 vector x₁, x₂, ... xₙ • Maximizes total profit earned • Don't forget the knapsack constraint! • Identify the right subproblems • Use auxiliary variables • KNAP(j, y) - consider only the first j items with a knapsack capacity of y

Insertion Sort - Analysis

• Runtime depends on the initial order of the items in the input array • ~n²/4 comparisons and ~n²/4 exchanges for an array of length n • Worst case: ~n²/2 comparisons and ~n²/2 exchanges • Best case: n-1 comparisons and 0 exchanges

Search Problem

• Search is fundamental for all kinds of problems • A problem is a search problem if we are searching for elements that satisfy a desired property/properties from an appropriate set S, the search space • Brute-force or exhaustive search - examine each element of S and see if it satisfies the desired property • Costly if S is large.

What is an Algorithm?

• Sequence of instructions which are • Precise - unambiguous and mathematically defined • Terminate - must end after a finite number of steps • Two aspects important for every algorithm • Efficiency - uses few resources (time, space) • Mathematical analysis - to ensure precise bounds

Shellsort

• Simple extension of the Insertion Sort algorithm

Primality Checking - Speed

• Slow algorithm, if n is prime • The for loop runs n-2 times • Run time is approximately proportional to the magnitude of the input number

Sorting Goals and Rules

• Sort any type of data • Each item in the array contains a key • Rearrange to order the keys in some ordering rule • Each entry's key is no smaller than the key in the entry with a lower index • Each entry's key is no larger than the key in the entry with a larger index

Overlapping Subproblems

• Space of subproblems must be small • A recursive algorithm for the problem solves the same subproblem over and over • Solve each subproblem once and store the solution in a table • Look up when needed (constant time operation)

Greedy Algorithm

• Special case of dynamic programming • Subproblems in DP • Decomposition produces only one subproblem • Results in an efficient algorithm • Key - choose the correct subproblem to solve • Non-trivial to show sufficiency in the chosen subproblem vs. all other subproblems • Problems have a special structure • Works in stages • Best/greedy decision made in each stage • Does not solve lots of optimization problems • Efficient and simple, if a solution is found

Dynamic programming - Maxsum: Given an array A of size n containing positive and negative integers, determine indices i and j, 1 ≤ i ≤ j ≤ n, such that A[i] + A[i + 1] + · · · + A[j] is a maximum.

• Understand the problem • Input, output and any constraints that need to be considered • For the Maxsum problem • Input - an array of values • Output - a contiguous subarray such that the sum is maximized • How many subarrays exist? • What are the constraints? • Focus on the value of the solution • Instead of focusing on the strategy to find the solution • For the Maxsum problem, focus on • Finding the maximum sum value • Not on finding the subarray that gives the maximum sum • Formulate the subproblems • Better to have smaller total number of subproblems. • Why? • Introduce auxiliary variables • Let M[i] be the solution to the max-sum problem considering only the first i elements of the array A • Two subproblems • Let Mᶦⁿ[i] be the solution to the maxsum problem considering first i elements of array A: A[1, . . . , i] • This solution includes the last element A[i] in the maxsum • Let Mᵒᵘᵗ[i] be the solution to the maxsum problem considering first i elements of array A: A[1, . . . , i] • This solution does not include the last element A[i] in the maxsum • Total number of subproblems: O(n)

Problem Solving Process

• Understanding - What are the input and output for a given problem? • Solving the problem - Construct an algorithm • How to solve smaller instances of a given problem • Break problems into smaller subproblems and combine the answers • Write the description of the algorithm in English • Write the algorithm in pseudocode • Analyze - the correctness and the performance of the algorithm • Mathematics behind the problem • Argue that the algorithm is correct for every possible input • Correctness is critical than performance. Why? • Running time of the algorithm - time complexity analysis • Memory usage of the algorithm - space complexity analysis • Implementation - Convert the pseudocode into a computer program

Insertion Sort - Summary

• Very good for partially sorted arrays and tiny arrays • Plays an important role in advanced sorting algorithms

Primality Checking - Problem solving

• What is a prime number? • Given a number n, check if it is between 2 and n-1 and divides by n • If any of these divides n, output False • If none them divides n, output True

Performance of QuickSort

• Worst case: O(n²) • Occurs when the partitioning results in subsets of sizes (n-1) and 0 • Average case and Best case: O(n log n) • Best case occurs when the pivot is the median element