Complete Algorithms
Dynamic programming
- A DP algorithm is an exact method useful for solving certain problems with the right properties. - It must be possible to divide up the original problem into sub-problems that are essentially smaller versions of the problem. - DP solves every sub-problem once and saves the solution in a table.
DP vs Divide and conquer
- D&C solves problems by combining the solutions of the sub-problems. - D&C is applicable when sub-problems are independent. - DP is applicable when sub-problems are not independent - sub-problems share sub-problems.
Approximate methods
- Exact methods guarantee to find the best solution to a problem given enough time. - Approximate methods attempt to find an acceptable solution in a reasonable amount of time. They often rely on heuristics to help navigate through a search space quickly to find a reasonable solution.
Longest common subsequence
- If match, form a bridge and add one from diagonal. - If no match, compare top and left values and take the largest value. - Work in reverse to locate LCS using the table.
Worst case
- Longest number of steps required. - Useful for guaranteed performance.
Reducibility
- NP complete problems are characterised by the concept of reducibility. - Any given NP complete problem can be mapped to any other NP complete problem - Thus, if a polynomial tim algorithm exists for one, it exists for all.
Average case
- Number of steps for 'typical' case. - Most useful metric in practice. - Average over all possible inputs.
Capacitated vehicle routing problem (CVRP)
- Objective: Minimise the number of vehicles and miles travelled. - Each vehicle has a fixed capacity. Merge feasibility: 1. Overload of the vehicle. 2. Internal customers - Only customers that are first or last can be involved in merge operations. 3. Customers both in the same route - If both customers are next to the depot but are in the same route - Join the directly.
Greedy algorithms
- Often used to solve optimisation problems. - Always choose the best option available.
Multifragment
- Selects edges rather than nodes/cities. - No city is allowed more than 2 edges. - No cycles may be formed until all nodes/cities have been included in the tour.
Simulated annealing
- Simulated annealing algorithms emulate the slow cooling of heated metal - Cooling down a solution to a minimisation problem. - Simulated annealing first generates a random solution and then alters this solution slightly - Producing neighbourhood solutions in an attempt to improve the solution. - The algorithm initially accepts anything, as search progresses, the algorithm is less likely to accept poor neighbourhood solutions.
Best case
- Smallest number of steps required. - Not very useful.
Genetic algorithms
- The first stage of a genetic algorithm is to generate and store a population of candidate solutions to a problem. - Genetic operators are applied to 'parents' in order to produce 'offspring'. - The best known operators used are 'mutation' and 'crossover' Mutation: - The mutation operator makes small neighbourhood moves - the same way as simulated annealing. - Mutation is carried out on a single individual. Crossover: - More difficult to devise - They involve combining parts of two parents. - Offspring need to inherit similarities from parents that help with the objective that is trying to be optimised.
NP complete problems
- The status of these problems is unknown. - No polynomial time algorithm has yet to be found for a NP complete problem. - The travelling sales man problem and the 1/0 knapsack problem are examples of NP complete problems.
Two-opt for the TSP
- We start with a solution and try to improve it. - We start with a random tour, or one produced by a constructive method. - Two-opt simply breaks the tour in two places, inverts the intervening sub-tour, and then rejoins the inverted path into the tour. - If the new arrangement is shorter, the new arrangement will be kept.
Exhaustive search
1. Explores all possible paths and picks the best. 2. This can be achieved by extending either breadth or depth first search. 3. Instead of stopping when you reach the goal, you carry on to the end. 4. The size of the search space can be enormous - with a branching factor 'b' and a depth 'd'.
Branch and bound search
1. Form a queue of partial paths. Let the initial queue consist of zero-length. 2. Until the queue is empty or the goal has been reached, determine if the first path in the queue reaches the goal node. 2.1. If the first path reaches the goal node, do nothing. 2.2. If the first path does not reach the goal node: 2.2.1. Remove the first path from the front of the queue. 2.2.2. Generate all of the paths children by extending one step. 2.2.3. Add new paths to the queue. 2.2.4. Sort the queue by the sum of cost accumulated so far and the lower bound estimate of the cost remaining, with the least cost paths in front. 2.2.5. If two or more paths reach a common node, delete all those paths except for the one that reaches the common node with the minimum cost. 3. If the goal is located announce success, otherwise, announce failure.
Breadth first search
1. Form a queue of partial paths. Let the initial queue consist of zero-length. 2. Until the queue is empty or the goal has been reached, determine if the first path in the queue reaches the goal node. 2.1. If the path reaches the goal node, do nothing. 2.2. If the path does not reach the goal node: 2.2.1. Remove the first path from the front of the queue. 2.2.2. Generate all the paths children by extending one step. 2.2.3. Add new paths to the back of the queue. 3. If the goal is located announce success, otherwise, announce failure.
Depth first search
1. Form a stack of partial paths. Let the initial stack consist of zero-length. 2. Until the stack is empty or the goal has been reached, determine if the first path in the stack reaches the goal node. 2.1. If the path reaches the goal node, do nothing. 2.2. If the path does not reach the goal node: 2.2.1. Remove the first path from the front of the stack. 2.2.2. Generate all the paths children by extending one step. 2.2.3. Add new paths to the front of the stack. 3. If the goal is located announce success, otherwise, announce failure.
Hillclimbing - simple local search
1. Pick a solution from the search space and evaluate it. Call this the current solution. 2. Choose a solution in the neighbourhood of the current one. 3. If the new solution is better than the current one, keep it and call it the current solution. 4. Return to step 2 until some stopping criterion applies.
Polynomial time algorithm
A polynomial time algorithm has a worst case run time of O(nᵏ) for some constant k.
Factorial function
Algorithm Factorial(n): Input: An integer, n. Output: n! if n = 1 then return (1) else return(n * Factorial(n-1))
Fibonacci algorithm
Algorithm Fibonacci(n): Input: An integer, n. Output: nᵗʰ fibonacci number. if n = 0 then return (0) if n = 1 then return (1) f0 ← 0; f1 ← 1 for i ← 2 to n do f ← f1 + f0 f0 ← f1 f1 ← f return(f)
Sum of squares
Algorithm sumSquares(n): Input: An integer, n. Output: The sum of squares 1...n. if n = 1 then return (1) else return(n * n + sumSquares(n-1))
Kruskal's algorithm
Builds up the tree by adding the shortest legal edge (i.e. no cycles allowed). 1. Initialise an MST edge list - MST ←∅. 2. Make n sets each containing a single vertex from V. 3. Sort all of the edges (u, v) ϵ E by non-decreasing weight. 4. Consider if the next (first) edge (u, v) ϵ E, in non-decreasing order of weight. 5. If vertices u and v are different in sets: 5.1. Combine the two sets containing u and v. 5.2. Update the MST edge list, MST ← MST ∪ (u, v). 6. Repeat from 4 until all vertices are in the same set.
Counting Sort
Complexities: Best: O(n + k) Average: O(n + k) Worst: O(n + k)
Merge Sort
Complexities: Best: O(n log(n)) Average: O(n log(n)) Worst: O(n log(n))
Quick Sort
Complexities: Best: O(n log(n)) Average: O(n log(n)) Worst: O(n²)
Bubble Sort
Complexities: Best: O(n) Average: O(n²) Worst: O(n²)
Insertion Sort
Complexities: Best: O(n) Average: O(n²) Worst: O(n²)
Selection Sort
Complexities: Best: O(n²) Average: O(n²) Worst: O(n²)
Intractable
Computational problems that can be solved by computer algorithms - but not in polynomial time.
Undecidable
Computational problems that cannot be solved by any algorithm at all.
Approximate methods
Concentrate on achieving a reasonable solution in a reasonable amount of time when exact methods are not feasible.
Minimum spanning tree
Given a connected, undirected graph, a spanning tree of that graph is a subgraph that is a tree and connects all the vertices together.
Algorithm examples
Greedy algorithm (exact): Prim's/Kruskal's Algorithm Greedy algorithm (approximate): Nearest neighbour heuristic Divide and conquer algorithm: Merge sort Branch and bound algorithm: The knapsack problem Dynamic programming algorithm: Recursive fibonacci Meta-heuristic method: Simulated annealing
Hard constraints
Hard constraints must be adhered to: - We need to avoid clashes - No student should be scheduled to sit two exams at the same time. - Probably limited space - Number of seats available.
Nearest neighbour
Having chosen an arbitrary node, iteratively construct new links by adding the unvisited nodes closest to the previous node. O(n²).
Metaheuristic method
Includes simulated annealing, tabu search, and genetic algorithms.
Programming in the large
Involves teams of people and software engineering life cycles.
Exact methods
Produce the best solution to a problem. Where a solution exists.
Dynamic programming continued
Properties required of problem for formulating a DP solution: 1. Optimal substructure The optimal solution to the original problem contains optimal solutions to sub-problems. 2. Overlapping sub-problems DP revisits the same sub-problems. It saves time by storing the results and looking them up in a table, instead of recalculating them each time.
Heuristic method
Simple rules of thumb are used.
Soft constraints
Soft constraints are desirable, but more flexible: - Maximise the spread of exams for individual students. - Schedule exams with lots of students earlier, giving plenty ofn time for marking.
Divide and conquer algorithms
Solves problems by: 1. Dividing data into parts. 2. Finding sub-solutions to each part. 3. Constructing final answer from sub-solutions.
Prim's algorithm
Starts with a single arbitrary vertex and builds up the tree by adding the closest vertex to the current subtree at each stage. 1. Form a set V consisting of a single, arbitrary vertex. Initialise the set of MST edges, MST to the empty set. 2. Find the shortest edge connecting a vertex V to a vertex not in V. Update V and MST with the new vertex and edge respectively. 3. Repeat step 2 until V contains all vertices. 4. The edges in MST make up a MST.
0/1 Knapsack problem
Takes as valuable load a possible, but carry at most W kilograms in the bag. O(2ⁿ).
Programming in the small
The design, testing and documentation of programs that consist of a few lines to a few pages.
Floyd Warshall
This algorithm uses n x n matrix A to compute the lengths of the shortest paths using a dynamic programming technique. - Let A[i, j] = C[i, j] ∀(i, j) ϵ E (i ≠ j). - If (i, j) is not an edge set A[i, j] = infinity. - Let A[i, i] = 0. ∀i ϵ ∀. Algorithm Floyd Warshall initialise A[i, j] ← C[i, j] ∀ (i, j) ϵ E set A[i, j] ← ∞ ∀ (i, j) ≠ E set A[i, i] ← 0 ∀i ϵ V for k ← 1 to n do for i ← 1 to n do for j ← 1 to n do if A[i, k] + A[k, j] < A[i, j] then A[i, j] ← A[i, k] + A[k, j] {it is quicker to go from i to j via k}
Big O Notation
We use this to describe the relationship between the time, T(n), taken by a particular algorithm and the number of inputs, n. Rule of thumb: 1 < log(n) < n < n log(n) < n² < n³ < 2ⁿ < 3ⁿ < n!
