Algorithmic complexity/Big-O/Asymptotic analysis
What is the worst-case complexity?
Worst-case complexity of an algorithm is the function defined by the max number of steps taken on any instance of size n.
What is an upper bound? With respect to f(n)
g(n) = O(f(n)) means C * f(n) is an upper bound on g(n) The definition implies a constant nₒ beyond which they are satisfied. We do not care about small values of n. The values must be greater than 0 for nₒ.
What is a tight bound? With respect to f(n)
g(n) = Θ(f(n)) means C1 * f(n) is an upper bound on g(n) and C2 * f(n) is a lower bound on g(n) C, C1, and C2 are all constants independent of n The definition implies a constant nₒ beyond which they are satisfied. We do not care about small values of n. The values must be greater than 0 for nₒ.
What is a lower bound? With respect to f(n)
g(n) = Ω(f(n)) means C * f(n) is a lower bound on g(n) The definition implies a constant nₒ beyond which they are satisfied. We do not care about small values of n. The values must be greater than 0 for nₒ.
What do the worst, best, and average case look at?
Each of the complexities defines a numerical function time vs size
True or False Big O notation does not ignore multiplicative constants in front of things but when both functions in a product are decreasing, both are important
FALSE Big O notation IGNORES multiplicative constants in front of things but when both functions in a product are INCREASING, both are important O(f(n)) * O(g(n)) -> O(f(n) * g(n)) Ω(f(n)) * Ω(g(n)) -> Ω(f(n) * g(n)) Θf(n)) * Θ(g(n)) -> Θ(f(n) * g(n))
True or False You do not want the lowest upper bound possible when analyzing an algorithm
FALSE You DO want the lowest upper bound possible
What is classic binary search?
Finds an element in an already sorted list It looks at the middle element of the array and asks "is the element i want greater, equal, or less than this?" if it's greater, then you know the element has to be on the right side of the array and you can only look at that in the future. This process is repeated with the smaller size array until the correct element is found Cuts the array size in half with each operation
What does classic binary search run in?
O(log n)
What does quick sort run on?
O(n log n)
What does bubble sort run on?
O(n^2) Each pass through of the outer loop will require to go through the entire list again. So one trip through a loop would work out as O(n), but you would have to go through the loop again for every single item in the inner loop O(n^2)
What is O(log n)
Occurs when data being used is decreased roughly by 50% each time through the algorithm (ex: binary search) - pretty fast as because as log n increases or n increases, n is going to increase much faster than log(n). Log(n) algorithms are very efficient because increasing the amount of data has little to no effect at some point early on because the amount of data is halved each time. Logarithmic
Do programs with a bigger Θ run slower or faster than programs with a smaller Θ?
Programs with a bigger Θ run SLOWER than programs with a smaller Θ.
For f(n) = Ω(g(n)) if there are positive constants nₒ and C such that to the right of nₒ, the value of f(n) always lies on or ____ C * g(n)
above
For f(n) = O(g(n)) if there are positive constants nₒ and C such that to the right of nₒ, the value of f(n) always lies on or ____ C * g(n)
below
For f(n) = Θ(g(n)) if there exist positive constants nₒ, C1, and C2 such that to the right of nₒ, the value of f(n) always lies ____ C1 * g(n) and C2 * g(n) inclusive
between
When does f(n) become Θ(g(n))?
f(n) is Θ(g(n)) if 1. f(n) is O(g(n)) 2. f(n) is Ω(g(n)) Both conditions must be met
Given a series of for loops that are sequential, does the slowest or fastest of them determine the asymptotic behavior of the program?
The slowest 2 nested loops followed by a single loop is asymptotically the same as the nested loop alone because the nested loops dominate the single loop.
What runs asymptotically slower than O(n)?
O(n^2) as the input size increases towards infinity, the O(n^2) runtime will eventually eclipse the O(n) algorithm for small input sizes O(n^2) might be fast, but not for larger ones
What time-complexity does selection sort run on?
O(n^2) inefficient with large lists
How many steps does each memory access take?
Each memory access takes exactly 1 step
How many computrons is equivalent to 1 cycle?
1 cycle = 1 computron
How many nested loops are in Θ(n^2)
2 nested loops - where each repeats exactly n times
Why does binary search run in O(log n)?
Doubling the size of the array increases the run-time by only one chunk of the code, thus operates in a logarithmic time O(log n)
What is the most important/useful complexity to look at?
Worst-case complexity
If the best and worst case runs at the same time, what does it run on?
Θ(1)
What is asymptotic behavior?
"Keeping the largest growing term" We're interested in the limit of function f as n tends to infinity.
What is arithmetic progression? What is the sum of it?
1 + 2 + 3 + 4 + ... + n = ? Because the sequence of partial sums fail to converge to a finite limit, the series does not have a sum
In f(n) = 6n + 4, what is the initialization constant?
4 is the initialization constant, but as n grows, the 4 becomes less relevant so the equation can be simplified to 6n
What is linear search?
A method for finding a target value within a list. It sequentially checks each element of the list for the target value until a match is found or until all the elements have been searched.
What is O(1)?
Algorithm set to run in constant time Code that executes in the same amount of time no matter how big the array is
What is O(n)?
Algorithm set to run in linear time The time to complete will be in direct proportion to the amount of data (ex: linear search) will have to look for exactly the item in an array
What is O(n^2)
Algorithm where the time it's complete will be proportional to the square of the amount of data (ex: bubble sort - things with nested iterations.) Try to avoid O(n^2) with large numbers Quadratic
What is an algorithm?
An algorithm are programs that perform just a computation.
What is a computron?
An elementary unit of computation flowing through your computer. 1 computron gets eaten every time you do an elementary step. Loops do not take 1 computron because it cycles. A subroutine sort might have a lot of hidden steps in the subroutine.
What is the least important/useful complexity to look at?
Average-case complexity Usually you don't really know what you mean by average in terms of what you will encounter (e.g. when discussing about averages of weight, are we talking about the average weight of the class or the average weight of the US or the average weight of the world?) average is too broad of a statement.
What is average-case complexity?
Average-case complexity of the algorithm is the function defined by an average number of steps taken on any instance of size n.
What is the best-case complexity?
Best-case complexity is the minimum or least number of steps taken on any instance of n.
What is a bubble sort?
Bubble sort, sometimes referred to as sinking sort, is a simple sorting algorithm that repeatedly steps through the list to be sorted, compares each pair of adjacent items and swaps them if they are in the wrong order.
What is quick sort?
Comparing and moving values very efficiently without shifting unlike other sorting algorithms. Thus values will only be compared once and will not be compared to each other over and over. Each comparison would reduce the final sorted list in half. The number of comparisons that you would have to do is going to equal out to log n!
What is complexity analysis?
Complexity analysis is a tool that allows us to explain how an algorithm behaves as the input grows larger.
How does the Random Access Machine model of computation work? (How many steps does a "simple" operation take?)
Each "simple" operation (+, -, =, if, call) takes 1 step. Loops and subroutine calls are not simple operations. They depend on the size of the data and the contents of a subroutine. "Sort" is not a single step operation.
What is selection sort?
Finds the minimum of our array, puts it at the end of a new array and removes it from the original array. Then it finds the minimum between the remaining values of our original array, appends that to our new array so that it now contains 2 elements and removes it from our original array. It continues this process until all items have been removed from the original and have been inserted into the new array, which means that the array has been sorted. ------------------------------------------------------ The selection sort algorithm sorts an array by repeatedly finding the minimum element (considering ascending order) from unsorted part and putting it at the beginning. The algorithm maintains two subarrays in a given array. 1) The subarray which is already sorted. 2) Remaining subarray which is unsorted. In every iteration of selection sort, the minimum element (considering ascending order) from the unsorted subarray is picked and moved to the sorted subarray.
What is big o notation?
How well a computer algorithm scales as the amount of data involved increases. Not always a measure of speed, but of how well an algorithm scales
What does the lower-case o denote?
If we found a complexity bound that is not tight (Θ), we can also use a lower-case o to denote that.
What is better to discuss than best, worst, and average complexities?
It's easier to talk about upper and lower bounds of the function. Asymptotic notation (O, Θ, Ω) are as well as we can practically deal with complexity functions. Best, worst, and average case complexities are difficult to deal with precisely because the details are very complicated.
If an algorithm is Θ(n), what is its tight complexity?
Its tight complexity is n. This algorithm is both O(n) and O(n^2). As the algorithm is Θ(n), the O(n) bound is a tight one, but the O(n^2) bound is not tight, so we can write that the algorithm is o(n^2) - which is "small o of n squared". Better if we find tight bounds because it gives more information about how the algorithm behaves.
What does linear search run on?
O(n)
By counting ____, simple programs can be analyzed.
Simple programs can be analyzed by counting the nested loops of the program. A single loop over n items yields f(n) = n. A loop within a loop yields f(n) = n^2. A loop within a loop within a loop yields f(n) = n^3.
True or False A program that is Θ(n^2) is also O(n^2), but a program that is O(n^2) may not be Θ(n^2)
TRUE
True or False Multiplication by a constant does not change the asymptotics
TRUE O(c*f(n)) -> O(f(n)) Ω(c*f(n)) -> Ω(f(n)) Θ(c*f(n)) -> Θ(f(n))
True or False If you are doing a binary search and get the number in the middle, you'd get it in constant time no matter ow big the rest of the array is?
TRUE Runs on Ω(1)
In f(n) = 6n +4, what is the constant multiplier?
The constant multiplier is 6 and can be ignored along with the initialization constant 4. Equation becomes f(n) = n
O(f(n)) * O(g(n)) -> O(f(n) * g(n)) What if I am doing f(n) things each of which took g(n) time, how much time will it take if I am doing it most f(n) things each of which is going to take it g(n) time? (What is the total amount of time going to be at most?)
The total amount of time is going to be at most f(n) * g(n)
Θ(1) is a ____ algorithm Θ(n) is a ____ algorithm Θ(n^2) is a ____ algorithm Θ(log(n)) is a ____ algorithm
constant-time linear quadratic logarithmic
What is the equation for a program that doesn't have any loops? f(n) = ?
f(n) = 1 because the number of instructions it needs is just a constant (unless it uses recursions).
What is the equation for a program with a single loop which goes from 1 to n? f(n) = ?
f(n) = n because it will do a constant number of instructions before the loop, a constant number of instructions after the loop, and a constant number of instructions within the loop which all run n times.
When does f(n) become O(g(n))?
f(n) is O(g(n)) If constant and nₒ (initial) f(n) ≥ Cg(n) for all n>nₒ
What is Ramanujan summation?
1 + 2 + 3 + 4 + ... = -1/12 The first 4 partial sums of the series 1 + 2 + 3 + 4 + ... The parabola is their smoothed asymptote; its y-intercept is -1/12