Time and space behaviour Chapter 20
Algorithmic complexity How many steps does it take to execute an algorithm given an input of size n?
Complexity of functions Consider: f n = 2 n 2 + 4 n + 13 This function has three components: a constant, 13 a term 4 n a term 2 n 2 As the value of n increases: the constant component is unchanged the 4 n component grows linearly the 2 n 2 component grows quadratically
Complexity of functions: big-oh O For large n, the quadratic term dominates in f. So, we say that f is of order n 2 and we write O(n 2 ) A function f::integer->integer is O(g) if there are positive integers m and d such that for all n m: f n d (g n) i.e., that f is bounded above by g: for sufficiently large n the value of f is no larger than a multiple of the function g. e.g., f from the previous slide is O(n 2 ), since for n 1: 2 n 2 + 4 n + 13 2 n 2 + 4 n 2 + 13 n 2 19 n 2
Complexity of functions: big-theta Θ For a tight bound, a function f is Θ(g) when: both f is O(g) and g is O(f) When f is O(g) but g is not O(f) we write f g. Also, when f is Θ(g) we write f g. n 0 n 1 n 2 n k 2 n n 0 log n n 1 n (log n) n 2
28 y=10 x y=10x 2 24 20 y=x 16 12 y=10 8 4 0 0.25 0.5 0.75 1 1.25 1.5 1.75 2
y=x 2 y=x log x y=x 1 10 7.5 5 2.5 y=log x y=x 0 0 2.5 5 7.5 10 12.5 15
Bisection Given a list of length n, how many times can it be bisected before all the pieces are of length one? After p cuts the length of each piece is n/(2 p ). We are looking for p such that ( n/(2 p ) ) 1 and ( n/(2 p-1 ) ) > 1, i.e. n 2 p and n > 2 p-1 : 2 p n > 2 p-1 p log 2 n > p-1 The function giving the number of steps in terms of the length n is Θ(log 2 n)
We are looking for p such: p log 2 n > p-1 log 2 n > p-1 => p < 1 + log 2 n p is a function of n (the length of a list) f(n) < 1 + log 2 n A function f::integer->integer is O(g) if there are positive integers m and d such that for all n m: f n d (g n) There exists m (e.g.: 10) and d (e.g.: 3) such that f(n) d * log 2 n for all n m For example f(10) < 1 + log 2 (10) <= 3 * log 2 (10) f(11) < 1 + log 2 (11) <= 3 * log 2 (11) Therefore, F(n) is O(log 2 n) (the upper bound). Similarly we can prove the lower bound, the other direction usung the other inequality. Hence, Θ(log 2 n).
Size of a balanced binary tree A tree is balanced if all of its branches are the same length. Given a balanced binary tree whose branches are length b, how many nodes are there in the tree? 1 + 2 + 4 + + 2 k-1 +... + 2 b = 2 b+1 1 Thus, the size of the tree is Θ(2 b ) in the length of the branches. Conversely, a balanced tree will have branches of length Θ(log 2 n) in the size of the tree.
Apples Given an apple a day for n days we will end up with n apples. Given n apples a day for n days we will end up with n 2 apples. Given 1 apple on the first day, 2 apples on the second day, etc., how many apples do we end up with? That is, what is the sum of the list [1..n]? 1 + 2 + 3 + + (n-1) + n = n + (n-1) + (n-2) + + 2 + 1 = ((n+1) + (n+1) + (n+1) + + (n+1) + (n+1)) / 2 = n (n+1) / 2 which is quadratic, Θ(n 2 ).
Measuring complexity Time taken to compute a result is given by the number of steps in a calculation. Space necessary for the computation. During calculation the expression being calculate grows and shrinks we need space to hold the largest expression. This is called the residency of the calculation, or its space complexity. Total space used by a computation, reflecting not just the size of the expression but the sizes of the values as well.
Factorial fac :: Integer -> Integer fac 0 = 1 fac n = n * fac (n-1) fac n n * fac (n-1) n * ((n-1) *... * (2 * (1 * 1))... ) n * ((n-1) *... * (2 * 1)... ) n * ((n-1) *... * 2... ) n! This takes 2n+1 steps and the largest expression contains n multiplication symbols. So, time and space is linear, Θ(n)
Insertion sort isort :: Ord a => [a] -> [a] isort [] = [] isort (x:xs) = ins x (isort xs) ins :: Ord a => a -> [a] -> [a] ins x [] = [x] ins x (y:ys) (x<=y) = x:y:ys otherwise = y:ins x ys
isort [] = [] isort (x:xs) = ins x (isort xs) isort [a 1, a 2,..., a n-1, a n ] ins a 1 (isort [a 2,..., a n-1, a n ])... ins a 1 (ins a 2 (... (ins a n-1 (ins a n []))... )) So, n steps followed by n invocations of ins.
ins x [] = [x] ins x (y:ys) (x<=y) = x:y:ys otherwise = y:ins x ys Assume[a 1,,..., a n ] is sorted. ins a [a 1, a 2,..., a n-1, a n ] Best case: a<=a 1, takes one step Worst case: a>a n, takes n steps Average case: takes n/2 steps
So for isort: ins a 1 (ins a 2 (... (ins a n-1 (ins a n []))... )) Best case: each ins takes one step, so n more steps, so 2n steps overall which is O(n) Worst case: first ins 1 step, second 2,, so overall O(n 2 ) Average case: ins s will take 1/2 + 2/2 + + (n-1)/2 + n/2 steps, so overall O(n 2 ) isort takes quadratic time in most cases, linear for (almost) sorted lists. Space usage is linear in all cases.
++ [a 1, a 2,..., a n-1, a n ] ++ x a 1 : ([a 2,..., a n-1, a n ] ++ x) a 1 : (a 2 : ([a 3,..., a n-1, a n ] ++ x)) n-3 steps a 1 : (a 2 :... : (a n :x)) So, the time taken is linear in the length of the first list.
Quick sort qsort :: Ord a => [a] -> [a] qsort [] = [] qsort (x:xs) = qsort [z z<-xs,z<=x] ++ [x] ++ qsort [z z<-xs,z>x] When the list is sorted and without duplicates the calculation goes: qsort [a 1, a 2,..., a n-1, a n ] n steps [] ++ [a 1 ] ++ qsort [a 2,..., a n-1, a n ] n-1 steps a 1 : ([] ++ [a 2 ] ++ qsort [a 3,..., a n-1, a n ]) n-2 steps a 1 : (a 2 : (a 3 :... a n : [])) [a 1, a 2,..., a n-1, a n ] So, overall 1+2+...+n steps, so qsort is quadratic for sorted lists: O(n 2 )
qsort :: Ord a => [a] -> [a] qsort [] = [] qsort (x:xs) = qsort [z z<-xs,z<=x] ++ [x] ++ qsort [z z<-xs,z>x] In the average case: qsort [a 1, a 2,..., a n-1, a n ] qsort [b 1,..., b n/2 ] ++ [a 1 ] ++ qsort [c 1,..., c n/2 ] Forming the two sublists takes O(n 1 ) steps, and O(n 1 ) steps to join the results. It takes O(log 2 n) bisections to obtain singleton lists. So qsort is on average O(n(log 2 n)).
Logarithmic behaviour is characteristic of divide-and-conquer algorithms: we split the problem into two smaller problems, solve them and recombine the results. They reach their base case in O(log 2 n) rather than O(n) steps.
Lazy evaluation arguments to functions are evaluated only when this is necessary for evaluation to continue an argument is not necessarily evaluated fully: only the parts that are needed are examined an argument is evaluated at most once (expressions are replaced by graphs and calculation is done over the graphs) evaluation order is from the outside in (for nested functions, e.g.: f 1 e 1 (f 2 e 2 5)) and from left to right (e.g.: f 1 e 1 + f 2 e 2 ).
Space behaviour: lazy evaluation Rule of thumb for space is the size of the largest expression produced during evaluation. This is accurate for computing numbers or Booleans, but not for data structures. Lazy evaluation means partial results are outputted, and discarded, once computed.
Space behaviour: lazy evaluation Consider: [m.. n] n >= m = m:[m+1.. n] otherwise = [] [1..n]?? n >= 1 1:[1+1.. n]?? n >= 1+1?? n >= 2 1:[2.. n] 1:2:[2+1.. n] 1:2:3: :n:[] The underlined pieces are printed and discarded as soon as possible. To measure space complexity we look at the non-underlined part (the residual evaluation), which is of constant size. So, space complexity is O(n 0 ).
Space behaviour: where clauses exam1 = [1..n] ++ [1..n] Takes time O(n 1 ) and space O(n 0 ) but calculates [1..n] twice! exam2 = list ++ list where list = [1..n] After evaluating list, the whole of the list is stored, giving space complexity O(n 1 )
Space behaviour: where clauses exam3 = [1..n] ++ [last [1..n]] exam4 = list ++ [last list] where list = [1..n] Space is O(n 0 ) Space is O(n 1 ) This is a space leak, since we only need one element of list.
Space behaviour: where clauses Avoiding redundant computation is (usually) always sensible, but it comes at the cost of space.
Saving space? fac 0 = 1 fac n = n * fac (n-1) Has O(n 1 ) space complexity from: n * ((n-1) *... * (2 * (1 * 1))... ) before it is evaluated. Alternative is to perform multiplication as we go: newfac :: Integer -> Integer newfac n = afac n 1 afac :: Integer -> Integer -> Integer afac 0 p = p afac n p = afac (n-1) (p*n)
newfac :: Integer -> Integer newfac n = afac n 1 afac :: Integer -> Integer -> Integer afac 0 p = p afac n p = afac (n-1) (p*n) newfac n afac n 1 afac (n-1) (1*n)?? (n-1) == 0 False afac (n-2) (1*n*(n-1)) afac 0 (1*n*(n-1)*(n-2)* *2*1) (1*n*(n-1)*(n-2)* *2*1) Still forms a large unevaluated expression, since its value is not needed until the end.
Consider: afac 0 p = p afac n p (p==p) = afac (n-1) (p*n) The guard test forces evaluation of the intermediate multiplications. This version has constant space behaviour. afac 4 1 afac (4-1) (1*4)?? (4-1) == 0 False?? (1*4) == (1*4)?? 4 == 4 True afac (3-1) (4*3)?? (3-1) == 0 False?? (4*3) == (4*3)?? 12 == 12 True afac (2-1) (12*2) afac 0 (24*1) (24*1) 24
Strictness A function is strict in an argument if the result is undefined whenever the argument is undefined. Examples: (+) is strict in both arguments (&&) is strict in only its first argument: True && x = x False && x = False A function that is not strict in an argument is said to be non-strict or lazy in that argument.