Round 1: Basics of algorithm analysis and data structures

Transcription

1 Round 1: Basics of algorithm analysis and data structures Tommi Junttila Aalto University School of Science Department of Computer Science CS-A1140 Data Structures and Algorithms Autumn 2017 Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

2 Material in Introduction to Algorithms, 3rd ed. (online via Aalto lib): Growth of functions: Chapter 3 Elementary Data Structures: Sections Dynamic tables: Section 17.4 Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

3 Describing algorithms In an abstract view, an algorithm can be seen as a method that transforms some input to output input algorithm output For instance, a sorting algorithm transforms an array of elements into another array that contains the same elements but in a sorted order In many cases, it is easier to describe an algorithm in natural language or in some form of pseudocode rather than in some specific programming language Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

4 As an example, recall the binary search algorithm introduced in the Programming 2 course Given a sorted array A of elements and an element e, it finds out whether the element occurs in the array In natural language, we may describe the algorithms as follows: 1 Initially, consider the whole array 2 If the array is empty, return false 3 If the middle element of the array is e, return true 4 Otherwise, if the middle element is greater than e, go to step 2 but suppose that the array is the subarray from the beginning to (but not including) the middle element 5 Otherwise, the middle element is less than e and we go to step 2 but suppose that the array is the subarray from (but not including) the middle element to the end of the array Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

5 In pseudocode, we may write something like // Search for the element e in the sorted array A binary-search(a, e): i search(a, e, 1, A.length) if A[i] = e: return true else: return false // A helper function: search for the index of e in the sub-array A[lo,...,hi] search(a, e, lo, hi): if hi lo: return lo // sub-array of size 1 or less? mid lo + (hi lo)/2 if A[mid] = e: return mid else if A[mid] < e: return search(a, e, mid + 1, hi) else: return search(a, e, lo, mid 1) Warning: in many cases, array indexing starts from 1 in pseudocode and in mathematics but from 0 in real programming languages Take this into account when implementing algorithms from the book, wikipedia etc Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

6 An implementation in Scala def binarysearch [ T <% Ordered [ T ] ] ( s : IndexedSeq [ T ], v : T ) : Boolean = { def i n n e r ( s t a r t : I n t, end : I n t ) : I n t = { i f (! ( s t a r t < end ) ) s t a r t else { val mid = s t a r t + ( ( end s t a r t ) / 2) val cmp = v compare s ( mid ) i f (cmp == 0) mid / / v == s ( mid ) else i f (cmp < 0) i n n e r ( s t a r t, mid 1) / / v < s ( mid ) else i n n e r ( mid+1, end ) / / v > s ( mid ) } } i f ( s. length == 0) false else s ( i n n e r ( 0, s. length 1)) == v } Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

7 Basics of Analysis of Algorithms Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

8 Resource and property types We want our algorithms to be efficient... in a broad sense of using the given resources efficiently Typical resources and properties of interest are running time memory parallelizability disk/network usage efficient use of cache (cache-oblivious algorithms)... Naturally, we may have to make trade-offs: an algorithm that uses less memory may use more time etc On this course, we will mainly focus on running time We wish to study the running time (or memory usage etc) of a program/algorithm as a function f(n) in the size n of the input or some some other interesting parameter Both n and f(n) are assumed to be non-negative and n is usually an integer Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

9 The O-notation We are usually interested in the asymptotic growth of functions Thus we use the big-o notation and its variants It is a mathematical tool that abstracts away Definition small inputs and constant factors cause by the clock speed, memory bus width etc We write f(n) = O(g(n)) if there are constants c,n 0 > 0 such that f(n) cg(n) for all n n 0. That is, f(n) grows at most as fast as g(n) (when we consider large anough values of n) Recall: f(n) and g(n) are non-negative Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

10 10n = O(n), 10n = O(n 2 ), and 10n = O(2 0.1n ) n 2 + 5n = O(n 2 ) and n 2 + 5n = O(2 0.1n ) but n 2 + 5n O(n) 2 0.1n O(n) and 2 0.1n O(n 2 ) Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

11 Some typical function families: O(1) constant O(log n) logarithmic O(n) linear O(n logn) linearithmic, n log n (O(logn!) = O(n logn)) O(n 2 ) quadratic O(n 3 ) cubic O(n k ) polynomial, k 0 O(c n ) exponential, c > 1 O(n!) factorial Constant, logarithmic, linear, linearithmic, quadratic, and cubic functions are all polynomial functions We use base-2 logarithms unless otherwise stated (but log c n = log 2 n log 2 c for any c > 1) Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

12 The worst-case running time of a program/algorithm is O(f(n)) if g(n) = O(f(n)) where the function g(n) is obtained by taking, for each n, the longest running time of any input of size n When we say a program/algorithm runs in time O(f(n)), we usually mean that its worst-case running time is O(f(n)) One may also consider best-case running time obtained by taking the smallest running times for each n average-case running time by taking the expected running times for each n over some set of average inputs Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

13 Example Consider a linear search algorithm that checks whether an element e is included in an unordered array A of n elements by comparing the array elements one-by-one with e. Assuming that array accesses and comparing two elements are constant time operations, the worst-case running time is O(n) as the element may not be included in the array so that all the n elements are compared to e, but the best-case running time is O(1) as e can be the first element in the array Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

14 Example: straightforward multiplication of square matrices Analyze the asymptotic running time of the multiplication method * First, initializing the result matrix takes O(n 2 ) steps as the n n matrix is allocated and initialized to 0 class Matrix ( val n : I n t ) { r e q u i r e ( n > 0, n must be p o s i t i v e ) val e n t r i e s = new Array [ Double ] ( n * n ) / * * Access elements by w r i t i n g m( i, j ) * / def apply ( row : I n t, column : I n t ) = { e n t r i e s ( row * n + column ) } / * * Set elements by w r i t i n g m( i, j ) = v * / def update ( row : I n t, c o l : I n t, val : Double ) { e n t r i e s ( row * n + c o l ) = val } / * * Returns the product of t h i s and t h a t * / } def * ( t h a t : M atrix ) : M atrix = { r e q u i r e ( n == t h a t. n ) val r e s u l t = new Matrix ( n ) for ( row < 0 u n t i l n ; column < 0 u n t i l n ) { var v = 0.0 for ( i < 0 u n t i l n ) v += this ( row, i ) * t h a t ( i, column ) r e s u l t ( row, column ) = v } r e s u l t } Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

15 Example: straightforward multiplication of square matrices The outer loop iterates row from 0 to n 1 The middle loop iterates col from 0 to n 1 The inner loop iterates i from 0 to n 1 Inside the inner loop, the multiplication and array accesses are constant time ops The running time of the * method is O(n 2 + n 3 ) = O(n 3 ) class Matrix ( val n : I n t ) { r e q u i r e ( n > 0, n must be p o s i t i v e ) val e n t r i e s = new Array [ Double ] ( n * n ) / * * Access elements by w r i t i n g m( i, j ) * / def apply ( row : I n t, column : I n t ) = { e n t r i e s ( row * n + column ) } / * * Set elements by w r i t i n g m( i, j ) = v * / def update ( row : I n t, c o l : I n t, val : Double ) { e n t r i e s ( row * n + c o l ) = val } / * * Returns the product of t h i s and t h a t * / } def * ( t h a t : M atrix ) : M atrix = { r e q u i r e ( n == t h a t. n ) val r e s u l t = new Matrix ( n ) for ( row < 0 u n t i l n ; column < 0 u n t i l n ) { var v = 0.0 for ( i < 0 u n t i l n ) v += this ( row, i ) * t h a t ( i, column ) r e s u l t ( row, column ) = v } r e s u l t } Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

16 Big theta, big omega, little-o and little omega In addition to O, one commonly also sees the following: f(n) = Ω(g(n)) if g(n) = O(f(n)) f grows at least as fast as g f(n) = Θ(g(n)) if f(n) = O(g(n)) and g(n) = O(f(n)) f and g grow at the same rate There are also little-o and little omega : Definition We write f(n) = o(g(n)) if for every constant c > 0 there is a constant n 0 > 0 such that f(n) < cg(n) for all n n 0. If f(n) = o(g(n)), f grows slower than g f(n) = ω(g(n)) if g(n) = o(f(n)), f grows faster than g Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

17 Recursion and Recurrences Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

18 For recursive programs, it can be more difficult to analyze the asymptotic running time directly from the (pseudo)code Example From the Programming 2 course, recall the binary search algorithm for finding elements in a sorted array, implemented below in Scala. def binarysearch [ T <% Ordered [ T ] ] ( s : IndexedSeq [ T ], v : T ) : Boolean = { def i n n e r ( s t a r t : I n t, end : I n t ) : I n t = { i f (! ( s t a r t < end ) ) s t a r t else { val mid = s t a r t + ( ( end s t a r t ) / 2) val cmp = v compare s ( mid ) i f (cmp == 0) mid / / v == s ( mid ) else i f (cmp < 0) i n n e r ( s t a r t, mid 1) / / v < s ( mid ) else i n n e r ( mid+1, end ) / / v > s ( mid ) } } i f ( s. length == 0) false else s ( i n n e r ( 0, s. length 1)) == v } Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

19 But in fact analyzing corresponding iterative versions may not be easier either... For instance, consider the iterative version of the binary search algorithm below def b i n a r y S e a r c h I t e r [ T <% Ordered [ T ] ] ( s : IndexedSeq [ T ], v : T ) : Boolean = { i f ( s. length == 0) return false var s t a r t = 0 var end = s. length 1 while ( s t a r t < end ) { val mid = s t a r t + ( ( end s t a r t ) / 2) val cmp = v compare s ( mid ) i f (cmp == 0) { s t a r t = mid ; end = mid} else i f (cmp < 0) end = mid 1 / / v < s ( mid ) else s t a r t = mid+1 / / v > s ( mid ) } } return s ( s t a r t ) == v Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

20 To obtain an asymptotic running time estimate for recursive programs, one can try to formulate the running time (or some other measure) T (n) on inputs of size n as a recurrence equation of form where T (n) =... n is some parameter usually depending on the (current) input size, and the right hand side recursively depends on the running time(s) T (n ) of some smaller values n and some other terms In addition, the base cases, usually T (0) or T (1), not depending on other T (n ) need to be given One then solves the equation somehow Note: T (n) is not the value that the function computes but the number of steps taken when running the algorithm on an input of size n Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

21 Example For binary search, the recursive part considers sub-arrays of size n (end - start + 1 in our implementation above). When n 1, the recursion terminates. n > 1, a comparison is made and the search either terminates (the element was found) or the same recursive function is called with a sub-array of size n/2 Assuming that array accesses and element comparisons are constant-time operations, we thus get the recurrence equations T (0) = T (1) = d and T (n) = c + T ( n/2 ) where d and c are some constants capturing the number of operations taken to compare the bound, comparison etc. Here T (n) is an upper bound for the running time (number of instructions executed) on arrays with n elements as we have omitted the early termination case when the element is found but assume that the recursion always runs until the base case of arrays of size 0 or 1. Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

22 Let s solve the recurrence equations T (0) = T (1) = d and T (n) = c + T ( n/2 ) In this case, we obtain an upper bound by considering inputs that are of length n = 2 k for some non-negative integer k. Now T (2 k ) = c + T (2 k 1 ) = c + (c + T (2 k 2 )) =... = kc + T (2 0 ) = ck + d As n = 2 k implying k = log 2 n, and c and d are constants, we get T (n) = T (2 k ) = ck + d = c log 2 n + d = O(log 2 n) When n is not a power of two, T (n) T (2 log 2 n ) = d + c log 2 n d + c(1 + log 2 n) = O(log 2 n) as T (n) is monotonic. Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

23 When using the O-notation, constants can be usually eliminated already in the beginning by replacing them with 1. In our binary search example, the recurrence is thus T (0) = T (1) = 1 and T (n) = 1 + T ( n/2 ). Solving this gives again T (n) = O(log 2 n). Sometimes one also writes T (0) = T (1) = O(1) and T (n) = O(1) + T ( n/2 ). Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

24 In addition to the number of instructions taken, we may sometimes be interested in some other aspects of the algorithm For instance, if we know that comparisons are very expensive, we may analyze how many of them are taken in a search or sorting algorithm Example For binary search, an asymptotic upper bound for the number of comparisons made on an input of elements n can be obtained from again the recurrence and thus T (n) = O(log 2 n). T (0) = T (1) = 1 and T (n) = T ( n/2 ) + 1 We ll see more complicated recurrences when analyzing sorting algorithms Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

25 Example: Exponentiation by squaring As a second example, consider an exponentiation algorithm computing x n mod 2 64 by using x n = k 1 i=0 x n i 2 i, where n k 1 n k 2...n 1 n 0 is the binary representation of n. For instance, x 71 = x 1 x 2 x 4 x 64 = x 20 x 21 x 22 x 26 The algorithm is called repeated squaring as it maintains and uses, starting from x 20 = x, the value x 2i+1 = x 2i x 2i. def pow( x : Long, n : Long ) : Long = { r e q u i r e ( n >= 0) i f ( n == 0) 1 else i f ( n == 1) x else i f ( ( n & 0x01 ) == 1) x * pow( x * x, n / 2 ) else pow( x * x, n / 2 ) } The time requirement with respect to the value n is T (0) = T (1) = 1 and T (n) = 1 + T (n/2), and thus T (n) = O(log 2 n). Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

26 Example Consider the subset sum problem Given a set S = {v 1,...,v n } of integers and a target integer t. Is there a subset S of S such that v S v = t? and a (very simple) recursive Scala program solving it: def subsetsum ( s : L i s t [ I n t ], t : I n t ) : Option [ L i s t [ I n t ] ] = { i f ( t == 0) return Some( N i l ) i f ( s. isempty ) return None val solnotincluded = subsetsum ( s. t a i l, t ) i f ( solnotincluded. nonempty ) return solnotincluded val s o l I n c l u d e d = subsetsum ( s. t a i l, t s. head ) i f ( s o l I n c l u d e d. nonempty ) return Some( s. head : : s o l I n c l u d e d. get ) return None } Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

27 Example: continues Let s form a recurrence for the running time of the recursive program with the parameter n being the size of the set. In the worst-case scenario, there is no solution and the recursion terminates only when the set is empty. In each call, the program makes some constant-time tests etc and then two recursive calls with a set that has one element removed Thus we get the recurrence T (0) = 1 and T (n) = 1 + T (n 1) + T (n 1) = 1 + 2T (n 1) Expanding the recurrence, we get T (n) = 1 + 2T (n 1) = 1 + 2(1 + 2T (n 2)) = (1 + 2T (n 3)) =... = n i=0 2i = 2 n+1 1 = Θ(2 n ) Thus the algorithm takes exponential time in the worst case In the (very rare) best case, the target equals to 0 and the program takes constant time to complete Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

28 Experimental analysis Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

29 In addition to mathematical analysis, we may also perform experimental performance analysis For instance, we may take implementations of two algorithms that perform the same task, run these on the same input set, and compare the running times Similarly, we may instrument the code to count all the comparison operators made and then run the algorithm on various inputs to see how many comparisons are made on average or in the worst case use some profiling tool (such as valgrind for C/C++/binary programs) to find out numbers of cache misses etc Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

30 Simple running time measurement As seen in the Programming 2 course, we can measure running times of functions quite easily 1 import java. lang. management.{ ManagementFactory, ThreadMXBean} package object t i m e r { val bean : ThreadMXBean = ManagementFactory. getthreadmxbean ( ) def getcputime = i f ( bean. iscurrentthreadcputimesupported ( ) ) bean. getcurrentthreadcputime ( ) else 0 def measurecputime [ T ] ( f : => T ) : ( T, Double ) = { val s t a r t = getcputime val r = f val end = getcputime ( r, ( end s t a r t ) / ) } def measurewallclocktime [ T ] ( f : => T ) : ( T, Double ) = { val s t a r t = System. nanotime val r = f val end = System. nanotime ( r, ( end s t a r t ) / ) } } 1 Wall clock times should be used for parallel programs as getcurrentthreadcputime measures the CPU time of only one thread Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

31 Unfortunately, things are not always that simple; consider the following code sorting twenty arrays of integers import timer. val a = new Array [ I n t ](10000) val rand = new scala. u t i l. Random ( ) val times = scala. c o l l e c t i o n. mutable. A r r a y B u f f e r [ Double ] ( ) for ( t e s t < 0 u n t i l 20) { for ( i < 0 u n t i l a. length ) a ( i ) = rand. n e x t I n t val (dummy, t ) = measurecputime { a. sorted } times += t } p r i n t l n ( times. mkstring (, ) ) which outputs , , , , , , The program measures the same code on similar inputs but the running times seem to become over ten times faster after some repetitions? Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

32 The above behaviour can be explained as follows Scala programs are compiled into Java bytecode Java bytecode is executed in a Java virtual machine (JVM) The execution may start in interpreted mode Some parts of the program can then be just-in-time compiled to native machine code during the execution of the program The compiled code may be globally optimized during the execution Garbage collection may happen during the execution when measuring running times of programs with shortish running times, be careful when interpreting the results for programs with larger running times this is usually not so big problem Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

33 ScalaMeter As we saw, reliably benchmarking programs written in interpreted and virtual machine based languages (such as Java and Scala) can be challenging, especially if the running times are small In this course, we ll sometimes use the ScalaMeter tool ( It tries to take the JVM-related issues into account by, among other things, warming up the virtual machine and using statistical analysis techniques See, e.g., the documentation of the tool and the article A. Georges et al: Statistically rigorous java performance evaluation, Proc. OOPSLA 2007, pp Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

34 Some elementary data structures Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

35 We review some data structures from earlier courses, and... see some other simple linear data structures Our presentation is not limited to Scala and also studies the basic operations that the data structure can handle efficiently Such a collection of operations (together with the semantics of the operations) is also called an abstract data type Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

36 Arrays A built-in type in most programming languages (Java virtual machine, C/C++ etc) A sequence of n elements Constant time indexed access to elements (read and write) Searching for an element takes Θ(n) time in the worst case (if the array is not sorted for binary search etc) The size n is fixed at the time of creation when a continuous block of memory is allocated for the array Elements cannot be appended at the beginning or end; if this is desired, a new array with bigger size must be allocated and the array contents copied there together with the new element, a Θ(n) time operation Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

37 In Java and Scala, elements in arrays are either primitive types or references to objects residing in their own memory locations Example: object diagrams for an array of integers (left) and for an array of integer pair objects (right) The third entry in the array on the right contains a null-reference not pointing to any object Some other languages, such as C and C++, allow array elements to be structured objects Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

38 Resizable arrays Also called dynamic tables etc, ArrayBuffer in Scala, vector in the C++ standard library Allow amortized constant-time insertion at the end of the array The idea: allocate an array a with extra space and maintain the capacity c, i.e. the size, of the array a, 2 and the size s of the resizable array, i.e. the number of elements actually used in the underlying array a When an element is appended at the end of the resizable array, simply write it at the index s of a if s < c and increase s by one, or when s = c, 1 allocate a new array of c α elements, where α > 1 is an expansion factor constant (usually 2), 2 copy the elements from the old array to the beginning of the new array, and 3 set c = c α, insert the new element at index s in the new array and increase s by one 2 in Java/Scala, this is already included in the low-level array but in C/C++, for instance, raw arrays do not carry any additional information Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

39 Example: Inserting an element into a full resizable array Inserting the element 21 to the full resizable array triggers expansion and results in the resizable array The old array will be garbage-collected later (or should be freed by the insertion operation in non-garbage collecting languages such as C/C++). Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

40 Example: appending elements to an ArrayBuffer, from the source code of ArrayBuffer and ResizableArray / * * Appends a s i n g l e element to t h i s b u f f e r and r e t u r n s * the i d e n t i t y of the b u f f e r. I t takes constant amortized time. * / def +=(elem : A ) : this. type = { ensuresize ( size0 + 1) array ( size0 ) = elem. asinstanceof [ AnyRef ] size0 += 1 this } / * * Ensure t h a t the i n t e r n a l array has at l e a s t n c e l l s. protected def ensuresize ( n : I n t ) { * / val arraylength : Long = array. length / / Use a Long to prevent overflows i f ( n > arraylength ) { var newsize : Long = arraylength while ( n > newsize ) * 2 newsize = newsize * 2 / / Clamp newsize to I n t. MaxValue i f ( newsize > I n t. MaxValue ) newsize = I n t. MaxValue val newarray : Array [ AnyRef ] = new Array ( newsize. t o I n t ) scala. compat. Platform. arraycopy ( array, 0, newarray, 0, size0 ) array = newarray } } Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

41 Resizing the underlying array multiplies its capacity with the constant α and thus the new array sizes grow exponentially Isn t exponential growth bad? Not really in this case: if we only insert elements, then the array is always at least 1 α-full. If α = 2, we thus only waste at most the same amount of memory that we actually use for storing the elements Compare this to linked lists in which each list entry contains at least the element, and the reference to the next entry in the list 3 linked lists also waste the same amount of memory 3 in Java/Scala, each entry also has some object type information Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

42 Why do we want to have exponential growth, why not just add some non-neglible fixed number, say 1024, elements when resizing? Exponential addition allows us to have amortized constant-time insertion of elements That is, if we insert n elements in a resizable array, then some insertions, i.e. those on which the resizing occurs, are quite costly as memory allocation and copying must be done, but the cumulative cost amortized to each insertion is constant. Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

43 Lets study the number of accesses done in each insertion (this is in linear correspondence to the running time) Suppose that α = 2, we have inserted n elements in an empty array, and that array resizing occurs when we insert the n + 1th element The number of extra array accesses when copying elements from an old array to the current new one in this and all the previous resizings is captured by the recurrence T (d) = 0 and T (n) = n + T (n/2), where d is the initial size of the array. Thus T (n) n + T (n/2) n + n/2 + n/4 + n/8... 2n Adding the 1 normal array access per insertion, we get the amortized number of array accesses per insertion: 2n+n = 3 n Similarly, for an arbitrary expansion factor α > 1, the extra cost is at most n i=0 1 = n 1 = n α = O(n) and thus the amortized cost per α i 1 α 1 α 1 insertion is a constant as well For α = 1.1, the amortized cost per insertion is at most = 12 array accesses. Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

44 For the expansion factor of 2, we can graphically illustrate the array accesses done when inserting 9 elements in a resizable array that had initially the capacity of 1 as follows: The blue dots are regular accesses when writing the new element, red ones are copy accesses from the previous expansions and the pink ones are the ones of the latest expansion. If we collapse the red towers right and the pink tower left, we get and see that each insert has at most 3 dots amortized to them. Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

45 Should one contract the resizable array when it only contains few elements compared to its capacity? Yes, we can do this and have amortized constant-time insert and remove operations. But the contraction triggering load factor, i.e. the fraction s of entries c actually used, should less than the multiplicative inverse of the expansion factor. That is, if we double the array capacity at some point and then immediately remove an element, we should not halve the array capacity immediately. If the expansion factor is 2.0 and contraction trigger is 0.25, we get amortized constant-time insert and remove operations, see Section 17.4 in Introduction to Algorithms, 3rd ed. (online via Aalto lib) Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

46 Linked lists Recall from the recursion round of the Programming 2 course: immutable linked lists Lists are not modified once created, entries can be shared between different lists Adding elements at the beginning is constant-time, linear-time at the end Example: Immutable linked lists An immutable linked list data structure: Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

47 Linked lists Recall from the recursion round of the Programming 2 course: immutable linked lists Lists are not modified once created, entries can be shared between different lists Adding elements at the beginning is constant-time, linear-time at the end Example: Immutable linked lists The list k is the tail of the list l: Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

48 Linked lists Recall from the recursion round of the Programming 2 course: immutable linked lists Lists are not modified once created, entries can be shared between different lists Adding elements at the beginning is constant-time, linear-time at the end Example: Immutable linked lists The list m obtained from l by adding 11 after 21: Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

49 One can also make a mutable version of linked lists The entries are not shared between lists One can insert elements in the list by just allocating new entry objects and manipulating references If the reference to the last entry is maintained, adding elements at the end of the list is also a constant-time operation The number of elements in the list can also be maintained so that the list length can be queried in constant time Example: Mutable linked lists A mutable linked list data structure: Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

50 One can also make a mutable version of linked lists The entries are not shared between lists One can insert elements in the list by just allocating new entry objects and manipulating references If the reference to the last entry is maintained, adding elements at the end of the list is also a constant-time operation The number of elements in the list can also be maintained so that the list length can be queried in constant time Example: Mutable linked lists The same list after adding the value 11 after the value 21: Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

51 A doubly-linked variant of mutable linked lists adds, to each entry, a reference pointing to the previous entry Such list can be traversed in reverse order efficiently Example: Mutable doubly-linked lists A doubly-linked variant of the list in the previous example: Given a pointer to a list entry anywhere in the list, removing it or adding an element before/after it is also easy due to forward and backwards pointers Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

52 Mutable linked lists in Scala: ListBuffer (source code) The older versions LinkedList and DoubleLinkedList are now deprecated list in C++ implements doubly linked lists Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

53 Queues Queues usually refer to first-in-first-out (FIFO) queues The basic operations are enqueue, which puts an element at the end of the queue, and dequeue, which removes the first element in the queue Queues are easy to implement with doubly-linked lists so that both enqueue and dequeue are constant-time operations In Scala, mutable Queue is implemented with linked lists (the internal MutableList class) In the C++ standard library, the queue class calls enqueue push and dequeue pop Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

54 Stacks Stacks are like queues but the use last-in-first-out The basic operations are push that puts an element on the top of the stack, and pop that removes the topmost element in the stack Stacks are easy to implement either with linked lists or with resizable arrays In Scala, both immutable and mutable stacks are implemented with linked lists In C++: the stack class Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51

55 More data structures ahead in the following rounds! Tommi Junttila (Aalto University) Round 1 CS-A1140 / Autumn / 51