Algorithms and Data Structures

Transcription

1 Algorithms and Data Structures Markus Würzl, Vienna University of Technology Solving Interesting Problems. Create data structures & algorithms to solve problems. 2. Prove algorithms work. Buggy algorithms are worthless! 3. Examine properties of algorithms. Simplicity, running time, space needed, When Does an Algorithm Solve a Problem? A problem instance is a specific task to be solved on a specific input. A problem is a collection of similar problem instances. An algorithm is correct for a given problem instance if it produces the desired output for this instance. An algorithm is correct for a problem P (solves problem P) if it is correct for all instances of P.

2 What Kinds of Problems? Decision problems Given input, does it have a certain property? Is an integer prime? Is a graph planar? (Can be drawn on a plane without overlapping edges?) What Kinds of Problems? Function problems Given input, compute the unique solution. Compute the product of two matrices. Sort an array. What Kinds of Problems? Search problems Given input, compute any one of a set of solutions. Find the factors of an integer. Find shortest path between two points in graph. Find shortest path between each pair of points in graph. Find longest path between two points in graph.

3 Properties of Algorithms Time efficiency As a function of its input size, how long does it take? Space efficiency As a function of its input size, how much additional space does it use? Analysis of Computational Resources Computational resources = CPU time (running time), memory usage (space), messages sent along the network, Resource consumption differs depending on the size of the input (length of text, number of records to be sorted, ) Specify resource consumption as a function of the inputs size. Resource consumption may even differ greatly for inputs of the same size, depending on their structure (highly unsorted input, almost sorted input, ) Properties of Algorithms Simplicity Informal notion Pragmatic benefits How easy to code correctly? Possibly low constant factors in efficiency Idealistic benefit Mathematical elegance

4 Best, Average, and Worst Case The best-case resource consumption is the minimum resource consumption over all possible inputs of a given size. The worst-case resource consumption is the maximum resource consumption over all inputs of a given size. The average-case resource consumption is the average resource consumption over all possible inputs of a given size. Things to Remember Algorithm: Description of a procedure to solve a given problem Model of computation: Specifies permissible operations and their costs An algorithm is correct = produces correct answer for every possible instance of the problem Resource consumption = function of input (size) Best-case, worst-case, average-case analysis Extended Example: Multiplication Illustrates some of course s interesting ideas. Just an overview not too much detail yet.? How to multiply two integers: x y. Common & familiar. Simple?? Suggested algorithms & efficiency??

5 Multiplication Algorithm It s a single basic machine operation, O() time.? Problem with this answer?? Realistic only if we have bounded-sized integers. Instead, assume unbounded-sized integers. E.g., as in Scheme. Explicitly include this assumption. Multiplication Algorithm 2 Repeated addition: x y = x+ +x = y+ +y Back to multiplication: O(n max(x,y)) = O(n 2 n ) y copies x copies How long for addition? Grade-school algorithm takes time number length. Define: n = log 2 (max(x,y)) Logarithm base irrelevant, since O(log c x) = O(log c x) O(n) Multiplication Algorithm 3 Grade-school long-hand algorithm. n multiplications of (x by bit of y) + n additions of the resulting products.? Total? O(n n + n n) = O(n 2 ) Much better. Also approximates hardware algorithms.? x= 38 y=

6 Multiplication Algorithm 4: Part Karatsuba s algorithm, 962 Break the bitstrings of x,y into halves. x = a 2 n/2 + b = a << n/2 + b y = c 2 n/2 + d = c << n/2 + d x=45 y=28 xy = ac << n + (ad+bc) << n/2 + bd Compute the 4 subproducts recursively. Example of a "divide-and-conquer" algorithm. a=5 b= c=3 d=4 Multiplication Algorithm 4: Part Form recurrence equation: How long does this take? k n = T( n) = n 4 T + k n n > 2 Solve recurrence equation. How? Discussed next week. T(n) = O(n 2 ) k = arbitrary constant to fill in for the details 4 recursive subproducts + additions & shifts No better than previous algorithm. Multiplication Algorithm 4: Part 2 Previous: xy = ac << n + (ad+bc) << n/2 + bd Regroup: u = (a+b) (c+d) v = ac w = bd xy = v << n + (u-v-w) << n/2 + w x=45 y=28 a=5 b= c=3 d=4 Only 3 subproducts!

7 Multiplication Algorithm 4: Part 2 How long does this take? k n = T ( n) = n 3 T + k n n > 2 k = a new, larger constant log 2 3 log 3.59 ( n) = 3 k n 2 k n = O( n ) O( n ) T 2 More complicated, but asymptotically faster. Multiplication Algorithm 4: Part 2 Previous: u = (a+b) (c+d) v = ac w = bd xy = v << n + (u-v-w) << n/2 + w An overlooked detail: a+b and c+d can be n/2+ bit numbers. Middle term can overflow into high term. Solution: a+b = a << n/2 + b c+d = c << n/2 + d Break sums into (n/2+) th bit and rest. (a+b) (c+d) = (a c ) << n + (a d + b c ) << n/2 + b d O() time O(n/2) time Multiplication Algorithms 5 & 6 Karp s FFT-based algorithm: O(n log 2 n) Schoenhage & Strassen s FFT-based algorithm, 97: O(n log n log log n) Example of log-shaving slightly better runningtimes with lower logarithmic terms. Best known.

8 Multiplication Algorithms 7,... Use parallelism. Even serial processors use some bit-level parallelism. Divide-&-conquer & FFT algorithms easily parallelized. Beyond the scope of this course. Multiplication Algorithms Summary Higher constant factors more complicated, asymptotically-faster algorithms are slower for typical input sizes. Karatsuba s algorithm somewhat practical: Used on large numbers in cryptography. Generalizes to matrix multiplication Strassen s algorithm, [CLRS] 28.2 Algorithm Analysis Summary. State problem. 2. Characterize the input size. 3. State algorithm. Often difficult, of course. We almost missed an important 4. Prove algorithm correctness. detail that would have produced Necessary! incorrect result. 5. Determine its resource usage. Often via recurrence equations. Allows comparison of algorithms. Can decide which algorithm suitable for given application. Can guide the search for better algorithms.

9 Lists Abstract Data Structures Abstractly, many data structures are simply collections of (key, value) pairs. Typically, interested in some of the following operations Basic Operations Almost always include these. Create : Insert : void collection collection kv void Search : collection key kv A (key,value) pair or a pointer to a pair. Alternatively, just a value.

10 Additional Operations Often include some of these. Min : Max : Pred : Succ : collection kv collection kv collection kv kv collection kv kv Requires keys from a totally ordered set. Union : Delete : collection collection void collection kv void Dictionaries A particularly common set of operations: Create Insert Delete Search Concrete Data Structures Will be looking at concrete data structures & their costs to implement those operations. First, variations of lists: Singly-linked 3,x 6,a,b 2,k Singly-linked & sorted,b 2,k 3,x 6,a Doubly-linked Doubly-linked & sorted 3,x 6,a,b 2,k,b 2,k 3,x 6,a

11 Mutability Reminder Unlike earlier courses, this course emphasizes mutable data structures. Why? Largely tradition. Immutable data structures sometimes less asymptotically efficient. Immutable data structures much less studied. I recommend Purely Functional Data Structures, Okasaki 998. Worst-Case Time Complexities Singly Singly, sorted Doubly Doubly, sorted Let n = collection size. Create Creation almost always O(). Insert n n Linear time to maintain sortedness. Search n n n n Linear search is only option. Singly Singly, sorted Doubly Doubly, sorted Min Max Pred Succ n n n or n n or n n n n or n n or n n or n Sortedness helps. Tail ptr helps if sorted. Must search, unless given ptr to kv as arg & previous ptr points to pred. Must search, unless given ptr to kv as arg.

12 Singly Singly, sorted Doubly Doubly, sorted Union Delete or n n or n n n n or n or n Unsorted append. Sorted merge. Tail ptr helps append. Must search, unless given ptr to kv as arg & have previous ptrs in list. Lessons There are many kinds of lists. To get lower complexities, generally need more space (double-linkage and/or tail pointers) and larger constant factors (maintaining double-linkage). Extra invariants sometimes help, sometimes hurt. Which is best depends on the operations needed. I.e., there are always tradeoffs. Other Concrete Data Structures? What about arrays?? Some advantages, but imposes maximum collection size. But, can occasionally enlargen array efficiently. Requires amortized analysis discussed later. Can combine arrays + lists into hash tables. Discussed later.

13 Other Concrete Data Structures? What about trees?? Again, can be singly- or doubly-linked, unsorted or sorted. Worst-case bounds same as lists. Balancing helps discussed soon Other Concrete Data Structures? What about heaps?? Some operations logarithmic: Insert, either Min or Max, either DeleteMin or DeleteMax. But Union is linear. Will discuss extensions to heaps to improve on this.

14 Elementary Data Structures Stacks, Queues, & Lists Amortized analysis Trees The Stack ADT ( 4.2.) The Stack ADT stores arbitrary objects Insertions and deletions follow the last-in first-out scheme Think of a spring-loaded plate dispenser Main stack operations: push(object o): inserts element o pop(): removes and returns the last inserted element Auxiliary stack operations: top(): returns the last inserted element without removing it size(): returns the number of elements stored isempty(): a Boolean value indicating whether no elements are stored Elementary Data Structures 2 Applications of Stacks Direct applications Page-visited history in a Web browser Undo sequence in a text editor Chain of method calls in the Java Virtual Machine or C++ runtime environment Indirect applications Auxiliary data structure for algorithms Component of other data structures Elementary Data Structures 3 Array-based Stack ( 4.2.2) A simple way of implementing the Stack ADT uses an array We add elements from left to right A variable t keeps track of the index of the top element (size is t+) S Algorithm pop(): if isempty() then throw EmptyStackException else t t return S[t +] Algorithm push(o) if t = S.length then throw FullStackException else t t + S[t] o 0 2 t Elementary Data Structures 4 Growable Array-based Stack In a push operation, when the array is full, instead of throwing an exception, we can replace the array with a larger one How large should the new array be? incremental strategy: increase the size by a constant c doubling strategy: double the size Algorithm push(o) if t = S.length then A new array of size for i 0 to t do A[i] S[i] S A t t + S[t] o Elementary Data Structures 5 Comparison of the Strategies We compare the incremental strategy and the doubling strategy by analyzing the total time T(n) needed to perform a series of n push operations We assume that we start with an empty stack represented by an array of size We call amortized time of a push operation the average time taken by a push over the series of operations, i.e., T(n)/n Elementary Data Structures 6

15 Analysis of the Incremental Strategy We replace the array k = n/c times The total time T(n) of a series of n push operations is proportional to n + c + 2c + 3c + 4c + + kc = n + c( k) = n + ck(k + )/2 Since c is a constant, T(n) is O(n + k 2 ),i.e., O(n 2 ) The amortized time of a push operation is O(n) Direct Analysis of the Doubling Strategy We replace the array k = log 2 n times The total time T(n) of a series of n push operations is proportional to n k = n + 2 k + = 2n T(n) is O(n) The amortized time of a push operation is O() geometric series Elementary Data Structures 7 Elementary Data Structures 8 Accounting Method Analysis of the Doubling Strategy The accounting method determines the amortized running time with a system of credits and debits We view a computer as a coin-operated device requiring cyber-dollar for a constant amount of computing. We set up a scheme for charging operations. This is known as an amortization scheme. The scheme must give us always enough money to pay for the actual cost of the operation. The total cost of the series of operations is no more than the total amount charged. (amortized time) (total $ charged) / (# operations) Elementary Data Structures 9 Amortization Scheme for the Doubling Strategy Consider again the k phases, where each phase consisting of twice as many pushes as the one before. At the end of a phase we must have saved enough to pay for the array-growing push of the next phase. At the end of phase i we want to have saved i cyber-dollars, to pay for the array growth for the beginning of the next phase. $ $ $ $ $ $ $ $ We charge $3 for a push. The $2 saved for a regular push are stored in the second half of the array. Thus, we will have 2(i/2)=i cyber-dollars saved at then end of phase i. Therefore, each push runs in O() amortized time; n pushes run in O(n) time. Elementary Data Structures 0 $ $ The Queue ADT ( 4.3.) The Queue ADT stores arbitrary objects Insertions and deletions follow the first-in first-out scheme Insertions are at the rear of the queue and removals are at the front of the queue Main queue operations: enqueue(object o): inserts element o at the end of the queue dequeue(): removes and returns the element at the front of the queue Auxiliary queue operations: front(): returns the element at the front without removing it size(): returns the number of elements stored isempty(): returns a Boolean value indicating whether no elements are stored Exceptions Attempting the execution of dequeue or front on an empty queue throws an EmptyQueueException Applications of Queues Direct applications Waiting lines Access to shared resources (e.g., printer) Multiprogramming Indirect applications Auxiliary data structure for algorithms Component of other data structures Elementary Data Structures Elementary Data Structures 2

16 Singly Linked List Queue with a Singly Linked List A singly linked list is a concrete data structure consisting of a sequence of nodes Each node stores element link to the next node elem next node We can implement a queue with a singly linked list The front element is stored at the first node The rear element is stored at the last node The space used is O(n) and each operation of the Queue ADT takes O() time r nodes f A B C D Elementary Data Structures 3 elements Elementary Data Structures 4 List ADT ( 5.2.2) The List ADT models a sequence of positions storing arbitrary objects It allows for insertion and removal in the middle Query methods: isfirst(p), islast(p) Accessor methods: first(), last() before(p), after(p) Update methods: replaceelement(p, o), swapelements(p, q) insertbefore(p, o), insertafter(p, o), insertfirst(o), insertlast(o) remove(p) Elementary Data Structures 5 Doubly Linked List A doubly linked list provides a natural implementation of the List ADT Nodes implement Position and store: element link to the previous node link to the next node Special trailer and header nodes header prev elem nodes/positions elements next node trailer Elementary Data Structures 6 Trees ( 6.) In computer science, a tree is an abstract model Computers R Us of a hierarchical structure A tree consists of nodes with a parent-child relation Sales Manufacturing Applications: Organization charts US International Laptops Desktops File systems Europe Asia Canada Programming environments R&D Tree ADT ( 6..2) We use positions to abstract nodes Generic methods: integer size() boolean isempty() objectiterator elements() positioniterator positions() Accessor methods: position root() position parent(p) positioniterator children(p) Query methods: boolean isinternal(p) boolean isexternal(p) boolean isroot(p) Update methods: swapelements(p, q) object replaceelement(p, o) Additional update methods may be defined by data structures implementing the Tree ADT Elementary Data Structures 7 Elementary Data Structures 8

17 Preorder Traversal ( 6.2.3) A traversal visits the nodes of a tree in a systematic manner In a preorder traversal, a node is visited before its descendants Application: print a structured document Make Money Fast! Motivations 2. Methods References 3 4. Greed.2 Avidity Stock Fraud Algorithm preorder(v) visit(v) for each child w of v preorder (w) 2.2 Ponzi Scheme 2.3 Bank Robbery Elementary Data Structures 9 Postorder Traversal ( 6.2.4) In a postorder traversal, a node is visited after its descendants Application: compute space used by files in a directory and its subdirectories 9 cs6/ hc.doc 3K 3 homeworks/ hnc.doc 2K DDR.java 0K Algorithm postorder(v) for each child w of v postorder (w) visit(v) 7 programs/ Stocks.java 25K Robot.java 20K todo.txt K Elementary Data Structures 20 8 Amortized Analysis of Tree Traversal Time taken in preorder or postorder traversal of an n-node tree is proportional to the sum, taken over each node v in the tree, of the time needed for the recursive call for v. The call for v costs $(c v + ), where c v is the number of children of v For the call for v, charge one cyber-dollar to v and charge one cyber-dollar to each child of v. Each node (except the root) gets charged twice: once for its own call and once for its parent s call. Therefore, traversal time is O(n). Elementary Data Structures 2 Binary Trees ( 6.3) A binary tree is a tree with the following properties: Each internal node has two children The children of a node are an ordered pair We call the children of an internal node left child and right child Alternative recursive definition: a binary tree is either a tree consisting of a single node, or a tree whose root has an ordered pair of children, each of which is a binary tree D Applications: arithmetic expressions decision processes searching Elementary Data Structures 22 B H E I A F C G Arithmetic Expression Tree Binary tree associated with an arithmetic expression internal nodes: operators external nodes: operands Example: arithmetic expression tree for the expression (2 (a ) + (3 b)) 2 a + 3 b Decision Tree Binary tree associated with a decision process internal nodes: questions with yes/no answer external nodes: decisions Example: dining decision Yes How about coffee? Want a fast meal? On expense account? Starbucks In N Out Antoine's Denny s No Yes No Yes No Elementary Data Structures 23 Elementary Data Structures 24

18 Properties of Binary Trees Notation n number of nodes e number of external nodes i number of internal nodes h height Properties: e = i + n = 2e h i h (n )/2 e 2 h h log 2 e h log 2 (n + ) Inorder Traversal In an inorder traversal a node is visited after its left subtree and before its right subtree Application: draw a binary tree x(v) = inorder rank of v y(v) = depth of v Algorithm inorder(v) if isinternal (v) inorder (leftchild (v)) visit(v) if isinternal (v) inorder (rightchild (v)) Elementary Data Structures 25 Elementary Data Structures 26 Euler Tour Traversal Generic traversal of a binary tree Includes a special cases the preorder, postorder and inorder traversals Walk around the tree and visit each node three times: on the left (preorder) from below (inorder) on the right (postorder) + L R B Elementary Data Structures 27 Printing Arithmetic Expressions 2 Specialization of an inorder traversal print operand or operator when visiting node print ( before traversing left subtree print ) after traversing right subtree a + 3 b Algorithm printexpression(v) if isinternal (v) print( ( ) inorder (leftchild (v)) print(v.element ()) if isinternal (v) inorder (rightchild (v)) print ( ) ) ((2 (a )) + (3 b)) Elementary Data Structures 28 Linked Data Structure for Representing Trees ( 6.4.3) A node is represented by an object storing Element Parent node Sequence of children nodes Node objects implement the Position ADT A C B D E F B A D F C E Elementary Data Structures 29 Linked Data Structure for Binary Trees ( 6.4.2) A node is represented by an object storing Element Parent node Left child node Right child node Node objects implement the Position ADT A B C D E A B C Elementary Data Structures 30 D E

19 Array-Based Representation of Binary Trees ( 6.4.) nodes are stored in an array A 2 3 let rank(node) be defined as follows: rank(root) = if node is the left child of parent(node), rank(node) = 2*rank(parent(node)) if node is the right child of parent(node), E F C J rank(node) = 2*rank(parent(node))+ 0 Elementary Data Structures 3 B G H D

20 Introduction to complexity Complexity: a measure of the performance of an algorithm An algorithm s performance depends on internal and external factors Internal The algorithm s efficiency, in terms of: Time required to run Space (memory storage) required to run External Size of the input to the algorithm Speed of the computer on which it is run Quality of the compiler Complexity measures the internal factors (usually more interested in time than space) CSM Complexity 2 Growth rates and big-o notation Growth rates capture the essence of an algorithm s performance Big-O notation indicates the growth rate. It is the class of mathematical formula that best describes an algorithm s performance, and is discovered by looking inside the algorithm Big-O is a function with parameter N, where N is usually the size of the input to the algorithm For example, if an algorithm depending on the value n has performance an 2 + bn + c (for constants a, b, c) then we say the algorithm has performance O(N 2 ) For large N, the N 2 term dominates. Only the dominant term is included in big-o CSM Complexity 3

21 Time Common growth rates Time complexity O() constant O(log N) log O(N) linear O(N log N) n-log-n O(N 2 ) quadratic O(N 3 ) cubic O(2 N ) exponential Example Adding to the front of a linked list Finding an entry in a sorted array Finding an entry in an unsorted array Sorting n items by divide-and-conquer Shortest path between two nodes in a graph Simultaneous linear equations The Towers of Hanoi problem CSM Complexity 4 Growth rates O(N 2 ) O(Nlog N) For a short time N 2 is better than NlogN Number of Inputs CSM Complexity 5 Calculating the actual time taken by a program (example) A program takes 0ms to process one data item (i.e. to do one operation on the data item) How long would the program take to process 000 data items, if time is proportional to: log 0 N N N log 0 N N 2 N 3 (time for item) x (big-o( ) time complexity of N items) CSM Complexity 6 2

22 Best, average, worst-case complexity In some cases, it is important to consider the best, worst and/or average (or typical) performance of an algorithm: Worst, O(N) or o(n): or > true function * Typical, Θ(N): true function * Best, Ω(N): true function * E.g., when sorting a list into order, if it is already in order then the algorithm may have very little work to do The worst-case analysis gives a bound for all possible input (and may be easier to calculate than the average case) * These approximations are true only after N has passed some value CSM Complexity 7 How do we calculate big-o? Five guidelines for finding out the time complexity of a piece of code Loops 2 Nested loops 3 Consecutive statements 4 If-then-else statements 5 Logarithmic complexity CSM Complexity 8 Guideline : Loops The running time of a loop is, at most, the running time of the statements inside the loop (including tests) multiplied by the number of iterations. executed n times for (i=; i<=n; i++) { m = m + 2; } constant time Total time = a constant c * n = cn = O(N) CSM Complexity 9 3

23 Guideline 2: Nested loops Analyse inside out. Total running time is the product of the sizes of all the loops. outer loop executed n times for (i=; i<=n; i++) { for (j=; j<=n; j++) { k = k+; } } constant time inner loop executed n times Total time = c * n * n * = cn 2 = O(N 2 ) CSM Complexity 0 Guideline 3: Consecutive statements Add the time complexities of each statement. constant time constant time outer loop executed n times x = x +; for (i=; i<=n; i++) { m = m + 2; } for (i=; i<=n; i++) { for (j=; j<=n; j++) { k = k+; } constant time } Total time = c 0 + c n + c 2 n 2 = O(N 2 ) executed n times inner loop executed n times CSM Complexity Guideline 4: If-then-else statements Worst-case running time: the test, plus either the then part or the else part (whichever is the larger). test: constant another if : constant + constant (no else part) if (depth( )!= otherstack.depth( ) ) { return false; } else { for (int n = 0; n < depth( ); n++) { if (!list[n].equals(otherstack.list[n])) return false; } } then part: constant else part: (constant + constant) * n Total time = c 0 + c + (c 2 + c 3 ) * n = O(N) CSM Complexity 2 4

24 Guideline 5: Logarithmic complexity An algorithm is O(log N) if it takes a constant time to cut the problem size by a fraction (usually by ½) Example algorithm (binary search): finding a word in a dictionary of n pages Look at the centre point in the dictionary Is word to left or right of centre? Repeat process with left or right part of dictionary until the word is found CSM Complexity 3 Performance isn t everything! There can be a tradeoff between: Ease of understanding, writing and debugging Efficient use of time and space So, maximum performance is not always desirable However, it is still useful to compare the performance of different algorithms, even if the optimal algorithm may not be adopted CSM Complexity 4 5

25 Binary Search Trees The Dictionary Data Structure A dictionary stores a dynamically changing set S and supports the following operations: Insert(x): Insert element x into S Delete(x): Removes element x from S Find(x): Decides whether x is in S and, if so, reports x If the elements in S can be compared, the following operations are also desirable: Successor(x) / Predecessor(x): Reports the next larger / smaller element from x in S Minimum / Maximum: Reports the smallest / largest element in S Range-Search(x, y): Reports all elements z in S such that x z y Binary Search Tree Property A binary search tree is a binary tree with the following binary search tree property: For any node v, the elements stored in the left subtree of v are not greater than the element stored at v; the elements in the right subtree are not less than the element stored at v

26 The Find Operation Searching for an Element Goal: Identify the node storing a given element x, if such a node exists x = 25 The Find Operation Searching for an Element Goal: Identify the node storing a given element x, if such a node exists Find(x, T) v root(t) 2 while v nil and x key(v) 3 do if x < key(v) 4 then v left_child(v) 5 else v right_child(v) 6 return v x = 25 The Find Operation Searching for an Element Goal: Identify the node storing a given element x, if such a node exists Lemma: If element x is stored in T, procedure Find finds the node storing x Lemma: The running time of procedure Find is O(height(T)). x = 25 2

27 The Minimum Operation Finding the Smallest Element Goal: Find the smallest element in S The Minimum Operation Finding the Smallest Element Goal: Find the smallest element in S The Minimum Operation Finding the Smallest Element Goal: Find the smallest element in S

28 The Minimum Operation Finding the Smallest Element Goal: Find the smallest element in S Minimum(T) v root(t) 2 if v nil 3 then return Subtree-Minimum(v) 4 else return nil Subtree-Minimum(v) while left_child(v) nil 2 do v left_child(v) 3 return v Lemma: The running time of procedure Minimum is O(height(T)). The Successor Operation Finding the Next Element Goal: Given an element x S, find the next larger element y S. Successor(v) if right_child(v) nil 2 then return Subtree-Minimum(right_child(v)) 3 w parent(v) 4 while w nil and v = right(w) 5 do v w 6 w parent(w) 3 7 return w The Successor Operation Finding the Next Element Goal: Given an element x S, find the next larger element y S. Successor(v) if right_child(v) nil 2 then return Subtree-Minimum(right_child(v)) 3 w parent(v) 4 while w nil and v = right(w) 5 do v w 6 w parent(w) 7 return w Lemma: The running time of procedure Successor is O(height(T)). 4

29 The Insert Operation Inserting a New Element Goal: Insert a new element x into the dictionary The Insert Operation Inserting a New Element Goal: Insert a new element x into the dictionary Insert(x, T) v root(t) 2 w nil 3 while v nil 4 do w v 5 if x < key(v) 6 then v left_child(v) 7 else v right_child(v) 8 u new node with key x 9 if w = nil 0 then root(t) u else parent(u) w 2 if x < key(w) 3 then left_child(w) u 4 else right_child(w) u The Insert Operation Inserting a New Element Goal: Insert a new element x into the dictionary. Lemma: Procedure Insert preserves the binary search tree property. Lemma: The running time of procedure Insert is O(height(T)). Insert(x, T) v root(t) 2 w nil 3 while v nil 4 do w v 5 if x < key(v) 6 then v left_child(v) 7 else v right_child(v) 8 u new node with key x 9 if w = nil 0 then root(t) u else parent(u) w 2 if x < key(w) 3 then left_child(w) u 4 else right_child(w) u 5

30 The Delete Operation Deleting Element Goal: Delete an element x from the dictionary The Delete Operation Deleting Element Goal: Delete an element x from the dictionary The Delete Operation Deleting Element Delete(v, T) if left_child(v) = nil or right_child(v) = nil 2 then w v 3 else w Successor(v) 4 if left_child(w) nil 5 then u left_child(w) 6 else u right_child(w) 7 if u nil 8 then parent(u) parent(w) 9 if parent(w) = nil 0 then root(t) u else if w = left_child(parent(w)) 2 then left_child(parent(w)) u 3 else right_child(parent(w)) u 4 if w v 5 then key(v) key(w) 6 Delete node w Goal: Delete an element x from the dictionary

31 The Delete Operation Deleting Element Delete(v, T) if left_child(v) = nil or right_child(v) = nil 2 then w v 3 else w Successor(v) 4 if left_child(w) nil 5 then u left_child(w) 6 else u right_child(w) 7 if u nil 8 then parent(u) parent(w) 9 if parent(w) = nil 0 then root(t) u else if w = left_child(parent(w)) 2 then left_child(parent(w)) u 3 else right_child(parent(w)) u 4 if w v 5 then key(v) key(w) 6 Delete node w Goal: Delete an element x from the dictionary. Lemma: The running time of procedure Delete is O(height(T)). Lemma: Procedure Delete preserves the binary search tree property. Binary Search Tree Operations Summary Theorem: A binary search tree supports operations Insert, Delete, Find, Minimum, Maximum, Predecessor, and Successor in O(height(T)) time. Best case: Tree is complete Height is Θ(lg n) Worst case: Tree is a linked list Height is Θ(n) 7

32 AVL Trees (See extra AVL notes too) AVL Trees A type of binary search tree which is nearly as good as a balanced tree for time complexity of the operations, and whose structure we can maintain as insertions and deletions proceed, is the AVL tree AVL trees are named after the Russian mathematicians G.M. Adelson-Velskii and E.M. Landis, who discovered them in 962 An AVL tree is a binary search tree in which the heights of the left and right subtrees of the root differ by at most, and in which the left and right subtrees of the root are again AVL trees Cx220 - Other Trees 2 Examples of AVL Trees AVL Non-Legal AVL AVL and balanced Legal AVL, but not balanced Cx220 - Other Trees 3

33 Implementing AVL Trees To implement algorithms for inserting and deleting from an AVL tree, we associate a balance factor with each node This is left high, right high or equal depending on whether the left subtree of the node has height greater than, less than or equal to that of the right subtree Thus the node may be represented by: public class AVLNode { Comparable data; int balancefactor; AVLNode left; AVLNode right; } public static final int LEFT_HIGH = -; public static final int RIGHT_HIGH = ; public static final int EQUAL_HIGH = 0; Cx220 - Other Trees 4 Examples of AVL Trees with Balance Factors R L E At L, the left subtree is higher than the right At R, the right subtree is higher than the left At E, the subtrees are equal (in fact, they are empty) E R R R E R R R E E E R (AVL trees can be quite skewed) E Cx220 - Other Trees 5 A Non-AVL Tree (height difference is more than one in places) L LL E RR R * Note that no AVL tree will ever have two double imbalances, like this, at the same time - we ll see why later. This is to illustrate how a tree can be unbalanced somewhere other than the root E E LL means that the left subtree is too high RR means that the right subtree is too high Cx220 - Other Trees 6 2 2

34 Insertion into an AVL Tree (the simple case) Consider constructing a binary search tree in the usual way from the sequence of keys 4, 7, 2, 8, 6,, 5, making each node with its balance factor as we go: 4:E 4:R 7:E 4:E 2:E 7:E 4:R 2:E 7:R 4:R 2:E 7:E 8:E 6:E 8:E Cx220 - Other Trees 7 Insertion contd (inserting, 5) 4:E 2:L 7:E :E 6:E 8:E :E 4:R 2:L 7:L 6:L 8:E 5:E This example shows how the balance factors change as new nodes are added - notice the changes are only on the path from new node to root node Cx220 - Other Trees 8 The Insertion Algorithm (for more complex cases) Insertion starts as for a binary search tree The only case in which we need to do anything special is if the new node is added to a subtree of the root that is higher than the other subtree and the new node increases the height of the subtree In this case we have to carry out some balancing operations in the neighbourhood of the root We consider the case of inserting into the right subtree (left is similar) There are two cases to consider: Cx220 - Other Trees 9 3 3

35 right hand subtree of root is right high after insertion 4:R 2:E 7:E 6:E R 8:E R 9:E Inserting 9 causes 4:R -> 4:RR Perform a left rotation 7:E 4:E 8:R 2:E 6:E 9:E The 6 has been relocated as a child of 4. The nodes , which comprise the outside edge of the tree, have been rotated one node to the left Cx220 - Other Trees 0 2 right hand subtree of root is left high after insertion 4:R 2:E 2 7:E 5 6:E L 8:E Inserting 5 Perform a double rotation right rotation of right subtree ( in this example) tree now like previous slide left rotation of whole tree, as 6:E before 4:E 7:R 2:E 5:E 8:E left hand child of right hand child of root becomes root left hand child of new root becomes right hand child of old root Cx220 - Other Trees The balancing rule: When you have found the node that is out of balance: If that node and the two nodes immediately below it are in a straight line then a single rotation is needed If the nodes lie in a dog-leg pattern, then you need a double rotation to restore the balance Although the algorithm seems complicated, it still gives insertion in O(log n) time Note: there is no LR or RL imbalance, only LL and RR. The LL means the subtree to the left of the node is too deep (the same with RR by symmetry). It does not refer to the pattern of the imbalance at all! Cx220 - Other Trees 2 4 4

36 Another example From "An introduction to data structures & algorithms with Java, G.W. Rowe Build an AVL tree using the following data: 0, 20, 30, 25, 27, 7, 4 0:E 0:R 0:RR 20:E 20:R 30:E The problem lies on a straight line, so apply single rotation Cx220 - Other Trees 3 Another example (contd.) 20:E 20: 0:E 30:E 0:E 30:LL AVL structure is now restored 25:R 27:E After inserting 25 and 27, node at 30 becomes unbalanced Cx220 - Other Trees 4 Rebalance by a double rotation 20: 20:R 0:E 30:LL 0:E 27:E 27:L 25:E 30:E 25:E First part of double rotation rotates 27 to replace 25 Second part of rotation moves 27 up to replace 30 Cx220 - Other Trees 5 5 5

37 Inserting 7, 4 20:E 0:L 27:E 7:E 25:E 30:E Inserting 7 causes no problem 4:E 7:L 0:LL 20: 27:E 25:E 30:E Inserting 4 causes a problem - it s a straight-line single rotation Cx220 - Other Trees 6 Rebalance by a single rotation 20:E 7:E 27:E 4:E 0:E 25:E 30:E Everything is nicely balanced again Cx220 - Other Trees 7 Applications of Binary Trees Sorting: We can sort data by reading it in, item by item, and constructing a binary search tree as we go When we have read all the data, we output it by doing in-order traversal of the tree If there is any possibility that the data items are nearly sorted already, or nearly in reverse order, then it is important to use an AVL tree Otherwise you end up with a very unbalanced tree! The time complexity of this method of sorting is, in general, O(nlog 2 n + n) i.e., O(nlog n) Cx220 - Other Trees 8 6 6

38 Searching maintaining an order Consider the problem of counting how many times each word appears in a piece of text (a concordance): Read the text and build a binary search tree in which the key is a word (order is alphabetical) and the data is the number of times it occurs As we meet each word, we check to see whether it's already in the tree If it is, we add one to the data field; otherwise we insert a new node with the word as key and with the data field set to one (No need to use an AVL tree because the words will not occur in alphabetical order) When we've finished, an in-order traversal allows us to print the words, with their frequencies, in alphabetical order Cx220 - Other Trees 9 Compiling arithmetic expressions We can represent an arithmetic expression such as (A + B) * (C + D) * 2 X / Y as a binary tree, in which the leaves are constants or variables and the nodes are operations: A + 3 * B * / 2 X Y + 2 A post order traversal then gives us the order in which the operations have C D to be carried out Cx220 - Other Trees

39 Elementary Graph Algorithms Paths and Cycles A path is a sequence of vertices P = (v 0, v,, v k ) such that, for i k, edge (v i, v i ) E. Path P is simple if no vertex appears more than once in P. A cycle is a sequence of vertices C = (v 0, v,, v k ) such that, for 0 i < k, edge (v i, v (i + ) mod k ) E. Cycle C is simple if the path (v 0, v, v k ) is simple. g f d i e a b c h j Paths and Cycles A path is a sequence of vertices P = (v 0, v,, v k ) such that, for i k, edge (v i, v i ) E. Path P is simple if no vertex appears more than once in P. A cycle is a sequence of vertices C = (v 0, v,, v k ) such that, for 0 i < k, edge (v i, v (i + ) mod k ) E. Cycle C is simple if the path (v 0, v, v k ) is simple. P = (g, d, e, b, d, a, h) is not simple. g f d i P 2 = (f, i, c, j) is simple. e a b c h j

40 Paths and Cycles A path is a sequence of vertices P = (v 0, v,, v k ) such that, for i k, edge (v i, v i ) E. Path P is simple if no vertex appears more than once in P. A cycle is a sequence of vertices C = (v 0, v,, v k ) such that, for 0 i < k, edge (v i, v (i + ) mod k ) E. Cycle C is simple if the path (v 0, v, v k ) is simple. C = (a, h, j, c, i, f, g, d) is simple. g f d i e a b c h j Paths and Cycles A path is a sequence of vertices P = (v 0, v,, v k ) such that, for i k, edge (v i, v i ) E. Path P is simple if no vertex appears more than once in P. A cycle is a sequence of vertices C = (v 0, v,, v k ) such that, for 0 i < k, edge (v i, v (i + ) mod k ) E. Cycle C is simple if the path (v 0, v, v k ) is simple. C = (a, h, j, c, i, f, g, d) is simple. g f d i C 2 = (g, d, b, h, j, c, i, e, b) is not simple. e a b c h j Subgraphs A graph H = (W, F) is a subgraph of a graph G = (V, E) if W V and F E. 2

41 Subgraphs A graph H = (W, F) is a subgraph of a graph G = (V, E) if W V and F E. Spanning Graphs A spanning graph of G is a subgraph of G that contains all vertices of G. Spanning Graphs A spanning graph of G is a subgraph of G that contains all vertices of G. 3

42 Connectivity A graph G is connected if there is a path between any two vertices in G. The connected components of a graph are its maximal connected subgraphs. Trees A tree is a graph that is connected and contains no cycles. Spanning Trees A spanning tree of a graph is a spanning graph that is a tree. 4

43 Adjacency List Representation of a Graph List of vertices Pointer to head of adjacency list List of edges Pointers to adjacency list entries Adjacency list entry Pointer to edge y Pointers to endpoints a y d c w v u c e u x a b x z b w e v z d Graph Exploration Goal: Visit all the vertices in the graph of G Count them Number them Identify the connected components of G Compute a spanning tree of G Graph Exploration ExploreGraph(G) Label every vertex and edge of G as unexplored 2 for every vertex v of G 3 do if v is unexplored 4 then mark v as visited 5 S Adj(v) 6 while S is not empty 7 do remove an edge (u, w) from S 8 if (u, w) is unexplored 9 then if w is unexplored 0 then mark edge (u, w) as tree edge mark vertex w as visited 2 S S Adj(w) 3 else mark edge (u, w) as a non-tree edge 5

44 Properties of ExploreGraph Theorem: Procedure ExploreGraph visits every vertex of G. Theorem: Every iteration of the for-loop that does not immediately terminate completely explores a connected component of G. Theorem: The graph defined by the tree edges is a spanning forest of G. Connected Components ConnectedComponents(G) Label every vertex and edge of G as unexplored 2 c 0 3 for every vertex v of G 4 do if v is unexplored 5 then c c + 6 assign component label c to v 7 S Adj(v) 8 while S is not empty 9 do remove an edge (u, w) from S 0 if (u, w) is unexplored then if w is unexplored 2 then mark edge (u, w) as tree edge 3 assign component label c to w 4 S S Adj(w) 5 else mark edge (u, w) as a non-tree edge Breadth-First Search BFS(G) Label every vertex and edge of G as unexplored 2 for every vertex v of G 3 do if v is unexplored 4 then mark v as visited Running time: O(n + m) 5 S Adj(v) 6 while S is not empty 7 do (u, w) Dequeue(S) 8 if (u, w) is unexplored 9 then if w is unexplored 0 then mark edge (u, w) as tree edge mark vertex w as visited 2 Enqueue(S, Adj(w)) 3 else mark edge (u, w) as a non-tree edge 6

45 Properties of Breadth-First Search Theorem: Breadth-first search visits the vertices of G by increasing distance from the root. Theorem: For Levery 0 edge (v, w) in G such that L 4 v L i and w L j, i j. L L 3 L 2 Depth-First Search DFS(G) Label every vertex and edge of G as unexplored 2 for every vertex v of G 3 do if v is unexplored 4 then mark v as visited Running time: O(n + m) 5 S Adj(v) 6 while S is not empty 7 do (u, w) Pop(S) 8 if (u, w) is unexplored 9 then if w is unexplored 0 then mark edge (u, w) as tree edge mark vertex w as visited 2 Push(S, Adj(w)) 3 else mark edge (u, w) as a non-tree edge An Important Property of Depth-First Search Theorem: For every non-tree edge (v, w) in a depth-first spanning tree of an undirected graph, v is an ancestor of w or vice versa. 7

46 A Recursive Depth-First Search Algorithm DFS(G) Label every vertex and every edge of G as unexplored 2 for every vertex v of G 3 do if v is unexplored 4 then DFS(G, v) DFS(G, v) mark vertex v as visited 2 for every edge (v, w) in Adj(v) 3 do if (v, w) is unexplored 4 then if w is unexplored 5 then mark edge (v, w) as tree edge 6 DFS(G, w) 7 else mark edge (v, w) as a non-tree edge 8