/633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Balanced Search Trees Date: 9/20/18

Transcription

1 /633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Balanced Search Trees Date: 9/20/ Introduction For next few lectures we will be looking at several important data structures. A data structure is a method for storing data so that operations you care about can be performed quickly. Data structures are typically used as part of some larger algorithm or system, and good data structures are often crucial when especially fast performance is needed. Today we will be focusing in particular on what are called dictionary data structures, that support insert and lookup operations (and usually delete as well). Specifically, Definition A dictionary data structure is a data structure supporting following operations: insert(key,object): insert (key, object) pair. For instance, this could be a word and its definition, a name and phone number, etc. The key is what will be used to access object. lookup(key): return associated object delete(key): remove key and its object from data structure. We may or may not care about this operation. One option is we could use a sorted array. Then, a lookup takes O(log n) time using binary search. However, an insert may take Ω(n) time in worst case because we have to shift everything to right in order to make room for new key. Anor option might be an unsorted list. In that case, inserting can be done in O(1) time, but a lookup may take Ω(n) time. Our goal is going to be to construct data structures which let us perform all operations in O(log n) time. A binary search tree is a binary tree in which each node stores a (key, object) pair such that all descendants to left have smaller keys and all descendants to right have larger keys (let s not worry about case of multiple equal keys). To do a lookup operation you simply walk down from root, going left or right depending on wher query is smaller or larger than key in current node, until you get to correct key or walk off tree. We will also talk about non-binary search trees that potentially have more than one key in each node, and nodes may have more than two children. For rest of this discussion, we will ignore object part of things. We will just worry about keys since that is all that matters as far as understanding data structures is concerned. 6.2 Simple Binary Search Trees The simplest way to maintain a binary search tree is to implement insert operations as follows: 1

2 insert(x): If tree is empty n put x in root. Orwise, compare it to root: if x is smaller n recursively insert on left; orwise recursively insert on right. Equivalently: walk down tree as if doing a lookup, and n insert x into a new leaf at end. Example: build a tree by inserting sequence: H O P K I N S. On positive side, this is very easy to implement (although deletes are a bit of a pain think of how you would implement m). However, worst-case performance is very bad. If tree has depth d, n an insert or a lookup could take Ω(d) time. So if tree is very unbalanced, operations could take Ω(n) time each! This is actually similar to what happens with quicksort if input is already sorted, n each recursive call only decreases input size (quicksort) or depth of tree (BSTs) by one. Main idea: The problem with basic binary search tree was that we were not maintaining balance. On or hand, if we try to maintain a perfectly balanced tree, we will spend too much time rearranging things. So, we want to be balanced but also give ourselves some slack. It s a bit like how in median-finding algorithm, we gave ourselves slack by allowing pivot to be near middle. For B-trees, we will make tree perfectly balanced, but give ourselves slack by allowing some nodes to have more children than ors. 6.3 B-trees and trees A B-tree is a search tree where for some pre-specified t 2 (think of t = 2 or t = 3) following three properties hold. 1. Each node has between t 1 and 2t 1 keys in it (except root has between 1 and 2t 1 keys). Keys in a node are stored in a sorted array. 2. Each non-leaf has degree (number of children) equal to number of keys in it plus 1. So, node degrees are in range [t, 2t] except root has degree in range [2, 2t]. If v is a node with keys [a 1, a 2,..., a k ] and children are [v 1, v 2,..., v k+1 ], n tree rooted at v i contains only keys that are at least a i 1 and at most a i (except edge cases: tree rooted at v 1 has keys less than a 1, and tree rooted at v k+1 has keys at least a k ). E.g., if keys are [a 1, a 2,..., a 10 ] n re is one child for keys less than a 1, one child for keys between a 1 and a 2, and so on, until rightmost child has keys greater than a All leaves are at same depth. The idea is that by using flexibility in sizes and degrees of nodes, we will be able to keep trees perfectly balanced (in sense of all leaves being at same level) while still being able to do inserts cheaply. Note that case of t = 2 is called a tree since degrees are 2, 3, or 4. As an example, here is a tree for t = 3 (so, non-leaves have between 3 and 6 children (though root can have fewer) and maximum size of any node is 5). 2

3 The idea is that by using flexibility in sizes and degrees of nodes, we will be able to keep trees perfectly balanced (in sense of all leaves being at same level) while still being able to do inserts cheaply. Note that case of t = 2 is called a tree since degrees are 2, 3, or 4. Example: here is a tree for t = 3 (so, non-leaves have between 3 and 6 children though root can have fewer and maximum size of any node is 5). H M R A B C D K L N O T Y Z Now, rules for lookup and insert turn out to be pretty easy: Now we have to define how to do inserts and lookups. Lookup: Just do binary search in array at root. This will eir return item you are Lookup: looking Just for do(in binary whichsearch case you in arearray done) at or a pointer root. This to will appropriate eir return child, in item which youcase are looking youfor recurse (in which on that casechild. you are done) or a pointer to appropriate child, in which case you recurse on that child. Insert: To insert, walk down tree as if you are doing a lookup, but if you ever encounter a full Insert: node To (a insert, node with walk down maximum tree 2t as 1 if keys you are in it), doing perform a lookup, a split but operation if you ever on it encounter (described a full node below) (a node before with continuing. maximum 2t 1 keys in it), perform a split operation on it immediately (described below) before continuing. When you re at a leaf node, insert new element into 9.4. Finally, B-TREES insert AND new TREES key into leaf reached. 46 array at that node. Split: To split a node, pull median of its keys up to its parent and n split remaining 2t 22t one keys 2with keys into twoelements nodes nodes of greater tof t1 than keys 1 keys each each median). (one (onewith Then elements connect se less than nodes to median ir parent and one with in appropriate elements greater way (one than as median). child tothen left connect of se median nodes and to one irasparent child in to appropriate its right). way (one If asnode child beingtosplit left is of root, median n create and one a fresh as new child root to its node right). to put If node being median split in. is root, n create a fresh new root node to put median in. Let s consider example above. If we insert an E n that will go into leftmost leaf, making Let s consider it full. If weexample now insert above. an F, If we n insert an process E n of walking that willdown go into treeleftmost we will split leaf, making full node, it full. bringing If we now insert C up an to F, n root. inso, after process inserting of walking F down we willtree nowwe have: will split full node, bringing C up to root. So, after inserting F we will now have: C H M R A B D E F K L N O T Y Z Question: We know that performing a split maintains requirement of at least t 1 keys per Sanity non-root check: node We (because know that we split performing at a median) split maintains but is it possible requirement for a split of at to least make t 1 keys parent per non-root over-full? node (because we split at median) but is it possible for a split to make parent over-full? Answer: No, since if if parent were was full already we would full n have wealready would split have split on it onway down. way down. Let s now continue above example, inserting S, U, V : C H M R U A B D E F K L N O S T V Y Z Now, suppose we insert P. Doing this will bring 3 M up to a new root, and n we finally insert P in appropriate leaf node: M

4 Question: We know that performing a split maintains requirement of at least t 1 keys per A B D E F K L N O T Y Z non-root node (because we split at median) but is it possible for a split to make parent Question: over-full? We know that performing a split maintains requirement of at least t 1 keys per non-root Answer: node No, (because since if weparent split at was full median) we would buthave is italready possiblesplit for it a split on toway make down. parent over-full? Let s now continue above example, inserting S, U, V : Answer: No, since if parent was full we would have already split it on way down. Let s now continue above example, inserting C H M S, R U U, V : C H M R U A B D E F K L N O S T V Y Z Now, suppose we insert P. Doing this will bring M up to a new root, and n we finally insert A B D E F K L N O S T V Y Z P in appropriate leaf node: Now, suppose we insert P. Doing this will bring M up to a new root, and n we finally insert Now, P insuppose appropriate we insert leaf P. node: Doing this will bring M up to a new root, and n we finally insert P in appropriate leaf node: M M C H R U C H R U A B D E F K L N O P S T V Y Z Question: is tree always height-balanced (all leaves at same depth)? A B D E F K L N O P S T V Y Z Answer: yes, since we only grow tree up. Question: So, we have is maintained tree always our desired height-balanced properties. (all What leaves about at running same depth)? time? To perform a lookup, Answer: Let s perform prove yes, (somewhat binary sincesearch weinformally) only each grownode tree correctness we pass up. through, of our so insert procedure. total time for In aparticular, lookup is let s O(depth prove that log t). allwhat nodesis store depth right of number tree? ofsince keys (between at each level t 1we and have 2t a 1branching for a non-root, So, we have maintained our desired properties. What about running time? To perform factorand of a at at lookup, least mostt 2t we (except 1 for perform possibly root), binary at that search root), all elements in each node depth in we pass is O(log subtree through, t n). between Combining two adjacent so total se time for toger, keys are a lookup we indeed is see O(depth that least log t smaller t). cancels and What out is in at most depth expression larger, of tree? forand lookup that Since time: all leaves are same depth. at each level we have a branching factor of at least t (except Theorempossibly at Our insert root), algorithm depthmaintains is O(log t all n). three Combining properties se of atoger, B-tree. we see that t Proof: cancels Weout will inprove expression this by induction. for lookup Note time: that if we insert into an empty B-tree we get simply one node with one element in it. All three properties of a B-tree are indeed true. Now suppose that we have a B-tree which satisfies all three properties, and insert an element x. We claim that resulting tree still satisfied all three properties. The third property is obvious since tree grows up : let h be height of tree before insert, which by induction is depth of all leaves. If our insert does not split root n distance between root and all leaves is unchanged (and so all of m are still h), while if we do split root n all leaves are at distance h + 1 from new root. For first property, note that whenever we do a split operation, parent gains one new key while each child has exactly t 1 keys in it. Since when we do an insert we split any node on path which has 2t 1 keys in it, whenever we do a split operation on a node, its parent must have strictly less than 2t 1 keys (or else parent would have already been split). Thus none of our split operations violate degree bounds. And when we do finally insert new key, node we insert into must have strictly less than 2t 1 keys before insert (or else we would split it). Hence when we do an insert we never violate any of degree upper bounds. Similarly, we never 4

5 violate a degree lower bound since we never create a node with less than t 1 keys or than root (which is allowed to have fewer keys). So it remains only to prove that for every node, tree rooted at ith child of node contains keys which are between (i 1) and (i + 1) key in node. Whenever we do a split operation we preserve this property, so it is preserved as we move down tree on our insert, and in end new key that we insert is placed in a leaf node so that this property continues to hold. Hence this property (and this all three properties) continues to hold after inserting a new key. So, we have maintained our desired properties. What about running time? Suppose that depth is d. To perform a lookup, we perform binary search in each node we pass through, so total time for a lookup is O(d log t). What is depth of tree? Since at each level we have a branching factor of at least t (except possibly at root), depth is O(log t n). Combining se toger, we see that t cancels out in expression for lookup time: Time for lookup = O(log t n log t) = O((log n)/(log t) log t) = O(log n) Inserts are similar to lookups except for two issues. First, we may need to split nodes on way down, and secondly we need to insert element into leaf. So, we have: Time for insert = lookup-time + splitting-time + time-to-insert-into-leaf. The time to insert into a leaf is O(t). The splitting time is O(t) per split, which could happen at each step on way down, for a total of O(t log n). So, if t is a constant, n we still get total time O(log n). Note that this motivates us to make t small, which is one reason that trees get ir own name. More facts about B-trees: B-trees are used a lot in databases applications because y fit in nicely with memory hierarchy when you use a large value of t. For instance, if you have 1 billion items and use t = 1, 000, n you can probably keep top two levels in fast memory and only make one disk access at bottom level. The savings in disk accesses more than makes up for extra O(t) cost for insert. From a student three years ago: I was personally asked in interviews why B-trees are well suited for extremely large data sets. If you use t = 2, you have what is known as a tree. What is special about trees is that y can be implemented efficiently as binary trees using an idea called red-black trees, which is what we ll discuss next. 6.4 Red-Black Trees B-trees are great, but it s generally easier to use binary trees. Lookups are simpler, people find m easier to understand, and for general-purpose (non-database) applications, y tend to be a bit faster and a bit more flexible. There have been a whole bunch of balanced binary search trees developed you may have seen AVL trees when you took Data Structures ( ). One of 5

6 Trees most popular and widely used are red-black trees y re default in linux kernel, are used to optimize Java HashMaps, and are generally balanced search tree of choice in practice. I don t have time to go over m in full detail, but you should read book. What I ll try to explain here is how we can view m as essentially binary versions of trees. One quick note: unlike B-trees, for red-black trees (and binary trees in general) we ll use convention that all missing pointers actually point to NULL. In or words, keys are only stored at internal nodes: all leafs are just NULL. This will make it a bit easier to analyze So let s start from a tree. How can we turn it into a binary tree and still have all of nice properties of a tree? Unfortunately, we can t we re not going to be able to get exact height-balance on any binary tree. But recall that we don t need exact balance in order to have efficient lookups; we just need depth to be O(log n). So let s be a bit flexible in balance requirement. Maybe we get lucky, and all nodes of our are actually degree-2 (y have one key stored Red-Black Tree at m). Then we don t need to do anything. But what if we have some nodes with 2 or 3 keys stored at m? Let s start with case of 3 keys stored, since that s simpler: let s just transform Represent tree as a BST. single degree-4 node into three degree-2 nodes!! Use "internal" edges for 3- and 4- nodes. red glue not 1-1 because 3-nodes swing eir way. So now we have two kinds of edges: some which correspond to original edges in tree (we all se black edges), and Red-Black some! Correspondence which Tree are internal between in trees larger and red-black nodes in trees tree (we call se red edges). When we represent a node with 2 keys stored (degree 3), we actually have two choices: Represent tree as a BST.! Use "internal" edges for 3- and 4- nodes. red glue plitting Nodes not 1-1 because 3-nodes swing eir way. So suppose that we start with a tree, and turn it into a binary tree using above correspondences.! Correspondence What important between properties trees and dored-black we have Red-Black trees. that Tree: show this Splitting is a Nodes good binary tree? 1. In every path from root to any or node, every red edge is followed by a black edge Two easy cases. Switch colors. (i.e., two red edges never appear in a row). 2. For each node, all paths to its descendent leaves have same number of black edges. When this node is root r, this value is called black-depth. Note that se are both simple consequences of properties of trees we ve talked about and correspondences we chose. Toger, y imply that path from root to farst 14 Two hard cases. Use rotations. Red-Black Tree: Splitting Nodes 6 Two easy cases. Switch colors. do single rotation do double rotation

7 not not because because 3-nodes 3-nodes swing swing eir eir way. way.! trees and! Correspondence between trees and red-black trees. leaf is no more than twice as long as path from root to nearest leaf. This in turn implies that depth of tree is only O(log n), so lookups will be efficient (see Lemma 13.1 in book). So we just need to figure out a way of inserting which preserves this correspondence (or more importantly, preserves two important properties). This turns out to be a little complicated, and is main reason that red-black trees can seem a bit mysterious. Note that I m going to do this slightly differently than how it s done in book, in order to make correspondence to trees a bit clearer. I strongly suggest you also read book on this. Informally, we re going to split full nodes on way down (like in B-trees), while book (and most real implementations) first insert and n repair by moving up tree. Red-Black Tree: Splitting Nodes So suppose we re trying to insert. The basic idea is to do a normal binary tree insert, where new link is red (since it s like inserting into a node at bottom of tree). If this were a tree, if we saw a full node on our way down when inserting, we would split it. What does that looktwo like Two ineasy red-black cases. trees? Switch Somecolors. cases are easy: we can just switch colors (bold is red) But or Two Two cases hard hard are harder cases. Use switching Use rotations. colors will violate one of red-black requirements: do do single single rotation do do double double rotation In order to handle se cases, we need to use idea of rotations. Tree rotations are a way of changing binary search tree while still preserving search tree property. They re useful all over place, and are used in search tree data structures beyond just red-black trees. The basic idea is simple, as following picture shows. 7

8 p u u p C A A B B C It turns out we can solve annoying cases of red-black trees by using eir one ( first case) or two ( second case) rotations. Let s see this through an example (proofs are in book read m!) Red-Black Tree: Splitting Nodes inserting G change colors X right rotate R! M left rotate E! L 17 We also need to handle some or complications even if we don t see any full nodes on our way down, inserting new nodered-black might mess up Tree: correspondence Insertion to trees and red-black properties. These can also be handled with Red-Black appropriate Tree: Balance tree rotations. For example, suppose we insert F (E in se notes since I ve copied m): Property A. Every path from root to leaf has same number of black links. Sym Property B. At most one red link in-a-row. E Property C. Height of tree is less than 2 lg N + 2. There are a few or complications, but this should illustrate basic idea. A red-black tree is a particular binary tree representation of a tree, and by carefully designing our insert and X A Implementatio Sorted array Unsorted list Hashing BST Randomized BS Splay Red-Black 8 M P Note. Compari 19

9 delete algorithm to respect this representation, we can get a binary search tree with O(log n) for lookups, inserts, and deletes. As mentioned, book does this slightly differently, which obscures relationship to trees (although it s still true) but makes red-black trees mselves quite a bit simpler. Please make sure you understand what s going on in book. 9