Algorithms. Algorithms 3.4 HASH TABLES. hash functions separate chaining linear probing context. Premature optimization. Hashing: basic plan

Size: px

Start display at page:

Download "Algorithms. Algorithms 3.4 HASH TABLES. hash functions separate chaining linear probing context. Premature optimization. Hashing: basic plan"

Simon Fox
5 years ago
Views:

Algorithms ROBERT SEDGEWICK KEVIN WAYNE Premature optimization Algorithms F O U R T H E D I T I O N.

and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered.

ROBERT SEDGEWICK KEVIN WAYNE http://algs4.cs.princeton.

1 Algorithms ROBERT SEDGEWICK KEVIN WAYNE Premature optimization Algorithms F O U R T H E D I T I O N.4 HASH TABLES hash functions separate chaining linear probing context Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical %. ROBERT SEDGEWICK KEVIN WAYNE Last updated on Mar 4, 5, 9:56 AM Symbol table implementations: summary Hashing: basic plan Save items in a -indexed table (index is a function of the ). implementation guarantee average case search insert delete search hit insert delete ordered ops? interface Hash function. Method for computing array index from. sequential search (unordered list) binary search (ordered array) N N N N N N equals() log N N N log N N N compareto() BST N N N log N log N N compareto() red-black BST log N log N log N log N log N log N compareto() Issues. hash("times") = Computing the hash function. Equality test: Method for checking whether two s are equal. hash("it") = Collision resolution: Algorithm and data structure to handle two s that hash to the same array index.?? "it" 4 5 Q. Can we do better? A. Yes, but with different access to the data. Classic space-time tradeoff. No space limitation: trivial hash function with as index. No time limitation: trivial collision resolution with sequential search. Space and time limitations: hashing (the real world). 4

Computing the hash function.4 HASH TABLES Idealistic goal. Scramble the s uniformly to produce a table index. Efficiently computable. Each table index equally likely for each.

Better: last three digits. table index 57 = California, 574 = Alaska (assigned in chronological order within geographic region) ROBERT SEDGEWICK KEVIN WAYNE http://algs4.cs.princeton.

2 Computing the hash function.4 HASH TABLES Idealistic goal. Scramble the s uniformly to produce a table index. Efficiently computable. Each table index equally likely for each. thoroughly researched problem, still problematic in practical applications Algorithms hash functions separate chaining linear probing context Ex. Social Security numbers. Bad: first three digits. Better: last three digits. table index 57 = California, 574 = Alaska (assigned in chronological order within geographic region) ROBERT SEDGEWICK KEVIN WAYNE Practical challenge. Need different approach for each type. 6 Hash tables: quiz Java s hash code conventions Which of the following would be a good hash function for U.S. phone numbers to integers between and 999? All Java classes inherit a method hashcode(), which returns a -bit int. A. First three digits. B. Second three digits. C. Last three digits. D. Either B or C. (69) Requirement. If x.equals(y), then (x.hashcode() == y.hashcode()). Highly desirable. If!x.equals(y), then (x.hashcode()!= y.hashcode()). x y E. I don't know. x.hashcode() y.hashcode() Default implementation. Memory address of x. Legal (but poor) implementation. Always return 7. Customized implementations. Integer, Double, String, File, URL, Date, User-defined types. Users are on their own. 7 8

3 Implementing hash code: integers, booleans, and doubles Implementing hash code: strings Java library implementations public final class Integer private final int value; public int hashcode() return value; public final class Boolean private final boolean value; public int hashcode() if (value) return ; else return 7; public final class Double private final double value; public int hashcode() long bits = doubletolongbits(value); return (int) (bits ^ (bits >>> )); convert to IEEE 64-bit representation; xor most significant -bits with least significant -bits Warning: -. and +. have different hash codes 9 Treat string of length L as L-digit, base- number: h = s[] L + + s[l ] + s[l ] + s[l ] public final class String private final char[] s; public int hashcode() int hash = ; for (int i = ; i < length(); i++) hash = s[i] + ( * hash); return hash; Java library implementation Horner's method: only L multiplies/adds to hash string of length L. String s = "call"; s.hashcode(); 4598 = = 8 + (8 + (97 + (99))) char Unicode 'a' 97 'b' 98 'c' 99 Implementing hash code: strings Implementing hash code: user-defined types Performance optimization. Cache the hash value in an instance variable. Return cached value. public final class String private int hash = ; private final char[] s; cache of hash code public final class Transaction implements Comparable<Transaction> private final String who; private final Date when; private final double amount; public Transaction(String who, Date when, double amount) /* as before */ public int hashcode() int h = hash; if (h!= ) return h; for (int i = ; i < length(); i++) h = s[i] + ( * h); hash = h; return h; Q. What if hashcode() of string is? return cached value store cache of hash code hashcode() of "pollinating sandboxes" is public boolean equals(object y) /* as before */ public int hashcode() nonzero constant int hash = 7; hash = *hash + who.hashcode(); hash = *hash + when.hashcode(); hash = *hash + ((Double) amount).hashcode(); return hash; typically a small prime for reference types, use hashcode() for primitive types, use hashcode() of wrapper type

4 Hash code design Hash tables: quiz "Standard" recipe for user-defined types. Combine each significant field using the x + y rule. If field is a primitive type, use wrapper type hashcode(). If field is null, use. If field is a reference type, use hashcode(). If field is an array, apply to each entry. applies rule recursively or use Arrays.deepHashCode() Which of the following is an effective way to map a hashable to an integer between and M-? A. return.hashcode() % M; x In practice. Recipe above works reasonably well; used in Java libraries. In theory. Keys are bitstring; "universal" family of hash functions exist. Basic rule. Need to use the whole to compute hash code; consult an expert for state-of-the-art hash codes. awkward in Java since only one (deterministic) hashcode() B. return Math.abs(.hashCode()) % M; C. Both A and B. D. Neither A nor B. E. I don't know. x.hashcode() hash(x) 4 Modular hashing Uniform hashing assumption Hash code. An int between - and -. Hash function. An int between and M - (for use as array index). Uniform hashing assumption. Each is equally likely to hash to an integer between and M -. typically a prime or power of return.hashcode() % M; x Bins and balls. Throw balls uniformly at random into M bins. bug return Math.abs(.hashCode()) % M; x.hashcode() in-a-billion bug hashcode() of "polygenelubricants" is - hash(x) Birthday problem. Expect two balls in the same bin after ~ π M / tosses. Coupon collector. Expect every bin has ball after ~ M ln M tosses. return (.hashcode() & x7fffffff) % M; correct Load balancing. After M tosses, expect most loaded bin has ~ ln M / ln ln M balls. 5 6

5 Uniform hashing assumption Uniform hashing assumption. Each is equally likely to hash to an integer between and M -. Bins and balls. Throw balls uniformly at random into M bins..4 HASH TABLES Algorithms hash functions separate chaining linear probing context ROBERT SEDGEWICK KEVIN WAYNE Hash value frequencies for words in Tale of Two Cities (M = 97) Java's String data uniformly distribute the s of Tale of Two Cities 7 Collisions Separate-chaining symbol table Collision. Two distinct s hashing to same index. Birthday problem can't avoid collisions. Coupon collector not too much wasted space. Load balancing no index gets too many collisions. unless you have a ridiculous (quadratic) amount of memory Use an array of M < N linked lists. [H. P. Luhn, IBM 95] Hash: map to integer i between and M -. Insert: put at front of ith chain (if not already in chain). Search: sequential search in ith chain. separate-chaining hash table (M = 4) put(l, ) hash(l) = hash("it") = I 8 D hash("times") =?? "it" 4 5 K J 9 E 4 A H 7 G 6 Challenge. Deal with collisions efficiently. FL 5 C B 9

6 Separate-chaining symbol table Separate-chaining symbol table: Java implementation Use an array of M < N linked lists. [H. P. Luhn, IBM 95] Hash: map to integer i between and M -. Insert: put at front of ith chain (if not already in chain). Search: sequential search in ith chain. separate-chaining hash table (M = 4) I 8 D get(e) hash(e) = public class SeparateChainingHashST<Key, Value> private int M = 97; // number of chains private Node[] st = new Node[M]; // array of chains private static class Node private Object ; private Object val; private Node next; no generic array creation (declare and value of type Object) return (.hashcode() & x7fffffff) % M; array doubling and halving code omitted K J 9 E 4 A H 7 G 6 public Value get(key ) int i = hash(); for (Node x = st[i]; x!= null; x = x.next) if (.equals(x.)) return (Value) x.val; return null; L F 5 C B Separate-chaining symbol table: Java implementation Analysis of separate chaining public class SeparateChainingHashST<Key, Value> private int M = 97; // number of chains private Node[] st = new Node[M]; // array of chains private static class Node private Object ; private Object val; private Node next; return (.hashcode() & x7fffffff) % M; Proposition. Under uniform hashing assumption, prob. that the number of s in a list is within a constant factor of N / M is extremely close to. Pf sketch. Distribution of list size obeys a binomial distribution. (,.5).5 Binomial distribution (N = 4, M =, = ) public void put(key, Value val) int i = hash(); for (Node x = st[i]; x!= null; x = x.next) if (.equals(x.)) x.val = val; return; st[i] = new Node(, val, st[i]); Consequence. Number of probes for search/insert is proportional to N / M. M too large too many empty chains. M too small chains too long. equals() and hashcode() M times faster than sequential search Typical choice: M ~ ¼ N constant-time ops. 4

7 Resizing in a separate-chaining hash table Deletion in a separate-chaining hash table Goal. Average length of list N / M = constant. Double size of array M when N / M 8; halve size of array M when N / M. Note: need to rehash all s when resizing. x.hashcode() does not change; but hash(x) can change Q. How to delete a (and its associated value)? A. Easy: need to consider only chain containing. before resizing (N/M = 8) before deleting C after deleting C A B C D E F G H I J K L M N O P K I P N L K I P N L after resizing (N/M = 4) J F C B J F B K I O M O M P N L E J F C B A O M H G D 5 6 Symbol table implementations: summary implementation guarantee average case search insert delete search hit insert delete ordered ops? interface sequential search (unordered list) N N N N N N equals().4 HASH TABLES binary search (ordered array) log N N N log N N N compareto() BST N N N log N log N N compareto() red-black BST log N log N log N log N log N log N compareto() Algorithms hash functions separate chaining linear probing context separate chaining N N N * * * equals() hashcode() ROBERT SEDGEWICK KEVIN WAYNE * under uniform hashing assumption 7

8 Collision resolution: open addressing Linear-probing hash table summary Open addressing. [Amdahl-Boehme-Rocherster-Samuel, IBM 95] Maintain s and values in two parallel arrays. When a new collides, find next empty slot, and put it there. Hash. Map to integer i between and M. Insert. Put at table index i if free; if not try i +, i +, etc. Search. Search table index i; if occupied but no match, try i +, i +, etc. Note. Array size M must be greater than number of -value pairs N. linear-probing hash table (M = 6, N =) s[] P M A C H L E R X s[] P M A C S H L E R X put(k, 4) hash(k) = 7 K 4 M = 6 vals[] Linear-probing symbol table: Java implementation Linear-probing symbol table: Java implementation public class LinearProbingHashST<Key, Value> private int M = ; private Value[] vals = (Value[]) new Object[M]; private Key[] s = (Key[]) new Object[M]; array doubling and halving code omitted public class LinearProbingHashST<Key, Value> private int M = ; private Value[] vals = (Value[]) new Object[M]; private Key[] s = (Key[]) new Object[M]; /* as before */ /* as before */ private void put(key, Value val) /* next slide */ private Value get(key ) /* prev slide */ public Value get(key ) for (int i = hash(); s[i]!= null; i = (i+) % M) if (.equals(s[i])) return vals[i]; return null; sequential search in chain i public void put(key, Value val) int i; for (i = hash(); s[i]!= null; i = (i+) % M) if (s[i].equals()) break; s[i] = ; vals[i] = val; sequential search in chain i

Clustering Knuth's parking problem Cluster. A contiguous block of items. Observation. New s likely to hash into middle of big clusters. Model. Cars arrive at one-way street with M parking spaces.

With M cars, mean displacement is ~ π M / 8. Key insight. Cannot afford to let linear-probing hash table get too full. 4 Analysis of linear probing Resizing in a linear-probing hash table Proposition.

9 Clustering Knuth's parking problem Cluster. A contiguous block of items. Observation. New s likely to hash into middle of big clusters. Model. Cars arrive at one-way street with M parking spaces. Each desires a random space i : if space i is taken, try i +, i +, etc. Q. What is mean displacement of a car? displacement = Half-full. With M / cars, mean displacement is ~ 5 /. Full. With M cars, mean displacement is ~ π M / 8. Key insight. Cannot afford to let linear-probing hash table get too full. 4 Analysis of linear probing Resizing in a linear-probing hash table Proposition. Under uniform hashing assumption, the average # of probes in a linear probing hash table of size M that contains N = α M s is: + + ( ) Goal. Average length of list N / M ½. Double size of array M when N / M ½. Halve size of array M when N / M ⅛. Need to rehash all s when resizing. search hit search miss / insert Pf. before resizing s[] vals[] E S R A Parameters. M too large too many empty array entries. M too small search time blows up. Typical choice: α = N / M ~ ½. # probes for search hit is about / # probes for search miss is about 5/ 5 after resizing s[] vals[] A S E R 6

10 Deletion in a linear-probing hash table ST implementations: summary Q. How to delete a (and its associated value)? A. Requires some care: can't just delete array entries. implementation guarantee average case search insert delete search hit insert delete ordered ops? interface before deleting S sequential search (unordered list) N N N N N N equals() s[] P M A C S H L E R X binary search (ordered array) log N N N log N N N compareto() vals[] BST N N N log N log N N compareto() after deleting S? doesn't work, e.g., if hash(h) = red-black BST log N log N log N log N log N log N compareto() separate chaining N N N * * * equals() hashcode() s[] vals[] P M A C H L E R X linear probing N N N * * * equals() hashcode() 7 * under uniform hashing assumption 8 -SUM (REVISITED) -SUM. Given N distinct integers, find three such that a + b + c =. Goal. N expected time case, N extra space. Hashing-based solution to -SUM. Insert each integer into a hash table. For each pair of integers a and b, search hash table for c = (a + b). (assuming c a and c b) Algorithms.4 HASH TABLES hash functions separate chaining linear probing context ROBERT SEDGEWICK KEVIN WAYNE 9

11 War story: algorithmic complexity attacks War story: algorithmic complexity attacks Q. Is the uniform hashing assumption important in practice? A Java bug report. A. Obvious situations: aircraft control, nuclear reactor, pacemaker, HFT, A. Surprising situations: denial-of-service attacks. Jan Lieskovsky -- ::47 EDT Description Julian Wälde and Alexander Klink reported that the String.hashCode() hash function is not sufficiently collision resistant. hashcode() value is used in the implementations of HashMap and Hashtable classes: malicious adversary learns your hash function (e.g., by reading Java API) and causes a big pile-up in single slot that grinds performance to a halt Real-world exploits. [Crosby-Wallach ] Bro server: send carefully chosen packets to DOS the server, using less bandwidth than a dial-up modem. Perl 5.8.: insert carefully chosen strings into associative array. Linux.4. kernel: save files with carefully chosen names. A specially-crafted set of s could trigger hash function collisions, which can degrade performance of HashMap or Hashtable by changing hash table operations complexity from an expected/average O() to the worst case O(n). Reporters were able to find colliding strings efficiently using equivalent substrings and meet in the middle techniques. This problem can be used to start a denial of service attack against Java applications that use untrusted inputs as HashMap or Hashtable s. An example of such application is web application server (such as tomcat, see bug #755) that may fill hash tables with data from HTTP request (such as GET or POST parameters). A remote attack could use that to make JVM use excessive amount of CPU time by sending a POST request with large amount of parameters which hash to the same value. This problem is similar to the issue that was previously reported for and fixed in e.g. perl: Algorithmic complexity attack on Java Diversion: one-way hash functions Goal. Find family of strings with the same hashcode(). One-way hash function. "Hard" to find a that will hash to a desired Solution. The base- hash code is part of Java's String API. value (or two s that hash to same value). Ex. MD4, MD5, SHA-, SHA-, SHA-, WHIRLPOOL, RIPEMD-6,. hashcode() hashcode() hashcode() known to be insecure "Aa" "AaAaAaAa" "BBAaAaAa" "BB" "AaAaAaBB" "BBAaAaBB" "AaAaBBAa" "AaAaBBBB" "BBAaBBAa" "BBAaBBBB" String password = args[]; MessageDigest sha = MessageDigest.getInstance("SHA"); byte[] bytes = sha.digest(password); "AaBBAaAa" "AaBBAaBB" "BBBBAaAa" "BBBBAaBB" /* prints bytes as hex string */ "AaBBBBAa" "BBBBBBAa" "AaBBBBBB" "BBBBBBBB" N strings of length N that hash to same value! Applications. Crypto, message digests, passwords, Bitcoin,. Caveat. Too expensive for use in ST implementations. 4 44

Separate chaining vs. linear probing Hashing: variations on the theme Separate chaining. Performance degrades gracefully. Clustering less sensitive to poorly-designed hash function. Linear probing.

12 Separate chaining vs. linear probing Hashing: variations on the theme Separate chaining. Performance degrades gracefully. Clustering less sensitive to poorly-designed hash function. Linear probing. Less wasted space. Better cache performance. 4 A 8 null X 7 L E S P Many improved versions have been studied. Two-probe hashing. [ separate-chaining variant ] Hash to two positions, insert in shorter of the two chains. Reduces expected length of the longest chain to ~ lg ln N. Double hashing. [ linear-probing variant ] Use linear probing, but skip a variable amount, not just each time. Effectively eliminates clustering. Can allow table to become nearly full. More difficult to implement delete. s[] vals[] P M A C S H L E R X M 9 H 5 C 4 R Cuckoo hashing. [ linear-probing variant ] Hash to two positions; insert into either position; if occupied, reinsert displaced into its alternative position (and recur). Constant worst-case time for search Hash tables vs. balanced search trees Hash tables. Simpler to code. No effective alternative for unordered s. Faster for simple s (a few arithmetic ops versus log N compares). Better system support in Java for String (e.g., cached hash code). Balanced search trees. Stronger performance guarantee. Support for ordered ST operations. Easier to implement compareto() correctly than equals() and hashcode(). Java system includes both. Red-black BSTs: java.util.treemap, java.util.treeset. Hash tables: java.util.hashmap, java.util.identityhashmap. linear probing separate chaining 47

sequential search (unordered list) binary search (ordered array) BST N N N 1.39 lg N 1.39 lg N N compareto()

Algorithms Symbol table implementations: summary implementation guarantee average case search insert delete search hit insert delete ordered ops? interface.4 HASH TABLES sequential search (unordered list)