Assignment 2 (programming): Problem Description

Transcription

1 CS2210b Data Structures and Algorithms Due: Monday, February 14th Assignment 2 (programming): Problem Description 1 Overview The purpose of this assignment is for students to practice on hashing techniques and hash table implementations. The application is an adapted version of the String Matching problem in the Intel 2009 Threading Challenge 1. Briefly, the String Matching problem in this contest is to write a (parallel) program to search a large set of DNA sequences (represented as strings of characters) for a given set of other DNA sequences. Dr. Bradley C. Kuszmaul (Massachusetts Institute of Technology) won the contest with a program written in Cilk++, an extension of C++ for parallel programing. Dr. Kuszmaul used Rabin-Karp hashing 2. In this assignment, students will implement a sequential Java program to perform the DNA sequence matching functionalities following Dr. Kuszmaul s techniques. The test suites will be built from the chromosome of Canis familiaris (domestic dog), available at 2 Problem Description The following is based on a report by Dr. Kuszmaul. The problem, as stated on Intel s web site is as follows. Write a threaded program to search a database of DNA sequences, represented as strings of characters, to find matches of other DNA subsequences. Both the database and the query strings are made up of only four characters: A, C, G and T. The output must report the location of an exact match within any given input DNA sequence for each input search query string. If the query string matches within multiple sequences within the database, each result must be reported; and if the query string matches multiple locations within the same database sequence, the earliest position that matches exactly must be reported. The file names for this problem (database file, queries file, output results) will be given on the command line. File formats: The input database and query files will have the same format. Each sequence will be prefixed by a line starting with a greater than character ( > ) followed by a description of the origin of the sequence of no more than 131 characters. The sequence will then begin on the next line for some number of lines. Each line will contain exactly string search algorithm 1

2 characters from the set A, C, G and T, except the last line, which may hold fewer than 80 characters. Following this last line will be the descriptor line from the next sequence until the end of the file has been reached, which will be signified by the descriptor ( >EOF ). For each query string contained within the second input file, the output file should print the descriptor of the query sequence and the descriptors of any database sequences that contain a match as well as the position within the database sequence of that exact match. If the query sequence string is not found within any database sequences, a message to that effect should be printed after the query descriptor. Thus the program takes as input two sets of strings: the Database strings and the Query strings. We will refer to these two sets of strings as the database and the querybase respectively. We make some assumptions about the input: Each string is less than 1,000,000 bytes 3. The entire database fits in main memory, but the entire querybase may be too big to fit in main memory 4. There are less than 2 32 strings in a database or a querybase 5. The output format is described in( -matching/topic/65663/). In that example, the database is: >DB description string 1 AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACT GGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATACTTTAACCAATATAGGCATAGCGCACAG ACAGATAAAAATTACA >DB description string 2 AACGGTGCGGGCTGACGCGTACAGGAAACACAGAAAAAAGCCCGCACCTGACAGTGCGGGCTTTTTTTTTCGACCAAAG GTAACGAGGTAACAACCATGCGAGTGTTGAAGTTCGGCGGTACATCAGTGGCAAATGCAGAACGTTTTCTGCGTGTTGC CGATATTCTGGAAAGCAATGC >EOF and an example querybase is: >Query description string 1 CATTCTGACTGCAA >Query description string 2 AAAAAAG >Query description string 3 GTAA >Query description string 4 AGAGAGAGAGAGAGAGAGAGAGAGAGAGAG >EOF

3 The output would then be Query description string 1 [DB description string 1] at offset 7 Query description string 2 [DB description string 1] at offset 47 [DB description string 2] at offset 33 Query description string 3 [DB description string 1] at offset 94 [DB description string 2] at offset 79 Query description string 4 NOT FOUND 3 Notation For describing the algorithm and analyzing performance, we define the following variables. If s is a string then s is the length of s and s i is the ith letter (starting from 0) of s while s <i,j> is the substring of s starting at position i (inclusive) through position j (exclusive). Q is the set of all query strings. D is the set of all database strings. T(Q) is the total size of all the query strings. That is T(Q) = q Q q T(D) is the total size of all the database strings. Q is the number of query strings. D is the number of database strings. O is Big-Oh notation. 4 Naive Serial Algorithm The simplest algorithm for implementing this string matching is to loop over combinations of the query strings and database strings and compare the strings. The algorithm is as follows. 3

4 Naive() 1 for q Q 2 for d D 3 for i below d q +1 4 for j below q 5 if d i+j q j then 6 continue to the next iteration of the i loop. 7 Match found for q at offset i of d. 8 continue to the next iteration of the outer loop. 9 No match found for q. In the worst case, the total number of instructions for the naive algorithm is O(T(D) T(Q)). In the worst case, essentially every character in the querybase needs to be compared to essentially every character in the database. Thus the total number of instructions can be quadratic in the input size. This case occurs when the querybase and the database are about the same size. 5 Rabin-Karp Matching There are algorithms for string matching that are much faster than the naive algorithm. For example, to match a single query string, the Knuth-Morris-Pratt algorithm [1] can be performed in O(T(Q) + T(D)) instructions, and the Boyer-Moore algorithm [2] is often even faster. However, it is not clear how to use those algorithms to search for multiple query strings quickly. Therefore, Dr. Kuszmaul used Rabin-Karp hashing [3]. 5.1 Rabin-Karp rolling hashing Rabin-Karp hashing is a technique for computing a hash of a string. Given the Rabin-Karp hash of a k-element substring of a text, one can quickly compute the hash of successive k-element substrings. For example, if we have the string ABCDEFG, we can quickly compute the hashes of ABCD, of BCDE, CDEF, and DEFG respectively. This kind of hash is sometimes called a rolling hash. Rabin-Karp hashing works as follows. First we chose a prime number P. The hash of a string q is computed as h(q i )P q i 1 (1) 0 i< q where h(q i ) is a non-negative integer less than P. In this case, since our alphabet consists of the four letters A, C, T, G we can choose P = 5, and h(a) = 0, h(c) = 1, h(t) = 2, h(a) = 3. Thus, if we have the string we can encode it as TATGTGAGAAGA,

5 To compute the hash of the first four characters we have hash(2023) = = = 261. To compute the hash of the next 4-character subsequence we have hash(0232) = = = 57. But that can be computed incrementally by multiplying the previous hash by 5, adding in the newly added character and subtracting the newly removed character. Thus in general we have hash(0232) = hash(2023) = = 57 hash(s <i+1,j+1> ) = (hash(s <i,j> )) P h(s i ) P j i +h(s j+1 ). (2) Since we know the length of the s in advance (4 in this example), we can compute P s in advance (in this example 5 4 = 625). Furthermore, we can do this arithmetic in a fixed-size word (e.g., in a 32-bit int in Java), and it will work out. 5.2 Rabin-Karp Searching We can thus implement the search as SimpleRabinKarp() 1 for q Q 2 h q hash(q) 3 for d D 4 for i below d q +1 5 h d hash(d <i,i+ q > ) 6 if h q = h d then 7 Match may be found for q at offset i of d. 8 Verify that it is really a match. 9 if it is really a match then 10 continue to the next iteration of the outer loop 11 No match found for q. If we are very unlucky, the hash will match when the strings do not really match (a so-called false match). In that case we ll do the work to verify a string that isn t actually a 5

6 match. We don t have to worry about the work to verify a string that actually is a match, because for each query string we stop searching d after the first match. In the databases that Dr. Kuszmaul worked with, he never encountered a false match on an entire query string. For SimpleRabinKarp, in the usual case, the number of instructions executed is O(T(Q)+ Q T(D)), since the database must be scanned once for every query string. 5.3 Dr. Kuszmaul s Rabin-Karp Searching We can do even better in many cases, however. For example, if all the query strings are the same size then we can compute hash(d <i,i+ q > ) only once and compare it to all the query hashes. We can store the query hashes in a hash table, so that all of those comparisons can run in O(1) instructions. In this case the total number of instructions executed is O(T(Q) + T(D)), which is asymptotically optimal. In practice, query strings are not all the same size. Dr. Kuszmaul applied the following heuristic to reduce the number of passes through each database string. He organized the query strings into groups, where the strings within each group are all within a factor of two of each other in size. Thus there are O(log Q ) groups. For a group G, if s is the shortest string G, then he used the hashes of the prefixes of length s for every string in the group. For example, if we have a group {ATAT, GATTA, CTAAT}, then the shortest string is of length 4. Instead of looking for hash matches against every element of the group, we look for matches against the length-4 prefixes of the group. So we look for hashes matching {ATAT, GATT, CTAA}. This increases the number of false matches, but reduces the number of passes over the database strings from Q to log Q. Dr. Kuszmaul found that for DNA sequences using a prefix is a good tradeoff, making the program run faster. This grouping is implemented by sorting the query strings by length. The system looks at the shortest string, and uses its length m. The system then includes all strings of length less than m in a group. The group is called an m-group. Then the system looks at the shortest string not in any group. That string s length n is guaranteed to 2m n. The system creates an m-group. And so forth, producing at most O(logN) groups where N is the longest string in the querybase. When looking for any string in an m-group the system hashes only the first m characters, and then simply compares the rest. One problem of using a prefix is that there are some bad cases that act like the naive algorithm. This can happen when we are using only the first k characters of a string s, but that substring is repeated several times. For example if we had 9 A s and one T as AAAAAAAAAT, and that we are only using the first 5 A s to match. We would encounter false matches many times (4 in this case) on the substring at offsets 0, 1, 2, and 3. We can avoid this false prefix matching by adopting the rule that if the grouping would produce a situation where a string s is in an m-group, and the substring s <0,m> is repeated as later in s, then the system creates a s -group. This gives us the property we need. If a string in an m-group is matched on the first m characters, then the hash won t match a later substring, so the system will not do much work comparing the same characters over and over. If the query strings don t exhibit too much self-matching, but all the queries are different sizes, then the resulting instruction count is O(T(Q)+T(D)log Q ). 6

7 Integrate the above techniques into Algorithm SimpleRabinKarp, we get the following improved algorithm for DNA sequence matching. KuszmaulRabinKarp() //Encode strings 1 Read database D into main memory (and find lengths). 2 Read querybase Q into main memory (and find lengths). //Preprocess the strings 3 for q Q 4 Compute the fingerprint of q. 5 Sort the querybase into ascending length. 6 Group the querybase strings by size and self-matchingness. 7 for q Q 8 L the length of the shortest string in q s group. 9 s q hash(s <0,L> ) 10 Store s q in a hash table. // Now start the actual search 11 for d D 12 for L {the unique shortest lengths} 13 for i below d L 14 h hash(d <i,i+l> ) 15 for q in the hash table matching h 16 if q is not already matched in d and i+ q d and the strings match then 17 Record the match of q at i on d. // Print matches 18 Sort the matches. 19 Print the sorted matches. 5.4 Special Case of Rabin-Karp Searching From the above introduction we notice that a special case where all the query strings are the samesizeisafundamental one. Wenameithereas equal-sizequerystrings, inshortesqs. If allthequery strings arethesame size q, we canhashthequery strings andstorethe query hashes in a hash table. For each string d in the database, we can compute hash(d <i,i+ q > ) only once and compare it to all the query hashes, and each comparison can run in O(1) instructions. In this case the total number of instructions executed is O(T(Q)+T(D)), which is asymptotically optimal. Dr. Kuszmaul s trick is to reduce the general case where query strings are not all the same size to the one which can be handled like ESQS s. Therefore, we focus on the matching of ESQS s, since it is the core of Algorithm KuszmaulRabinKarp. In this assignment, we will implement the following special algorithm for DNA sequence matching based on Algorithm KuszmaulRabinKarp. 7

8 EsqsRabinKarp() //Encode strings 1 Read database D into main memory (encode strings and find their lengths). 2 Read querybase Q into main memory (encode strings and find their lengths). //Hash the query strings 3 L the length of the query string 4 for q Q 5 s q hash(s <0,L> ) 6 Store q with hash value of s q in a hash table. // Now start the actual search 7 for d D 8 for i below d L 9 h hash(d <i,i+l> ) 10 for q in the hash table matching h 11 if q is not already matched in d and i+l d and the strings match then 12 Record the match of q at i on d. 13 Print the matches. References [1] Donald E. Knuth, J.H. Morris, and Vaughn B. Pratt. Fast pattern matching in strings. SIAM Journal of Computing, 6(2):323350, [2] Robert S. Boyer and J.S. Moore. A fast string searching algorithm. Communications of the ACM, 20(10):762772, October [3] Richard M. Karp and Michael O. Rabin. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development, 31, March