Computational Biology 6.095/6.895 Database Search Lecture 5 Prof. Piotr Indyk

Transcription

1 Computational Biology 6.095/6.895 Database Searh Leture 5 Prof. Piotr Indyk

2 Previous letures Leture -3: Global alignment in O(mn) Dynami programming Leture -2: Loal alignment, variants, in O(mn) Leture -1: Exat string mathing in O(n) Hashing: number mod q Quiz: What do these problems have in ommon? Answer: they enable omparison of two sequenes

3 The Big Piture Gene Finding DNA Sequene alignment Database lookup

4 Database searh Database searh: AIKWQPRSTW. Database IKMQRHIKW. HDLFWHLWH. Query: RGIKW Output: sequenes similar to query

5 What does similar mean? Simplest idea: just ount the number of ommon amino-aids E.g., RGRKW mathes RGIKW at 4 positions, or idper = 80% Not all mathes are reated equal - soring matrix In general,we should allow insertions and deletions as well

6 How to answer the query We ould just san the whole database But: Query must be very fast Most sequenes will be ompletely unrelated to query Individual alignment needs not be perfet. Can finetune Exploit nature of the problem If you re going to rejet any math with idper < 90%, then why bother even looking at sequenes whih don t have a fairly long streth of mathing a.a. in a row.

7 W-mer indexing IKW Preproessing: For every W-mer (e.g., IKZ W=3) store every loation in the database where it ours (an use hashing if W is large) Query: Generate W-mers and look them up in the database. Proess the results Running time benefit: RGIKW AIKWQPRSTW. IKMQRHIKW. HDLFWHLWH. For W=3, if the sequenes are IKW random, then roughly one W-mer in 23 3 will math, i.e., one in a ten... thousand We hit only a small fration of all sequenes

8 BLAST Speifi (and very effiient) implementation of the W-mer indexing idea How to generate W-mers from the query How to proess the mathes Basi loal alignment searh tool SF Altshul, W Gish, W Miller, EW Myers, DJ Lipman - J. Mol. Biol, inaoep.mx... Basi Loal Alignment Searh Tool Stephen F. Altshul', Warren Gish', Webb Miller2 Eugene W. Myers3 and David J. Lipmanl... Page S. F. Altshul et al.... Cited by 14181

9 THE BLAST SEARCH ALGORITHM Query word (W =3) Query: GSVEDTTGSQSLAALLNKCKTPQGQRLVNQWIKQPLMDKNRIEERLNLVEAFVEDAELRQTLQEDL Neighborhood words PQG PEG PRG PKG PNG PDG PHG PMG PSG PQA PQN Neighborhood sore threshold (T=13) et... X Query: 325 SLAALLNKCKTPQGQRLVNQWIKQPLMDKNRIEERLNLVEA 365 +LA++L+ TP G R++ +W+ P+ D + ER + A Sbjt: 290 TLASVLDCTVTPMGSRMLKRWLHMPVRDTRVLLERQQTIGA 330 High-soring Segment Pair (HSP) Figure by MIT OCW.

10 Blast Algorithm Overview Reeive query Split query into overlapping words of length W Find neighborhood words for eah word until threshold T Look into the table where these neighbor words our: seeds Extend seeds until sore drops off under X Evaluate statistial signifiane of sore Report sores and alignments PMG W-mer Database

11 Extending the seeds Query: Sbjt: 325 SLAALLNKCKTPQGQRLVNQWIKQPLMDKNRIEERLNLVEA 365 +LA++L+ TP G R++ +W+ P+ D + ER + A 290 TLASVLDCTVTPMGSRMLKRWLHMPVRDTRVLLERQQTIGA 330 High-soring Segment Pair ( HSP) Cumulative Sore Figure by MIT OCW. Break into two HSPs Extend until the umulative sore drops Extension

12 Length and Perent Identity

13 Why this works. I.e., what do we miss? In the worst ase W-mer: W=3 No neighborhoods Query: RKIWGDPRS Datab.: RKIVGDRRS 7 idential a.a

14 Pigeonhole priniple Pigeonhole priniple If you have 2 pigeons and 3 holes, there must be at least one hole with no pigeon

15 Pigeonhole and W-mers RKI WGD PRS RKI VGD RRS Pigeonholing mis-mathes Two sequenes, eah 9 amino-aids, with 7 identities There is a streth of 3 amino-aids perfetly onserved In general: Sequene length: n Identities: t Can use W-mers for W= [n/(n-t+1)]

16 True alignments: Looking for K-mers Personal experiment run in Kb region of human, and mouse 450Kb ortholog. Blasted every piee of mouse against human (6,50) Identify slope of best fit line.two sets of blast alignments. 320 olinear / 770 alignments Can ask the question: What makes a blast hit on the line look good. What makes a blast hit off the diagonal look bad. Count K-mers How many k-mers do we find: n How long are they: k Counted their distribution inside and outside the sequene.

17 True alignments: Looking for K-mers number of k-mers that happen for eah length of k-mer. Red islands ome from olinear alignments Blue islands ome from off-diagonal alignments Note: more than one data point per alignment. Linear plot Log Log plot

18 Extensions Ideas beyond W-mer indexing? Faster Better sensitivity (less false negatives)

19 Extensions: Filtering Low omplexity regions an ause spurious hits Filter out low omplexity in your query Filter most over-represented items in your database

20 Extensions: Two-hit blast Improves sensitivity for any speed Two smaller W-mers are more likely than one longer one Therefore it s a more sensitive searhing method to look for two hits instead of one, with the same speed. Improves speed for any sensitivity No need to extend a lot of the W-mers, when isolated

21 Extensions: beyond W-mers W-mers (without neighborhoods): RGIKW RGI, GIK, IKW No reason to use only onseutive symbols Instead, we ould use ombs, e.g., RGIKW R*IK*, RG**W, Indexing same as for W-mers: For eah omb, store the list of positions in the database where it ours Perform lookups to answer the query How to hoose the ombs? Randomized projetion: Califano-Rigoutsos 93, Buhler 01, Indyk-Motwani 98 Choose the positions of * at random Analyze false positives and false negatives

22 Combs and Random Projetions Assume we selet k positions, whih do not ontain *, at random with replaement What is the probability of a false negative? At most: 1-idper k In our ase: 1-(7/9) 4 = What is we repeat the proess l times, independently? Miss prob. = 0.63 l For l=5, it is less than 10% Query: RKIWGDPRS Datab.: RKIVGDRRS k=4 Query: *KI*G***S Datab.: *KI*G***S

23 Suffix trees Great tool for text proessing E.g., searhing for exat ourrene of a pattern Suffix tree for: xabxa a x b a x b x a a b x a

24 Suffix tree definition x a b x a a b x a b x a x a a 1 a x x b a a x 6 b b a x a 2 Definition: Suffix tree ST for text T[1..n] Rooted, direted tree T, n leaves, numbered 1..n Text labels on the edges Path to leaf i spells out the suffix S[i..], by onatenating edge labels Common prefixes share ommon paths, diverge to form internal nodes

25 Properties of suffix trees a x x b a a x 6 b b a x a 1 2 How muh spae do we need to represent a suffix tree of T[1..n]? Only O(n) At most O(n) edges Eah edge label an be represented as T[i j]

26 Exat string mathing with suffix trees Given the suffix tree for text T Searh for pattern P[1 m] For every harater in P, traverse T: xabxa the appropriate path of the tree, P: abx reading one harater eah time If P is not found in a path, P does not our in T x If P is found in its entirety, then all a ourrenes of P in T are exatly b the hildren of that node x Every hild orresponds to a x exatly one ourrene a Simply list eah of the leaf indies Time: O(m) 1 2 b a 4 5 b x 6 a 3

27 Suffix Tree Constrution x a b x a a b x a b x a x a a x a b x a a b x a b x a Running time: O(n 2 ) Can be improved to O(n)

28 Today Searh among many database sequenes W-mer indexing BLAST Combs and random projetions Suffix trees