Introduction. Resources

Itroductio Data Structures ad Algorithms Alexadre Duret-Lutz & Aupam Gupta adl@lrde.epita.fr, aupamg@iitk.ac.i September 9, 2010 Objective To acquire a algorithmic thikig to solve problems. Meas Practical culture: learig basic data structures learig classical algorithms for commo problems learig desig strategies for algorithms Theoretical culture: learig to reaso about algorithms (provig that a algorithm does what it is desiged to do, aalyzig its complexity,...) ADL & AG Data Structures ad Algorithms 1 / 133 ADL & AG Data Structures ad Algorithms 2 / 133 Pla for the course Resources First half of the semester (we me): Week 1 (this week): itroducig cocepts by studyig how real-life algorithms ca be adapted to computers, Weeks 23 Deig the complexity of algorithms, ad itroducig tools to compute complexity. Weeks 24 May algorithms to sort data. Weeks 56 Data structures. mid semester exams Secod half of the semester (with Aupam Gupta): Lecture otes for this course (this documet) i http://www.lrde.epita.fr/~adl/es/iitj/eso211/. Itroductio to Algorithms (3rd editio), by Thomas Corme, Charles Leiserso, Roald Rivest, ad Cliord Stei. You ca d may web pages, ad books dedicated to the topics discussed here. Commo desig strategies for algorithms Graph algorithms ADL & AG Data Structures ad Algorithms 3 / 133 ADL & AG Data Structures ad Algorithms 4 / 133

Lookig up a word i a dictioary (1/3) We ca thik of may algorithms: 1 You have ever see a dictioary i your life ad do ot kow how it is orgaized. Algo 1: Read the pages oe by oe util you d the word. Note: you ca read the pages i ay order as log as you read them all. (You may wat to tear the pages you have read if you use a radom order.) 2 You kow a dictioary is alphabetically sorted. Algo 2: Ope the dictioary i the middle. Look at the rst word of the right page, ad usig the order o words tear the half of the dictioary that caot cotai the words you are lookig. Repeat util you are left with oe page.. 3 You have a idea of the statistical distributio of words i the dictioary. Algo 3: If your word starts with D, you may wat to ope the dictioary ear the begiig. ADL & AG Data Structures ad Algorithms 5 / 133 Lookig up a word i a dictioary (3/3) Ca we compare the eciecy of these three algorithms? We ca give a dictioary to three people, ad ask them to look up the same word usig each a dieret algorithm. This is oly oe test case: it does ot mea aythig. For istace if we ask the three people to lookup for the word A, Algo 1 will termiate rst sice this is the rst word of the dictioary. Should we coclude that Algo 1 is better tha the other two algorithms? Ca we dee a best case ad a worst case for each algorithm? Ca we evaluate the speed of a algorithm idepedetly of the people executig it (some people are faster tha others, but that should ot iuece our algorithm compariso). Ca we dee a average case? ADL & AG Data Structures ad Algorithms 7 / 133 Lookig up a word i a dictioary (2/3) We obviously expect Algo 1 to be slower tha Algo 2 itself slower tha Algo 3. Algo 1 works eve if the dictioary is ot sorted: data ca be arraged i ay way. Algo 2 & 3 are faster because the data is orgaized i a way that helps: the words of the dictioary are sorted. I geeral is it always good to orgaize items i a way that ease operatios o these items. 1 Algo 3 requires further kowledge o the data. Without this kowledge, you ca make a educated guess o the distributio (e.g. uiform), because it is likely to speedup the search ayway. Such a approximatio is called a heuristic. 1 For aother example: thik of the way a library is orgaized i order to make the followig three operatios eciet: search a book i the library, remove a book from the library, ad isert it back. ADL & AG Data Structures ad Algorithms 6 / 133 Searchig for a track o a music tape A dictioary ca be ope aywhere: it takes the same time. A music tape (or audio cassette) works dieretly: if you are asked to play the fourth track, you rst have to fast forward the tape to the start of that track (it costs some time) ad the you ca actually play that track. If a dictioary was recorded o the tape: you would ot aturally cosider the use Algo 2 or 3, because of the time it take to seek to the middle to tape, read a word, the seek elsewhere, etc. If the dictioary ll the whole tape, ca you compute the distace of the tape we will have to fast-forward or rewid i the worst case durig the executio of Algo 2? I computer terms, we say that a dictioary has a radom access (i.e. immediate access to arbitrary locatios) while a tape (without rewid ad fast-forward) oly has sequetial access: elemets ca oly be accessed i a predeed order. ADL & AG Data Structures ad Algorithms 8 / 133

Searchig a card i a deck of cards You are give a (shued) deck of cards, ad asked to remove the Jack of spades. Algo 1 still works. Algo 2 & 3 require you to sort the deck of cards before you actually search for the card. Is it worth it? Maybe we ca sort the card so quickly tha sortig plus searchig with Algo 2 or 3 will still be faster tha Algo 1 aloe. Ca you make a argumet why it is impossible? We will use this kid of argumets to prove lower bouds o the complexity of algorithms. Here is a challegig exercise (for which we shall study the aswer latter): You are give a huge pile of 256 exam papers that have already bee graded. Your job is the d the media grade, i.e., the paper that would be i the middle of the pile if that pile was sorted. Ca you do that i a way that is faster tha sortig the pile? ADL & AG Data Structures ad Algorithms 9 / 133 Sortig cards (2/2) Sortig cards (1/2) Ca you describe a algorithm to sort a had of cards? Here are two: isertio sort stack the usorted cards i frot of you, the pick the cards oe by oe ad place it at the right place ito your had. It also work if you place the usorted cards i oe side of your had ad the sorted cards at the other side. selectio sort put all the usorted cards i your had, remove the smallest oe, ad place it o a stack i frot of you. Repeat util you have stacked all the cards i order. Would you use same sortig algorithms to sort a etire pack of cards? Why ot? Here is a possible algorithm: make a rst pass o the pack to build 4 stacks, oe for each suit. Sort each stack as if it was a had. ADL & AG Data Structures ad Algorithms 10 / 133 Coutig people i the room (1/2) We use a dieret algorithm because the umber of card is too large for our hads. But it is also the case that sortig a had of cards usig the algorithm we use to sort a pack of cards would be slower. O a computer we might have similar tradeos: Sometimes we have so much data to process it will ot t i memory: we eed to devise way to process the data i smaller chuks. It is ofte the case that a algorithm that is eciet for processig a lot of data, will be less eciet o a smaller umber of data. Usig aother algorithm whe the umber of item is small is a commo implemetatio trick. Here are two algorithms: Algo A: Look at each perso i order, ad icremet a couter i your head. This assumes that is easy to dee the order (for istace if everybody is seated i the room). Algo B: This algorithm requires participatio from everybody ad goes as follows: 1 Everybody stad up ad remember the umber 1 2 If you are stadig up ad are ot aloe, d somebody else stadig up, ad add your umbers. Oe of you two should ow seat dow. 3 Repeat step 2 util you are stadig up aloe. The shout your umber, this is the umber of people i the room. ADL & AG Data Structures ad Algorithms 11 / 133 ADL & AG Data Structures ad Algorithms 12 / 133

Coutig people i the room (2/2) Radom Access Machies We apparetly have aother good-for-large/bad-for-small tradeo here: Algo B will be a lot faster tha Algo A with a lot of people. Algo A will be faster with oly a hadful of people. Algo B is a example of parallel algorithm: there are several uits of executio (the people) workig at the same time, while Algo A is a sequetial algorithm with oly oe uit of executio. If you imagie Algo B ruig i waves, where half of the people stadig up seat dow durig each wave, you should see a similarity with Algo 2 for dictioary lookup (discardig half of the dictioary at each step). Ca you tell how may waves it will take to cout a room of people? To study algorithm, we will work o a idealized computer: a Radom Access Machie (or RAM). A RAM has a uique processig uit that executes istructios sequetially radom access to the memory (i costat time) iite memory To measure the time complexity of a algorithm, we will study the time it takes to execute. We ca do that by coutig the umber of istructios executed. We ca also measure the space complexity of a algorithm by studyig the size of the memory it requires to work. (We will mostly focus o time complexity i this course.) ADL & AG Data Structures ad Algorithms 13 / 133 ADL & AG Data Structures ad Algorithms 14 / 133 Pseudo Code Dictioary Lookup: Data Structure To describe algorithms, we will use some pseudo-code. Pseudo code is midway betwee Eglish ad actual source code. It use covetios from computer laguages (like usig loops, fuctios) but without obeyig sytax rules; the goal is to provide a compact ad high-level descriptio of the algorithm. It may iclude some mathematical expressios or atural laguage descriptios of some operatios. How to represet the dictioary i memory i order to access each word easily? Problem: words i dictioary do ot have costat size. A rst solutio: Cocateate all words i memory, usig a special symbol to separate words. a$aback$abado$...$zucchii$zweiback Here the separatig symbol is $, but o a C/C++ implemetatio you would likely use the strig termiator \0. Is this represetatio suitable for our three dictioary lookup algorithms? ADL & AG Data Structures ad Algorithms 15 / 133 ADL & AG Data Structures ad Algorithms 16 / 133

Dictioary Lookup: Pseudo-Code for Algo 1 Dictioary Lookup: Setiel Value Iput: a array D of characters, of size s, i which words are delimited by `$'; a word w to search. Output: a idex i {0,..., s 1} such that D[i] is the start of a word equal to w, or 1 if w D. DictioaryLookup(D, s, w) : 1 pos 0 2 while pos < D do 3 ed pos 4 repeat ed ed + 1 util ed > s or D[ed] =`$' 5 if w = D[pos..ed] the 6 retur pos 7 pos ed + 1 8 doe 9 retur 1 A typical trick to get away with out-of-bouds checks such as es > s is to add a setiel value at the ed of the array. Here, if we replace a$aback$abado$...$zucchii$zweiback by the followig ecodig a$aback$abado$...$zucchii$zweiback$ the we ca simplify repeat ed ed + 1 util ed > s or D[ed] =`$' ito repeat ed ed + 1 util D[ed] =`$ because we kow that we ca always d a `$' after a word. ADL & AG Data Structures ad Algorithms 17 / 133 ADL & AG Data Structures ad Algorithms 18 / 133 Dictioary Lookup: Idirectio Ca we adapt Algo 2 to this kid of ecodig of a dictioary? Problem: because words have dieret sizes we caot d the middle word easily. We ca oly easily d the middle character. Example: Cosider a$car$chace$schoolteacher ad search for a. You will oly elimiate oe word at a time. A idea: build a idex table for all the words. I.e., a array that gives the startig positio of each word. This secod array may cotai idices, or it may cotai directly poiters to the actual words i the dictioary. We will ow assume the latter ad \0 termiatio. Dictioary Lookup: Liear Search Iput: a array A of (poiters to) strigs, the size s of A, ad a strig w to search. Output: a idex i {0,..., s 1} such that D[i] is the start of a strig equal to w, or 1 if w D. LiearSearch(A, s, w) : 1 for pos 0 to s 1 do 2 if w = D[pos] the 3 retur pos 4 doe 5 retur 1 This algorithm is ot restricted to strigs: it will work with ay kid of data. Ca you give a upper boud o the umber of iteratios of the loop if w A? (easy!) ADL & AG Data Structures ad Algorithms 19 / 133 ADL & AG Data Structures ad Algorithms 20 / 133

Dictioary Lookup: Liear Search Speedup How to speedup the detectio of w A if A is sorted? Iput: a sorted array A of (poiters to) strigs, the size s of A, ad a strig w to search. Output: a idex i {0,..., s 1} such that D[i] is the start of a strig equal to w, or 1 if w D. LiearSearchSorted(A, s, w) : 1 for pos 0 to s 1 do 2 if w = D[pos] the 3 retur pos 4 if w < D[pos] the 5 retur 1 6 doe 7 retur 1 Ca you see how to use a setiel value to remove the last lie? Ca you give a upper boud o the umber of iteratios of the loop if w A? This algorithm ca work o ay type of data that is ordered (i.e. supports the < operator). ADL & AG Data Structures ad Algorithms 21 / 133 Notes o Biary Search Dictioary Lookup: Biary Search Iput: a sorted array A[l..r] of strigs, a strig v to lookup Output: a idex i {l,..., r} such that A[i] = v or 1 if v A[l..r]. BiarySearch(A, l, r, v) : 1 while l r do 2 m (l + r)/2 3 if v = A[m] the 4 retur m 5 else 6 if v < A[m] the 7 r m 1 8 else 9 l m + 1 10 doe 11 retur 1 ADL & AG Data Structures ad Algorithms 22 / 133 Compare iterative ad recursive BiarySearch There are two ways to exit this algorithm: either at lie 4 (if v is foud) or at lie 11 (if v is ot foud). How ca we prove that it will exit the loop if v is ot foud? Ca you give a upper boud o the umber of iteratios if the loop if v A[l..r]? This algorithm works o ay type of data that is ordered. BiarySearch(A, l, r, v): while l r do m (l + r)/2 if v = A[m] the retur m else if v < A[m] the r m 1 else l m + 1 doe retur 1 BiarySearch(A, l, r, v): if l r the m (l + r)/2 if v = A[m] the retur m else if v < A[m] the retur BiarySearch(A, l, m 1, v) else retur BiarySearch(A, m + 1, r, v) else retur 1 ADL & AG Data Structures ad Algorithms 23 / 133 ADL & AG Data Structures ad Algorithms 24 / 133

Represetig a Had of Cards Isertio Sort o a Array How ca we best represet a had of cards? We assume the order of the cards i the had matters. A array? accessig the ith card is fast swappig two cards is easy movig oe card to aother place require to shift all cards i-betwee (costly) A liked list 2? accessig the ith card is slow swappig two cars is easy (if you have poiters to them) movig a card to aother place is eciet if you kow the destiatio Iput: a array A of items (e.g. cards) to sort Output: the array A sorted i icreasig order IsertioSortArray(A) 1 for j 2 to legth(a) do 2 key A[j] 3 i j 1 4 while i > 0 ad A[i] > key do 5 A[i + 1] A[i] 6 i i 1 7 A[i + 1] key 2 http://e.wikipedia.org/wiki/liked_list ADL & AG Data Structures ad Algorithms 25 / 133 ADL & AG Data Structures ad Algorithms 26 / 133 Isertio Sort o a Liked List Isertio Sort: Array vs. List Iput: a liked list L of items to sort Output: the list L sorted i icreasig order IsertioSortList(L) 1 if L = the retur L 2 res L; L L.ext; res.ext 3 while L 4 tmp L.ext 5 if res.data > L.data the 6 L.ext res; res L 7 else 8 dst res 9 while dst.ext ad dst.ext.data L.data do 10 dst dst.ext 11 L.ext dst.ext; dst.ext L 12 L tmp 13 retur res Note how the two data structures commad slightly dieret algorithms eve though the basic idea is the same. From distace the two algorithms do the same thig: Cosider all items from left to right. For each item, d its place i the previous items ad isert it there. Fidig the place ca be doe usig a search from left to right, or from right to left. A Sigly Liked List forbids a search from right to left, so we have to work from left to right. Isertig i a array requires to shift all elemets at the right of the isertio, so it is more eciet to shift these elemets as we search from right to left. ADL & AG Data Structures ad Algorithms 27 / 133 ADL & AG Data Structures ad Algorithms 28 / 133

Measurig complexity Let us show how we ca measure the time complexity of a algorithm. What we wat is to see how the algorithm scales as the iput grows larger. I other words, the time complexity is a fuctio T () where is the size of the iput. We measure time formally by coutig executed istructios. Dieret istructios may have dieret costs (=ru time), so we will have to weight them. Actual cost of a istructio i pseudo-code is depedet o the programmer who traslated pseudo-code to a programmig laguage the compiler who traslated to programmig laguage ito machie code the CPU who is executig the machie code Evetually, we will abstract from these implemetatio details factors ADL & AG Data Structures ad Algorithms 29 / 133 Let's do that for IsertioSort. IsertioSort: Best ad Worst cases Best case: the array is sorted. t j = 1. T () = c 1 + c 2 ( 1)+c 3 ( 1)+c 4 ( 1)+c 7 ( 1) This is a liear fuctio of the form a + b. Worst case: the array is reversed. t j = j. Recall that j = (+1) 1 ad j=2 2 ( 1) (j 1) = j=2 2. T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( ( + 1) 2 + c 5 ( 1) 2 + c 6 ( 1) 2 + c 7 ( 1) This is a quadratic fuctio of the form a 2 + b + c. 1 ADL & AG Data Structures ad Algorithms 31 / 133 ) Isertio Sort o a Array Iput: a array A of items (e.g. cards) to sort Output: the array A sorted i icreasig order IsertioSortArray(A) cost occ. 1 for j 2 to legth(a) do c 1 2 key A[j] c 2 1 3 i j 1 c 3 1 4 while i > 0 ad A[i] > key do c t 4 j=2 j 5 A[i + 1] A[i] c (t 5 j=2 j 1) 6 i i 1 c (t 6 j=2 j 1) 7 A[i + 1] key c 7 1 T () = c 1 + c 2 ( 1) + c 3 ( 1) t j + c 5 (t j 1) + c 6 + c 4 j=2 j=2 What are the best cases? worst cases? (t j 1) + c 7 ( 1) ADL & AG Data Structures ad Algorithms 30 / 133 IsertSort: Average case The best case gives a upper boud for the complexity The worst case gives a lower boud for the complexity The geeral case is obviously i betwee We ca also do average case aalysis: Assume the array cotais radomly chose umbers (followig a uiform distributio). For a key picked at lie 2, we expect half of the values of the array to be greater tha key, ad half less tha key. Therefore t j = t 2. j=2 We have t = (+1) 2 ad t 1 = ( 3)+2 j=2 2 4 j=2 2 4 T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( + 1) 2 4 + c 5 ( 3) + 2 4 + c 6 ( 3) + 2 4 + c 7 ( 1) This is agai a quadratic fuctio of the form a 2 + b + c. ADL & AG Data Structures ad Algorithms 32 / 133

Fuctio Compariso I practice 40 3 e 2 log 2 l 2 Assumig 10 6 operatios per secod. log 2 log 2 2 3 2 30 20 10 0 5 10 15 20 log 10 /2 l ADL & AG Data Structures ad Algorithms 33 / 133 10 2 6.6 µs 0.1 ms 0.6 ms 10 ms 1 s 4 10 6 y 10 3 9.9 µs 1 ms 10 ms 1 s 16.6 mi 10 4 13.3 µs 10 ms 0.1 s 1.6 mi 11.6 d 10 5 16.6 µs 0.1 s 1.6 s 2.7 h 347 y 10 6 19.9 µs 1 s 19.9 s 11.6 d 10 6 y 10 7 23.3 µs 10 s 3.9 mi 3.17 y 10 8 26.6 µs 1.6 mi 44.3 mi 317 y 10 9 29.9 µs 16.6mi 8.3 h 31709 y ADL & AG Data Structures ad Algorithms 34 / 133 Machie Idepedece Asymptotic Equivalece of Fuctios I our formulas for T (), coeciets c 1, c 2,..., c 7 are machie-depedet (ad compiler-depedet, ad programmer-depedet). We would like to: igore machie-depedet costats, study to growth of T () whe. = Let's perform a asymptotic aalysis of the ru-time complexity. Θ(g()) = {f () c 1 R +, c 2 R +, 0 N, 0, 0 c 1 g() f () c 2 g()} By covetio (ad abuse) we write f () = Θ(g()) istead of f () Θ(g()). If a 2 > 0, we have a 2 2 + a 1 + a 0 = Θ( 2 ). For istace let c 1 = a 2 2 ad c 2 = 3a2, the show that 2 a 2 2 a 2 + a 1 + a 0 2 }{{} 0 whe 3a 2 2 ADL & AG Data Structures ad Algorithms 35 / 133 ADL & AG Data Structures ad Algorithms 36 / 133

Asymptotic Complexity of Isertio Sort Best case T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( 1) + c 7 ( 1) Worst case = Θ() T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( ( + 1) 2 Average case + c 5 ( 1) 2 + c 6 ( 1) 2 1 + c 7 ( 1) = Θ( 2 ) T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( + 1) 2 4 + c 5 ( 3) + 2 4 + c 6 ( 3) + 2 4 ) + c 7 ( 1) = Θ( 2 ) ADL & AG Data Structures ad Algorithms 37 / 133 Let us Simplify the Calculatios Oe way to simplify the calculatio is to pick oe fudametal operatio (or oe familly of operatios) for the problem ad cout oly its umber of executios. The choice is good if the total umber of operatios is proportioal to the cout of fudametal operatios executed. Examples: problem additio of biary umbers matrix multiplicatio sortig a array fudametal operatio all biary operatios scalar multiplicatio comparisos of elemets ADL & AG Data Structures ad Algorithms 38 / 133 Simplied Calculatios Simplied Calculatios IsertioSortArray(A) cost occ. 1 for j 2 to legth(a) do c 1 2 key A[j] c 2 1 3 i j 1 c 3 1 4 while i > 0 ad A[i] > key do c t 4 j=2 j 5 A[i + 1] A[i] c (t 5 j=2 j 1) 6 i i 1 c (t 6 j=2 j 1) 7 A[i + 1] key c 7 1 IsertioSortArray(A) 1 for j 2 to legth(a) do 2 key A[j] 3 i j 1 4 while i > 0 ad A[i] > key do 5 A[i + 1] A[i] 6 i i 1 7 A[i + 1] key occ. j=2 t j ADL & AG Data Structures ad Algorithms 39 / 133 ADL & AG Data Structures ad Algorithms 39 / 133

Asymptotic Complexity of Isertio Sort Best case T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( 1) + c 7 ( 1) Worst case = Θ() T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( ( + 1) 2 Average case + c 5 ( 1) 2 + c 6 ( 1) 2 1 + c 7 ( 1) = Θ( 2 ) T () = c 1 + c 2 ( 1) + c 3 ( 1) + c 4 ( + 1) 2 4 + c 5 ( 3) + 2 4 + c 6 ( 3) + 2 4 ) + c 7 ( 1) = Θ( 2 ) ADL & AG Data Structures ad Algorithms 40 / 133 Asymptotic Complexity of Isertio Sort Best case T () = 1 = Θ() Worst case ( + 1) T () = 1 2 = Θ( 2 ) Average case ( + 1) 2 T () = 4 = Θ( 2 ) ADL & AG Data Structures ad Algorithms 40 / 133 Asymptotic Higher ad Lower Bouds O(g()) = {f () c R +, 0 N, 0, 0 f () cg()} Ω(g()) = {f () c R +, 0 N, 0, 0 cg() f ()} Note that f () = Θ(g()) f () = O(g()) et f () = Ω(g()). I other words Θ(g()) = O(g()) Ω(g()). Sice Θ() O( 2 ) ad Θ( 2 ) O( 2 ), we ca say that the ru-time complexity of Isertio Sort for elemets is i O( 2 ). ADL & AG Data Structures ad Algorithms 41 / 133 Properties g() If lim = c > 0 the g() = Θ(f ()). f () g() If lim = 0 the g() = O(f ()) ad f () O(g()). f () g() If lim = the f () = O(f ()) ad g() O(f ()). f () f () = O(g()) i g() = Ω(f ()). If f 1 () = Θ(g 1 ()), f 2 () = Θ(g 2 ()), ad k is costat, we have: f 1 () f 2 () = Θ(g 1 () g 2 ()) f 1 () + f 2 () = Θ(g 1 () + g 2 ()) k f 1 () = Θ(g 1 ()) The above rules are also true for O ad Ω. ADL & AG Data Structures ad Algorithms 42 / 133

Exercises Complexity Aalysis with Asymptotic Notatios Fid two fuctios f ad g such that f () O(g()) ad g() O(f ()). We say that the two fuctios are icomparable usig O. Show that Θ(f () + g()) = Θ(max(f (), g())). If f () = Θ(g()), do we have 2 f () = Θ(2 g() )? Let us dee the followig (partial) order: Θ(f ()) Θ(g()) si f = O(g()) Θ(f ()) < Θ(g()) si f = O(g()) et g O(f ()) Order the sets Θ(...) cotaiig the followig fuctios:, 2, log, l, + 7 5, log,, e, 2 1, 2, 2 + log, log log, 3, (log ) 2,!, 3/2. Cosider a geeral iput of size, ad cout the occurreces of each istructio usig asymptotic otatios. IsertioSortArray(A) 1 for j 2 to legth(a) do Θ()+ 2 do key A[j] Θ()+ 3 i j 1 Θ()+ 4 while i > 0 ad A[i] > key do O( 2 )+ 5 do A[i + 1] A[i] O( 2 )+ 6 i i 1 O( 2 )+ 7 A[i + 1] key Θ() O( 2 ) ADL & AG Data Structures ad Algorithms 43 / 133 ADL & AG Data Structures ad Algorithms 44 / 133 Selectio Sort Idea: Fid miimum of A[1..] the swap it with A[1]. Fid miimum of A[2..] the swap it with A[2]. Etc. A[1..k] is sorted after k iteratios. SelectioSort(A) 1 for i from 1 to do Θ() 2 mi i Θ() 3 for j from i to do Θ( 2 ) 4 if A[j] < A[mi] the mi j Θ( 2 ) 5 A[mi] A[i] Θ() Θ( 2 ) It is worse tha IsertioSort which is i O( 2 ) but ot i Ω( 2 ). ADL & AG Data Structures ad Algorithms 45 / 133 Merge Sort Iput: a array A of itegers, two idices l, r Output: array A, with it subarray A[l..r] sorted i icreasig order MergeSort(A, l, r) T (1) T () 1 if l < r the Θ(1) Θ(1) 2 m (l + r)/2 Θ(1) 3 MergeSort(A, l, m) T ( /2 ) 3 MergeSort(A, m + 1, r) T ( /2 ) 4 Merge(A, l, m, r) Θ() Usig = r l + 1, we ca express T () usig a recursive equatio: { Θ(1) if = 1 T () = 2T (/2) + Θ() if > 1 Exercise: Write pseudo-code for Merge ad prove its Θ() complexity. ADL & AG Data Structures ad Algorithms 46 / 133

Merge Iput: a array A, three idices l, m, r such that A[l..m] ad A[m + 1..r] are sorted Output: array A, with it subarray A[l..r] sorted Merge(A, l, m, r) i l; j m + 1; k l Θ(1) while i m ad j r do O() if A[i] A[j] the O() B[k] A[i]; k k + 1; i i + 1 O() else O() B[k] A[j]; k k + 1; j j + 1 O() if i m the Θ(1) B[k..r] A[i..m] O() else O(1) B[k..r] A[j..r] O() A[l..r] B[l..r] Θ() retur A Θ(1) Θ() ADL & AG Data Structures ad Algorithms 47 / 133 Solvig Recursios by Ufoldig As secod way to solve T () = 2T (/2) + c is by ufoldig. T () = c + 2T (/2). = c + 2(c/2) + 4T (/4) = 2c + 4T (/4) = 2c + 4(c/4) + 8T (/8) = 3c + 8T (/8) = kc + 2 k T (/2 k ) We ca cotiue ufoldig util we reach T (1) = Θ(1). This happes whe 2 k =. Substitutig k = log 2, we get: T () = c log 2 + T (1) = Θ( log ) + Θ() = Θ( log ) ADL & AG Data Structures ad Algorithms 49 / 133 Call Tree for MergeSort Let us solve T () = 2T (/2) + c graphically, for a costat c > 0. c c h = log Θ(1) c/2 c/2 c/4 c/4 c/4 c/4 We have T () = Θ( log ). leaves c c Θ() ADL & AG Data Structures ad Algorithms 48 / 133 Geeral Theorem for Recurrece Equatios Let T () = with a 1, b > 1, 0 N. The { Θ(1) si 0 at (/b) + f () si > 0 if f () = O( log b a ε ) for some ε > 0, the T () = Θ( log b a ). if f () = Θ( log b a ), the T () = Θ( log b a log ). if f () = Ω( log b a+ε ) for some ε > 0, ad if af (/b) cf () for some costat c < 1 ad all large eough values of, the T () = Θ(f ()). Beware, there are holes i this theorem: a fuctio f () may belog to oe of the three cases. The ε costrais fuctios f () to be polyomially smaller or greater tha log b a. ADL & AG Data Structures ad Algorithms 50 / 133

Applicatios of the Theorem Perfect Tree T () = 2T (/2) + Θ() a = b = 2, log b a =. We have T () = Θ( log ). T () = 4T (/2) + a = 4, b = 2, log b a = 2. ɛ = 1 ad = O( 2 1 ), we thus have T () = Θ( 2 ) A perfect tree is a complete biary tree i which leaves from the deepest level are all grouped o the left (if that level is ot complete). T () = 3T (/3) + 2 a = 3, b = 3, log b a =. ɛ = 1 ad 2 = Ω( 1+1 ), furthermore 3(/3) 2 c 2 for c = 1/3, cosequetly T () = Θ( 2 ). T () = 4T (/2) + 2 / log a = 4, b = 2, log b a = 2. We caot d a ε > 0 such that 2 / log = O( 2 ε ). Ideed, 2 / log c 2 ε implies ε c log. The theorem caot apply. ADL & AG Data Structures ad Algorithms 51 / 133 ADL & AG Data Structures ad Algorithms 52 / 133 Biary Heap A biary heap is a perfect tree with the heap property: each ode is greater tha or equal to each of its childre. 18 12 11 6 10 9 4 2 3 8 Ay perfect tree ca be ecietly stored as a array. This is how we will store biary heaps too. 18 12 11 6 10 9 4 2 3 8 ADL & AG Data Structures ad Algorithms 53 / 133 Properties of a biary heap 18 12 11 6 10 9 4 2 3 8 A = 18 12 11 6 10 9 4 2 3 8 Idex 1 i the array correspods to the root of the tree Paret(i) = i/2 (if i > 0) LeftChild(i) = 2i (if it exists) RightChild(i) = 2i + 1 (if it exists) Heap property: i > 0, A[Father(i)] A[i]. ADL & AG Data Structures ad Algorithms 54 / 133

Operatio o Biary Heaps (1): Heapify Operatios o Heaps: BuildHeap Iput: array A, ad two idexes i ad m such that A[LeftChild(i)] ad A[RightChild(i)] are roots of heaps i A[1..m], Output: the array A such that A[i] is the root of a heap. Heapify(A, i, m) 1 l LeftChild(i) 2 r RightChild(i) 3 if l m ad A[l] > A[i] 4 the largest l 5 else largest i 6 if r m ad A[r] > A[largest] 7 the largest r 8 if largest i the 9 A[i] A[largest] 10 Heapify(A, largest, m) T () T (2/3) + Θ(1) with the size of the subtree rooted at i, ad 2/3 the maximum umber of odes of the subtree recursively explored. We deduce T () = O(log ) We ca also write T (h) = O(h) with h the height of the tree rooted at i. Iput: a array A Output: the array A orgaized as a heap. BuildHeap(A) 1 for i legth(a)/2 dow to 1 do 2 Heapify(A, i, legth(a)) The elemets betwee legth(a)/2 + 1 ad the ed of the array are the leaves: they are already heaps. We x the rest of the array i a bottom-up way. Ituitively if = legth(a), T () = Θ() }{{} the loop O(log ) }{{} Heapify = O( log ). I fact the time spet i Heapify depeds o the size of the subtree cosidered, ot the etire tree. The complexity is better. ADL & AG Data Structures ad Algorithms 55 / 133 ADL & AG Data Structures ad Algorithms 56 / 133 Complexity of BuildHeap Let us calculate T () for BuildHeap more precisely. The height of a complete biary tree of odes is log. The umber of subtrees of height h i a heap is at most /2 h+1. The complexity of Heapify o a subtree of height h is est O(h). We get: T () = Sice k=0 log h=0 kx k = 2 h+1 Fially T () = O(). x O(h) = O, we have 2 (1 x) log h=0 h=0 h 2 h+1 = O log h=0 h 2 = 1/2 h (1 1/2) = 2. 2 h 2 h Heap Sort HeapSort(A) 1 BuildHeap(A) 2 for i legth(a) dow to 2 do 3 A[1] A[i] 4 Heapify(A, 1, i 1) The complexity is easy to express: T () = O() }{{} BuildHeap +O( }{{} loop log ) = O( log ) }{{} Heapify ADL & AG Data Structures ad Algorithms 57 / 133 ADL & AG Data Structures ad Algorithms 58 / 133

Quick Sort Quick Sort: algorithm Origi Sir Charles Atoy Richard Hoare, 1962. Geeral idea Partitio the array i two parts, such that elemets from the rst part are smaller tha elemets from the secod. Sort both parts recursively. How to partitio? Pick a value ad use it as pivot. Usig successive swaps, arrage the array i two blocs such that elemets at the begiig are less or equal to the pivot elemets at the ed are greater tha or equal to the pivot Our choice: the pivot is the rst elemet. Iput: a array A, ad two idices l ad r Output: the array A with A[l..r] sorted i icreasig order QuickSort(A, l, r) 1 if l < r the 2 p Partitio(A, l, r) 3 QuickSort(A, l, p) 4 QuickSort(A, p + 1, r) Iput: a array A, two idices l ad r Output: a idex p, the array A arraged so that A[l..p] A[p + 1..r]. Partitio(A, l, r) 1 x A[l]; i l 1; j r + 1 2 repeat forever 3 do i i + 1 util A[i] x 4 do j j 1 util A[j] x 5 if i < j the 6 A[i] A[j] 7 else 8 retur j T Partitio () = Θ() of = r l + 1. ADL & AG Data Structures ad Algorithms 59 / 133 ADL & AG Data Structures ad Algorithms 60 / 133 Complexity of QuickSort Ufavorable case The choice of pivot is ulucky ad yields a ubalaced partitio. This happes if the iput is already sorted (i ay way). T () = Θ() + Θ(1) + T ( 1) = T ( 1) + Θ() = Θ( 2 ) Best case The partitio always splits the array i the middle. T () = Θ() + 2T (/2) Same equatio as MergeSort. We kow the aswer is T () = Θ( log ). ADL & AG Data Structures ad Algorithms 61 / 133 Ituitio for Average Complexity Let's assume a imbalace by a costat ratio: 1/10 : 9/10 ( ( ) 9 T () = T + T + Θ() 10) 10 Draw the call tree. The smallest brach has height log 10 = Θ(log ): the complexity of the call tree limited to this level is Θ( log ). We deduce that T () = Ω( log ). The logest brach has height log 10/9 ad the complexity at each level is. We get that T () log 10/9 = O( log ). Fially T () = Θ( log ). Ay partitio usig a costat ratio implies T () = Θ( log ). ADL & AG Data Structures ad Algorithms 62 / 133

Calculatio of the Average Complexity (1) Let us assume a uiform distributio o the possible partitios. The partitio cuts A[1..] i A[1..i] ad A[i + 1..] with 1 possible choices for i. T () = 1 1 (T (i) + T ( i)) + Θ() = 2 1 T (i) + c 1 1 Furthermore: i=1 T ( 1) = 2 2 T (i) + c( 1) 2 Let's try to make T ( 1) appear i T (): ( ) 2 2( 2) T () = T ( 1) + T (i) + c( 1 + 1) ( 1)( 2) i=1 ADL & AG Data Structures ad Algorithms 63 / 133 i=1 i=1 Calculatio of the Average Complexity (2) ( ) 2 2( 2) T () = T ( 1) + T (i) + c( 1 + 1) ( 1)( 2) i=1 = 2 ( 2 T ( 1) + T ( 1) + c( 1) 1 2 ) 1 1 1 = T ( 1) + 2c 1 Divide left ad right by : T () = T ( 1) 1 + 2c ADL & AG Data Structures ad Algorithms 64 / 133 + c Calculatio of the Average Complexity (3) Questios T () Let's itroduce Y () = T () T ( 1) = 1 + 2c Y () = Y ( 1) + 2c = 1 2c 1 T () = 2c i=1 i i=1 i What happes of all the elemets of the array have the same value? It seems that a radom array will be sorted more ecietly tha a sorted array. How ca we modify QuickSort to esure that it will have the same (averge) complexity o radom arrays ad sorted arrays? Usig Euler's formula 1 i=1 i = l + γ + o(1) = Θ(log ) we get T () = Θ()Θ(log ) = Θ( log ) ADL & AG Data Structures ad Algorithms 65 / 133 ADL & AG Data Structures ad Algorithms 66 / 133

Stochastic QuickSort A simple Idea: choose the pivot radomly i the array. RadomizedPartitio(A, l, r) 1 x A[Radom(l, r)]; i l 1; j r + 1 2 repeat forever 4 do j j 1 util A[j] x 3 do i i + 1 util A[i] x 5 if i < j the 6 A[i] A[j] 7 else 8 retur j The eect is as if we had radomized the array before callig QuickSort. Pro: o particular iput is kow to always provoke the worst case. Cos: Radom() is a slow fuctio. Callig it so much (how may time is it called?) is a sure way to slow-dow your implemetatio. ADL & AG Data Structures ad Algorithms 67 / 133 Coclusio o Quick Sort We have T () = O( 2 ) i geeral but T () = Θ( log ) o the average. I practice Quick Sort is faster tha the other sortig algorithms preseted so far (assumig is ot ridiculously small). For a smaller, Isertio Sort is a better choice. The qsort() implemetatio i GNU Libc (ad others) use these tricks: Use media-of-3 pivot (extremities ad middle). Switch to Isertio Sort if the array has 4 elemets. Order the two recursive calls such that the rst oe sees the smallest subarray, ad the latter oe (which is a tail recursio) use the largest subarray. Do ot actually perform recursive calls: tail recursio ca be replaced by a loop, ad the rst call to QuickSort requires a explicit stack. ADL & AG Data Structures ad Algorithms 69 / 133 Aother idea: media pivot (The media of 2k + 1 values is the (k + 1)st largest value.) Idea: use as pivot the media of some values of the arrays (ot all values, it would take too log to d the media). For istace use the media of the rst three value, or better (why?) the media of A[l], A[ l+r ] ad A[r]. 2 ADL & AG Data Structures ad Algorithms 68 / 133 Itrospective Sort Origi David Musser, 1997 Used i SGI's Stadard Template Library. (std::sort) Iterest Modicatio of Quick Sort so that T () = Θ( log ) always. Idea Detect whe the values to sort are causig trouble to Quick Sort, ad use a Heap Sort i this case. I practice We boud the umber of recursive calls to O(log ). Musser suggests 2 log. ADL & AG Data Structures ad Algorithms 70 / 133

Itrospective Sort: Algorithm ItroSort(A, l, r) 1 ItroSort'(A, l, r, 2 log(r l + 1) ) ItroSort'(A, l, r, depth_limit) 1 if depth_limit = 0 the 2 HeapSort(A, l, r) 3 retur 4 else 5 depth_limit depth_limit 1 6 p Partitio(A, l, r) 7 ItroSort'(A, l, p, depth_limit) 8 ItroSort'(A, p + 1, r, depth_limit) This is a ifty implemetatio trick that you caot thik of without studyig the complexity of your algorithms. Sortig as a Problem Iput: A sequece of umbers a 1, a 2,... a Output: A permutatio a 1, a 2,... a of the iput sequece such that a 1 a 2 a. These umbers might be attached to other data. For istace they might be a eld i a record, ad we wat to sorts to records accordig to this eld (called the key for the purpose of sortig). The structure used to represet A is usually a array. (We have also looked at IsertioSort o a list.) For ow, we have oly looked at compariso sorts, i.e. algorithms that perform comparisos to order the elemets. ADL & AG Data Structures ad Algorithms 71 / 133 ADL & AG Data Structures ad Algorithms 72 / 133 I place ad Stable Sortig Algorithm Summary of Sortig Algorithms Studied so far I Place Sort A sortig algorithm is i place if the umber of memory it requires i additio to the iput is idepedet of, or at most Θ(log ). Especially you are ot allowed to use a temporary array of size i a i place sort, sice that would require Θ() memory. Stable Sort A sortig algorithm is stable if the order of equal elemets is preserved. This matters whe the key used for sortig is part of a larger structure (with other data attached), ad several sorts are chaied usig dieret elds as key. ADL & AG Data Structures ad Algorithms 73 / 133 complexity average i place? stable? isertio sort O( 2 ) Θ( 2 ) yes yes selectio sort Θ( 2 ) yes o merge sort Θ( log ) o yes heap sort O( log ) 1 yes o quick sort O( 2 ) Θ( log ) yes 2 o itro sort Θ( log ) yes 2 o 1 The complexity is i fact Θ( log ), but we have ot proved it. 2 The umber of temporary variables used locally by QuickSort ad Partitio is costat, but because of the recursive calls we are actually creatig several copies of them (as may copies as the depth of the call tree). QuickSort requires O(log ) extra memory whe orderig the partitio so that the largest part is hadled by the tail recursio (it requires O() memory if you do ot use such a trick). ADL & AG Data Structures ad Algorithms 74 / 133

Complexity of a problem Deitio The complexity C() of a problem P is the complexity if the best algorithm that solves P. Cosequeces If a algorithm A solves P i O(f ()), the C() = O(f ()). If we ca prove that all algorithms that solve P have a complexity i Ω(g()), the C() = Ω(g()). If these two bouds are equivalets (i.e., f () = Θ(g())) the C() = Θ(f ()) = Θ(g()), ad this is the complexity of the problem. For ow, we have proved that sortig umbers is i O( log ). That does ot mea it is impossible to do better It is always possible to do worse :-) ADL & AG Data Structures ad Algorithms 75 / 133 Lower Boud for Worst Case of Compariso Sort Recall the problem Iput: A sequece of umbers a 1, a 2,... a Output: A permutatio a 1, a 2,... a of the iput sequece such that a 1 a 2 a. Argumetatio Sorts ca be represeted by a biary tree (decisio tree). Iteral odes are comparisos betwee two elemets: the left child represet a egative aswer ad the right child a positive aswer. Leaves of the tree represet a permutatio to apply to sort the array. There exists! possible permutatios of a 1, a 2,... a. Our biary tree of! leaves, should therefore have a height of at least log!. Usig Stirlig's formula 3 we obtai that Ω(log(!)) = Ω( log ). The worst case of ay compariso sort uses Ω( log ) comparisos. 3! = 2π(/e) (1 + Θ(1/)) ADL & AG Data Structures ad Algorithms 76 / 133 Coutig Sort Characteristics Stable sort, ot i place. May be used oly if the keys belog to a small iterval. Here we assume they are i {1,..., k}. Algorithm CoutigSort(A, B, k) 1 for i 1 to k do C[i] 0 Θ(k) 2 for i 1 to legth(a) do C[A[j]] C[A[j]] + 1 Θ() 3 for i 1 to k do C[i] C[i] + C[i 1] Θ(k) 4 for i legth(a) dow to 1 do Θ() 5 B[C[A[i]]] A[i] Θ() 6 C[A[i]] C[A[i]] 1 Θ() Θ() + Θ(k) Complexity If k = O(), the T () = Θ(). ADL & AG Data Structures ad Algorithms 77 / 133 Bucket Sort Characteristics Ustable sort, ot i place. Assume the elemets are uiformly distributed. Here we assume they are i the iterval [0, 1[. Algorithm BucketSort(A) 1 legth(a) Θ() 2 for i 1 to do Θ() 3 isert A[i] ito the list B[ A[i] ] Θ() 4 for i 0 to 1 do Θ() 5 sort B[i] 1 with IsertioSort i=0 O(2 i 6 cocateate B[0], B[1],..., B[ 1] together i order Θ() Complexity It depeds o the size i of the buckets B[i] do sort. ADL & AG Data Structures ad Algorithms 78 / 133

Complexity of Bucket Sort Biomial Distributio Remider Best case If each B[i] has size i = 1, the calls to IsertioSort at lie 5 all cost Θ(1). The complexity is Θ(). Worst case If (1) all elemets lad i the same bucket, ad (2) this bucket happes to be sorted i reverse order (worst case of IsertioSort) the lie 5 costs Θ( 2 ). The al complexity is Θ( 2 ). Average case What is i o the average? i.e. E[ i ] What is 2 i o the average? i.e. E[ 2 i ] We evetually wat to compute 1 i=0 O(E[2 i ]). Expected value of a radom variable It is its mea: E[X ] = xpr{x = x} x Variace Var[X ] = E[(X E[X ]) 2 ] = E[X 2 ] E 2 [X ] Biomial Distributio Throw balls i r baskets, ad assume balls have equal chaces to lad i each basket (p = 1/r). Let X i deote the umber of balls i basket i. We have Pr{X i = k} = ( ) p k (1 p) k. It ca be k show that E[X i ] = p ad Var[X i ] = p(1 p). ADL & AG Data Structures ad Algorithms 79 / 133 ADL & AG Data Structures ad Algorithms 80 / 133 Probabilistic Study of Bucket Sort Let i deote the size of a bucket to sort with Isertio Sort. If the value to sort are uiformly distributed, they have equal chaces to lad i each bucket. It is like throwig balls ito baskets (i.e. p = 1/). Therefore E[ i ] = p = 1 ad Var[ i ] = p(1 p) = 1 1. Isertio Sort of elemets takes O( 2 ), so for all B[i] we have 1 O(E[ 2 ]) = O i i=0 = O ( 1 i=0 ( 1 E[ 2 i ] ) = O ( 1 i=0 ( 1 + 1 ) ) 1 = O() i=0 Fially T () = O() + Θ() = Θ() o the average. ( E 2 [ i ] + Var[ i ] )) ADL & AG Data Structures ad Algorithms 81 / 133 Data structures You kow of a couple of data structures, i.e., ways to orgaize data i memory to ease certai operatios. Array Sigly liked list Whe you programmed last year, you probably used C++ cotaier classes like std::vector (a array that ca chage its size) ad list std::list (a doubly liked list). As we saw with the dictioary lookup example, we ca ofte choose betwee several data structures. The dierece is i the operatios they allow to perform, ad the complexity of these operatios. ADL & AG Data Structures ad Algorithms 82 / 133

Abstract Data Type A abstract data type is a mathematical specicatio of a data set, ad of a set of operatios you ca apply to this set. It is a cotract that a data structure has to implemet. For istace the stack abstract data type that represets a ordered set allowig two operatio push adds a item to the set i Θ(1), pop removes ad returs the last item added to the set i Θ(1). Such a abstract data type ca be implemeted usig a sigly liked list or array (Questio: how would you implemet these operatios so that the complexity costraits are hoored?) Algorithms may be described usig abstract data types, sice such a abstractios precisely specify the expected behavior. The choice of the data structure used to implemet the abstract data type ca be delayed util the algorithm is actually implemeted. ADL & AG Data Structures ad Algorithms 83 / 133 Some Data Structures to Represet a Set of Data Sequeces: array, vector liked list (sigly-, doubly-) stack queue priority queue. double etry queue (a.k.a deque) Associative array, search structures hash table self-balacig biary search tree skip list You pick a data structure accordig to the operatios you eed to execute o your data ad the complexity of these operatios for this data structures. ADL & AG Data Structures ad Algorithms 84 / 133 Some Operatios o Sequeces v Access(S,k) Retur the k th elemet. p Search(S,v) Retur a poiter (or idex) to a elemet of S whose value is v. Isert(S,x) Add elemet x to S. Delete(S,p) Delete the elemet of S that is at positio (poiter or idex) p. v Miimum(S) Retur the miimum of S. v Maximum(S) Retur the maximum of S. p Successor(S,p) Retur the positio of the successor of (= the smallest value greater tha) the elemet at positio p i S. p Predecessor(S,p) Guess. These are just some operatios we will study for all data structures we preset. Of course more operatios exist (like sort, uio, split,...) ad would deserve to be studied too. ADL & AG Data Structures ad Algorithms 85 / 133 Arrays No eed to preset arrays usorted sorted operatio array array v Access(S, k) Θ(1) Θ(1) p Search(S, v) O() O(log ) Isert(S, x) Θ(1) O() Delete(S, p) Θ(1) 4 O() v Miimum(S) Θ() Θ(1) v Maximum(S) Θ() Θ(1) p Successor(S, p) Θ() Θ(1) p Predecessor(S, p) Θ() Θ(1) 4 Sice the order does ot matter, we ca replace the elemet that is deleted by the last elemet of the array. ADL & AG Data Structures ad Algorithms 86 / 133

Dyamic Arrays (1/2) Array whose size ca vary. I C, we eed to call realloc() whe there is ot eough uused etries left for isertio. I C++, std::vector will perform realloc() by itself. Reallocatig a array requires Θ() time, sice it has to be copied. Isertig i a array, usually is a Θ(1) operatio, becomes Θ() if a reallocatio is required. You do ot wat to reallocate a array just to add oe etry, because isertio would the cost Θ() every time. What is a suitable reallocatio scheme? We ca study the amortized complexity of a isertio i a sequece of isertios. Lists ADL & AG Data Structures ad Algorithms 87 / 133 We distiguish betwee sigly liked lists (where oly the ext item is kow) ad doubly liked lists (where a predecessor is also kow). S.L.L. D.L.L. operatio usorted sorted usorted sorted v Access(S, k) O() O() O() O() p Search(S, v) O() O() O() O() Isert(S, x) Θ(1) O() Θ(1) O() Delete(S, p) O() 1 O() 1 Θ(1) Θ(1) v Miimum(S) Θ() Θ(1) Θ() Θ(1) v Maximum(S) Θ() Θ(1) Θ() Θ(1) p Successor(S, p) Θ() Θ(1) Θ() Θ(1) p Predecessor(S, p) Θ() O() Θ() Θ(1) 1 I practice the deletio ca be i Θ(1) if you kow the previous elemet somehow. ADL & AG Data Structures ad Algorithms 89 / 133 Dyamic Arrays (2/2) Let us cosider the case of a isertio that leads to reallocatio, with two dieret ways to elarge the array: Add k ew etries. There will be a reallocatio every k isertios, so the average cost of the last k isertios is (k 1)Θ(1) + 1Θ() k = Θ() Double the size. Sice the last reallocatio there have bee /2 1 isertios i Θ(1) followed by oe isertio i Θ(). The average cost of the last /2 isertios is (/2 1)Θ(1) + 1Θ() /2 = Θ() + Θ() = Θ(1) We say that isertio is i amortized Θ(1) whe the operatio is usually i Θ(1), ad the slow cases are ifrequet eough so that their cost ca be amortized o the fast cases. ADL & AG Data Structures ad Algorithms 88 / 133 Stack ad Queues Stack Sequece i which isertios ad deletios are always doe at the same ed. Isert() ad Delete() are usually called Push() ad Pop(). LIFO = Last I First Out Usually implemeted o top of a array or sigly liked list. Queue Sequece i which isertios ad deletios are doe at opposite eds. Isert() ad Delete() are usually called Push() ad Pop(). FIFO = First I First Out Usually implemeted o top of a sigly liked list with tail poiter. Isert at tail i Θ(1), delete at head with Θ(1). Double eded queue (deque) Isertio ad deletio ca be doe at both eds. Ca be implemeted usig a doubly liked list. Isertio ad deletio i Θ(1). ADL & AG Data Structures ad Algorithms 90 / 133

Bouded Queues ad Circular Arrays If the size of a queue (simple or double eded) is bouded it ca be implemeted ecietly usig a circular array. 5 1 tail 3 7 2 8 head I that case, access to the kth elemet ca be doe i Θ(1) istead of Θ() with a list. Access(S, k) = A[(head + k 1) mod ] The couterpart is that, Isert() ad Erase() at positio r become O(mi(r, r)) istead of O(1). How ca we exted this circular scheme to ubouded queues? ADL & AG Data Structures ad Algorithms 91 / 133 Ubouded Queues ad Circular Arrays How to add a etry to a circular array that is full? 1st idea Augmet the size of the array. I practice: ew memory allocatio the copy. The isertio becomes i Θ() whe it happes. Complexity stays i amortized Θ(1) with proper growth. 2d idea Make a dyamic circular array of dyamic arrays that have costat size. Oly the arrays at both eds are ot full. 5 1 tail head 3 7 2 head tail 8 2 5 3 9 0 tail head 4 9 4 2 head tail It is agai amortized Θ(1) because we sometimes (but less ofte) have to reallocate the master array. This is the implemetatio of the std::deque cotaier i C++. ADL & AG Data Structures ad Algorithms 92 / 133 Priority Queues The elemet removed from a priority queue is always the greatest. Some say Largest I, First Out (but do ot mistake with LIFO = Last I First Out). If the priority queue is doe with a sorted list, the Push() is i O() ad Pop() i Θ(1). If the priority queue is doe with a heap, the Push() is i O(log ) ad Pop() i O(log ). (To Pop() a heap: like i Heap Sort you remove the rst value of the heap, replace it by the last, ad call Heapify to x the heap structure.) Push for Heap Iput: A array A[1..m] with heap property, a value v to isert, Output: A array A[1..m + 1] with heap property ad cotaiig v. HeapPush(A, m, v) 1 i m + 1 2 A[i] v 3 while i > 1 ad A[Paret(i)] < A[i] do 4 A[Paret(i)] A[i] 5 i Paret(i) I the worst case the umber of operatios is proportioal to the height of the heap, so T () = O(log ). Ca you explai how to do Push() o a heap? ADL & AG Data Structures ad Algorithms 93 / 133 ADL & AG Data Structures ad Algorithms 94 / 133

Associative Array A associative array (or dictioary or map) is a abstract data type that ca be see as a geeralizatio of arrays for o-cosecutive idices. Because these idices may ot be itegers, we call these keys. (These keys ca be associated to auxiliary data as i sortig algorithms.) Typical operatios: Addig Deletig Searchig (by key). Updatig (of auxiliary data). I the sequel we shall ot show the auxiliary data for simplicity, but you have to assume that they follow the key every time it is copied (but they are ot used durig comparisos). ADL & AG Data Structures ad Algorithms 95 / 133 Hash Table Goal: represet a set F of items (the keys), let's say a subset of a domai K. We wat to quickly test membership to this subset. If K = N, we ca use a array to represet F. Assumig 0-based array we the have F i A[] 0. However if max(f) is big this array will take a lot of place eve if F is small. Furthermore, this scheme wo't work if K does ot represet itegers. Idea: for ay F K, let's d a fuctio f : K {0,..., m} so we ca the test set-membership as follows: x F i A[f (x)] 0. f has to be ijective for this test to be correct (ad such fuctio f ca exist oly if m 1 K ). These membership tests are i Θ(1) if f is simple. ADL & AG Data Structures ad Algorithms 96 / 133 Ijectivity i K or F Dearth of Ijective Fuctios Give F = {"chat", "chie", "oie", "poule"} to map to {0,..., 30}. Let's take the fuctio f (mot) = (mot[2] 'a'). This fuctios distiguishes words of F by their third letter. f (chat) = 0 f (chie) = 8 f (oie) = 4 f (poule) = 20 It is ot ijective i K: f (loup) = 20 A solutio is to represet the key i the array. The x F i A[f (x)] = x. If m F we ca d a ijective fuctio i F. A[i] i 0 chat 1 / 2 / 3 / 4 oie. / 8 chie. / 20 poule. / Such ijective fuctio f ca hardly be foud by luck. Let F be a set of = 30 items that we wat to represet i a array of m = 40 etries. There are 40 30 10 48 fuctios from F to {0,..., m 1}. Amog these fuctios, oly 40 39 11 = 40!/10! 2.10 41 are ijective. We therefore have oe agaist 5 millio chaces to pick a ijective fuctio at radom. Aother typical example of the dearth of ijective fuctios is the birthday problem: with 23 people, the probability that two people are bor o the same day is more tha 1/2. Still the birthday fuctio oers 365 possible choices! http://e.wikipedia.org/wiki/birthday_problem ADL & AG Data Structures ad Algorithms 97 / 133 ADL & AG Data Structures ad Algorithms 98 / 133