4 Sorting Atomic Items

Transcription

1 4 Sortig Atomic Items 4.1 The merge-based sortig paradigm A lower boud o I/Os The distributio-based sortig paradigm Partitioig Pivot Selectio Limitig the umber of recursio levels The qsort stadard algorithm Sample Sort 4.4 Sortig with multi-disks A geeral techique: disk stripig Sortig o multi-disks This lecture will focus o the very-well kow problem of sortig a set of atomic items, the case of variable-legth items (aka strigs) will be addressed i the followig chapter. Atomic meas that they occupy O(1) space, typically they are itegers or reals represeted with a fixed umber of bytes, such as 4 (32 bits) or 8 (64 bits) bytes each. The sortig problem. Give a sequece of atomic items S [1, ] ad a total orderig betwee each of them, sort S i icreasig order. We will cosider two complemetal sortig paradigms: the merge-based paradigm, which uderlies the desig of Mergesort, ad the distributio-based paradigm which uderlies the desig of Quicksort. We will adapt them to work i the disk model (see Chapter 1), aalyze their I/O-complexities ad propose some useful tools that ca allow to speed up their executio i practice, such as the Sow Plow techique ad Data compressio. We will also demostrate that these disk-based adaptatios are I/O-optimal by provig a sophisticated lower-boud o the umber of I/Os ay exteralmemory sorter must execute to produce a ordered sequece. I this cotext we will relate the Sortig problem with the so called Permutig problem, typically eglected whe dealig with sortig i the RAM model. The permutig problem. Give a sequece of atomic items S [1, ] ad a permutatioπ[1, ] of the itegers{1, 2,...,}, permute S accordig toπthus obtaiig the ew sequece S [π[1]], S [π[2]],...,s[π[]]. Clearly Sortig icludes Permutig as a sub-task: to order the sequece S we eed to determie its sorted permutatio ad implemet it (possibly these two phases are itricately itermigled). So Sortig should be more difficult tha Permutig. Ad ideed i the RAM model we kow that sortig atomic items takes Θ( log ) time (via Mergesort or Heapsort) whereas permutig them takesθ() time. The latter time boud ca be obtaied by just movig oe item at a time accordig to what is idicated i the arrayπ. But surprisigly we will show that this complexity gap does ot exist i the disk model, i that they exhibit the same I/O-complexity uder some reasoable c Paolo Ferragia 4-1

2 4-2 Paolo Ferragia coditios o the iput ad model parameters, M, B. This elegat ad deep result was obtaied by Aggarwal ad Vitter i 1998 [1], ad it is surely the result that spurred the huge amout of algorithmic literature thereafter produced o the I/O-subject. Philosophically speakig, AV s result formally proves the ituitio that movig items i the disk is the real efficiecy bottleeck, rather tha the uderlyig computatio eeded to determie those movemets. Ad ideed researchers ad software egieers typically speak about the I/O-bottleeck to characterize this issue i their (slow) algorithms. We will coclude this lecture by briefly metioig at two solutios for the problem of sortig items o D-disks: the disk-stripig techique, which is at the base of RAID systems ad turs ay efficiet/optimal 1-disk algorithm ito a efficiet D-disk algorithm (typically loosig its optimality, if ay), ad the Greed-sort algorithm, which is specifically tailored for the sortig problem ad achieves I/O-optimality. 4.1 The merge-based sortig paradigm We recall the mai features of the exteral-memory model itroduced i Chapter 1: it cosists of a iteral memory of size M ad it allows blocked-access to disk data by readig/writig B items at oce. Algorithm 4.1 The biary merge-sort: MergeSort(S, i, j) 1: if (i< j) the 2: m=(i+ j)/2; 3: MergeSort(S, i, m 1); 4: MergeSort(S, m, j); 5: Merge(S, i, m, j); 6: ed if Mergesort is based o the Divide&Coquer paradigm. Step 1 checks if the array to be sorted cosists of at least two items, otherwise it is already ordered ad othig has to be doe. Otherwise it splits the iput array S ito two halves, accordig to the positio m of the middle elemet, ad the recurse o each part. As recursio eds, the two halves S [i, m 1] ad S [m, j] are sorted so that Step 5 ivokes procedure Merge that fuses their cotets ad produce a uique sorted sequece that is stored i S [i, j]. This mergig eeds a auxiliary array of size, so that MergeSort is ot a i-place sortig algorithm (ulike Heapsort ad Quicksort) but eeds O() extra workig space. Give that at each recursive call we halve the size of the iput array to be sorted, the total umber of recursive calls is O(log ). The Merge-procedure ca be implemeted i O( j i+1) time by usig two poiters, say x ad y, that start at the begiig of the two halves S [i, m 1] ad S [m, j]. The S [x] is compared with S [y], the smaller is writte out i the fused sequece, ad its poiter is advaced. Give that each compariso advace oe poiter, the total umber of steps is bouded above by the total umber of poiter s advacemets, which is upper bouded by the legth of S [i, j]. So the time complexity of MergeSort(S, 1, ) ca be modeled via the recurret relatio T()=2T(/2)+O()=O( log ), as well kow from ay basic algorithm course. 1 Let us assume ow that > M, so that S must be stored o disk ad I/Os because the most im- 1 I all our lectures whe the base of the logarithm is ot idicated, it meas 2.

3 Sortig Atomic Items 4-3 portat resource to be aalyzed. I practice every I/O takes 5ms o average, so oe could thik that every item compariso takes oe I/O ad thus estimate the ruig time of Mergesort o massive data as: 5ms Θ( log ). If is of the order of few Gigabytes, say which is actually ot much massive for the curret-size of commodity PCs, the previous time estimate would be of about >10 8 ms, amely more tha 1 day of computatio. However, if we ru Mergesort o a commodity PC it completes i few hours. This is ot surprisig because the previous evaluatio totally eglected the existece of the iteral memory, of size M, ad the sequetial patter of memory-accesses iduced by Mergesort. Let us therefore aalyze the Mergesort algorithm i a more precise way withi the disk model. First of all we otice that O(z/B) I/Os is the cost of mergig two ordered sequeces of z items i total. This holds if M 2B, because the Merge-procedure i Algorithm?? ca keep i iteral memory the 2 pages that cotai the two poiters scaig S [i, j]. Every time a poiter advaces ito aother disk page, a I/O-fault occurs, the page is fetched i iteral memory, ad the fusio cotiues. Give that S is stored cotiguously o disk, it occupies O(z/B) pages ad this is the I/O-boud for mergig two sub-sequeces of total size z. Similarly, the I/O-cost for writig the merged sequece is O(z/B) because it occurs sequetially from the smallest to the largest item of S [i, j]. As a result the recurret relatio for the I/O-complexity of Mergesort ca be writte as T() = 2T(/2)+O(/B) = O( B log ) I/Os. But this formula does ot explai completely the good behavior of Mergesort i practice, because it does ot accout for the memory hierarchy yet. I fact as Mergesort recursively splits the sequece S, smaller ad smaller sub-sequeces are geerated that have to be sorted. So whe a subsequece of legthlfits i iteral memory, amelyl=θ(m), the it will be etirely cached by the uderlyig operatig system usig O(l/B) I/Os ad thus, the subsequet sortig steps would ot icur i ay I/Os. The et result of this simple observatio is that the I/O-cost of sortig a sub-sequece ofl=o(m) items is o loger T(l)=Θ( l B logl) but O(l/B). This savig applies to all S s subsequeces of sizeθ(m) o which Mergesort is recursively ru, which areθ(/m) i total. So the overall savig isθ( B log M), which leads us to re-formulate the Mergesort s complexity asθ( B log M ) I/Os. This boud is particularly iterestig because relates the I/O-complexity of Mergesort ot oly to the disk-page size B but also to the iteral-memory size M, ad thus to the cachig available for the computatio. Moreover this bouds suggests three immediate optimizatios to the classic pseudocode of Algorithm 4.3 the we discuss below. Optimizatio #1: Takig care of iteral-memory size. The first optimizatio cosists of itroducig a threshold o the subsequece size, say j i<cm, which triggers the stop of the recursio, the fetchig of that subsequece etirely i iteral-memory, ad the applicatio of a iteralmemory sorter o this sub-sequece (see Figure 4.1). The value of the parameter c depeds o the space-occupacy of the sorter, which must be guarateed to work etirely i iteral memory. As a example, c is 1 for i-place sorters such as Isertiosort ad Heapsort, it is much close to 1 for Quicksort (because of its recursio), ad it is less tha 0.5 for Mergesort (because of the extra-array used by Merge). As a result, we should write cm istead of M ito the I/O-boud above, because recursio is stopped at cm items: thus obtaiigθ( B log cm ). This substitutio is useless whe dealig with asymptotic aalysis, give that c is a costat, but it is importat whe cosiderig the real performace of algorithms. I this settig it is desirable to make c as closer as possible to 1, i order to reduce the logarithmic factor i the I/O-complexity thus preferrig i-place sorters such as Heapsort or Quicksort. We remark that Isertiosort could also be a good choice (ad ideed it is) wheever M is small, as it occurs whe cosiderig the sortig of items over the 2-levels: L1 ad L2 caches, ad the iteral memory. I this case M would be few Megabytes. Optimizatio #2: Virtually elargig M. Lookig at the I/O-complexity of mergesort, i.e.θ( B log M ), is clear that the larger is M the smaller is the umber of merge-passes over the data. These are clearly the bottleeck to the efficiet executio of the algorithm especially i the presece of disks with low

4 4-4 Paolo Ferragia badwidth. I order to circumvet this problem we ca either buy a larger memory, or try to deploy as much as possible the oe we have available. As algorithm egieer we opt for the secod possibility ad thus propose two techiques that ca be combied together i order to elarge (virtually) M. The first techique is based o data compressio ad builds upo the observatio that the rus are icreasigly sorted. So, istead of represetig items via a fixed-legth codig (e.g. 4 or 8 bytes), we ca use iteger compressio techiques that squeeze those items i fewer bits thus allowig us to pack more of them i iteral memory. A followig lecture will describe i detail several approaches to this problem (see Chapter??), here we cotet ourselves metioig the ames of some of these approaches: γ-code, δ-code, Rice/Golomb-codig, etc. etc.. I additio, sice the smaller is a iteger the fewer bits are used for its ecodig, we ca eforce the presece of small itegers i the sorted rus by ecodig ot just their absolute value but the differece betwee oe iteger ad the previous oe i the sorted ru (the so called delta-codig). This differece is surely o egative (equals zero if the ru cotais equal items), ad smaller tha the item to be ecoded. This is the typical approach to the ecodig of iteger sequeces used i moder search egies, that we will discuss i a followig lecture (see Chapter??). FIGURE 4.1: Whe a ru fits i the iteral memory of size M, we applyqsort over its items. I gray we depict the recursive calls that are executed i iteral memory, ad thus do ot elicit I/Os. Above there are the calls based o classic Mergesort, oly the call o 2M items is show. The secod techique is based o a elegat idea, called thesow Plow ad due to D. Kuth [2], that allows to virtually icrease the memory size of a factor 2 o average. This techique scas the iput sequece S ad geerates sorted rus of whose legth has variable size loger tha M ad 2M o average. Its use eeds to chage the sortig scheme because it first creates these sorted rus, of variable legth, ad the applies repeatedly over the sorted rus the Merge-procedure. Although rus will have differet legths, the Merge will operate as usual requirig a optimal umber of I/Os for their mergig. Hece O(/B) I/Os will suffice to halve the umber of rus, ad thus a total of O( B log 2M ) I/Os will be used o average to produce the totally ordered sequece. This correspods to a savig of 1 pass over the data, which is o egligible if the sequece S is very log. For ease of descriptio, let us assume that items are trasferred oe at a time from disk to memory, istead that block-wise. Evetually, sice the algorithm scas the iput items it will be apparet that the umber of I/Os required by this process is liear i their umber (ad thus optimal). The algorithm proceeds i phases, each phase geerates a sorted ru (see Figure 4.2 for a illustrative example). A phase starts with the iteral-memory filled of M (usorted) items, stored i a heap data structure called H. Sice the array-based implemetatio of heaps requires o additioal space, i additio to the idexed items, we ca fit ih as may items as we have memory cells available.

5 Sortig Atomic Items 4-5 FIGURE 4.2: A illustratio of four steps of a phase i Sow Plow. The leftmost picture shows the startig step i whichu is heapified, the a picture shows the output of the miimum elemet i H, hece the two possible cases for the isertio of the ew item, ad fially the stoppig coditio i whichh is empty adu fills etirely the iteral memory. The phase scas the iput sequece S (which is usorted) ad at each step, it writes to the output the miimum item withih, saymi, ad loads i memory the ext item from S, sayext. Sice we wat to geerate a sorted output, we caot storeext ih ifext<mi, because it will be the ew heap-miimum ad thus will be writte out at the ext step thus destroyig the property of icreasig ru. So i that caseext must be stored i a auxiliary array, calledu, which stays usorted. Of course the total size ofh adu is M over the whole executio of a phase. A phase stops wheeverh is empty ad thusu cosists of M usorted items, ad the ext phase ca thus start (storig those items i a ew heaph ad emptyigu). Two observatios are i order: (i) durig the phase executio, the miimum ofh is o decreasig ad so it is the output, (ii) the items ih at the begiig of the phase will be evetually writte to output which thus is loger tha M. Observatio (i) implies the correctess, observatio (ii) implies that this approach is ot less efficiet tha the classic Mergesort. Algorithm 4.2 A phase of the Sow-Plow techique Require: U is a usorted array of M items 1: H= build a mi-heap overu s items; 2: SetU= ; 3: while (H ) do 4: mi=extract miimum fromh; 5: Writemi to the output ru; 6: ext=read the ext item from the iput sequece; 7: if (ext < mi) the 8: writeext iu; 9: else 10: isertext ih; 11: ed if 12: ed while Actually it is more efficiet tha that o average. Suppose that a phase readsτitems i total from S. By the while-guard i Step 3 ad our commets above, we ca derive that a phase eds wheh is empty ad U = M. We kow that the read items go i part ih ad i part iu. But sice items are added tou ad ever removed durig a phase, M of theτitems ed-up iu. Cosequetly (τ M) items are iserted ih ad evetually writte to the output (sorted) ru. So the legth of the sorted ru at the ed of the phase is M+ (τ M), where the first addedum accouts for the items ih at the begiig of a phase, whereas the secod addedum accouts for the items read

6 4-6 Paolo Ferragia from S ad iserted ih durig the phase. The key issue ow is to compute the average ofτ. This is easy if we assume a radom distributio of the iput items. I this case we have probability 1/2 thatext is smaller thami, ad thus we have equal probability that a read item is iserted either ih or iu. Overall it follows thatτ/2 items go toh adτ/2 items go tou. But we already kew that the items iserted iu were M, so we ca set M=τ/2 ad thusτ=2m. Substitutig i the formula M+ (τ M) we get that the average umber of items writte i the sorted ru is 2M. FACT 4.1 Sow-Plow builds O(/M) sorted rus, each loger tha M ad actually of legth 2M o average. Usig Sow-Plow for the formatio of sorted rus i a Merge-based sortig scheme achieves a I/O-complexity of O( B log 2 2M ). Optimizatio #3: From biary to multi-way Mergesort. Previous optimizatios deployed the iteral-memory size M to reduce the umber of recursio levels by icreasig the size of the iitial (sorted) rus. But the the mergig was biary i that it fused two iput rus at a time. This biarymerge impacted oto the base 2 of the logarithm of the I/O-complexity of Mergesort. Here we wish to icrease that base to a much larger value, ad i order to get this goal we eed to deploy the memory M also i the mergig phase by elargig the umber of rus that are fused at a time. I fact the merge of 2 rus uses oly 3 blocks of the iteral memory: 2 blocks are used to cache the curret disk pages that cotai the compared items, amely S [x] ad S [y] from the otatio above, ad 1 block is used to cache the output items which are flushed whe the block is full (so to allow a blockwise writig to disk of the merged ru). But the iteral memory cotais a much larger umber of blocks, i.e. M/B 3, which remai uused over the whole mergig process. The third optimizatio we propose cosists of deployig all those blocks by desigig a k-way mergig scheme that fuses k rus at a time, with k 2. Let us set k=(m/b) 1, so that k blocks are available to read block-wise k iput rus, ad 1 block is reserved agai for a block-wise writig of the merged ru to disk. This scheme poses a challegig mergig problem because at each step we have to select the miimum amog k cadidates items ad this caot be obviously doe brute-forcedly by iteratig amog them. We eed a smarter solutio that agai higes oto the use of a mi-heap data structure, which cotais k pairs (oe per iput ru) each cosistig of two compoets: oe deotig a item ad the other deotig the origi ru. Iitially the items are the miimum items of the k rus, ad so the pairs have the form R i [1], i, where R i deotes the ith iput ru ad i=1, 2,...,k. At each step, we extract the pair cotaiig the curret smallest item ih (give by the first compoet of its pairs), write that item to output ad isert i the heap the ext item i its origi ru. As a example, if the miimum pair is R m [x], m the we write i output R m [x] ad isert ih the ew pair R m [x+1], m, provided that the mth ru is ot exhausted, i which case o pair replaces the extracted oe. I the case that the disk page cotaiig R m [x+1] is ot cached i iteral memory, a I/O-fault occurs ad that page is fetched, thus guarateeig that the ext B reads from ru R m will ot elicit ay further I/O. It should be clear that this mergig process takes O(log 2 k) time per item, ad agai O(z/B) I/Os to merge k rus of total legth z. As a result the mergigscheme recalls a k-way tree with O(/M) leaves (rus) which ca have bee formed usig ay of the optimizatios above (possibly via Sow Plow). Hece the total umber of mergig levels is ow O(log M/B M ) for a total volume of I/Os equal to O( B log M/B M ). We observe that sometime we will also write the formula as O( B log M/B M ), as it typically occurs i the literature, by exchagig the deomiator withi the logarithm s argumet. This makes o differece asymptotically give that log M/B M =Θ(log M/B B ). THEOREM 4.1 Multi-way Mergesort takes O( B log M/B M ) I/Os ad O( log ) time to sort atomic items i a two-level model i which the iteral memory has size M ad the disk page has

7 Sortig Atomic Items 4-7 size B. The use of Sow-Plow or iteger compressors would virtually icrease the value of M with a twofold advatage i the fial I/O-complexity, because M occurs twice i the I/O-boud. I practice the umber of mergig levels will be very small: assumig a block size B=4KB ad a memory size M= 4GB, we get M/B=2 32 /2 12 = 2 20 so that the umber of passes is 1/20th smaller tha the oes eeded by biary Mergesort. Probably more iterestig is to observe that oe pass is able to sort = M items, but two passes are able to sort M 2 /B items, sice we ca merge M/B-rus each of size M. It goes without sayig that i practice the iteral-memory space which ca be dedicated to sortig is smaller tha the physical memory available (typically MBs versus GBs). Nevertheless it is evidet that M 2 /B is of the order of Terabytes already for M= 128MB ad B=4KB. 4.2 A lower boud o I/Os At the begiig of this lecture we commeted o the relatio existig betwee the Sortig ad the Permutig problems, cocludig that the former oe is more difficult tha the latter i the RAM model. The gap i time complexity is give by a logarithmic factor. The questio we address i this sectio is whether this gap does exist also whe measurig I/Os. Surprisigly eough we will show that i terms of I/O-volume Sortig is equivalet to Permutig. This result is amazig because it ca be read as sayig that the I/O-cost for sortig is ot i the computatio of the sorted permutatio but rather the movemet of the data o the disk to realize it. This is the so called I/O-bottleeck that has i this result the mathematical proof ad quatificatio. Before diggig ito the proof of this lower boud, let us briefly show how a sorter ca be used to permute a sequece of items S [1, ] i accordace to a give permutatioπ[1, ]. This will allow us to derive a upper boud to the umber of I/Os which suffice to solve the Permutig problem o ay S,π. Recall that this meas to geerate the sequece S [π[1]], S [π[2]],...,s[π[]]. I the RAM model we ca jump amog S s items accordig to permutatioπad create the ew sequece S [π[i]] takig O() optimal time. O disk we have actually two differet algorithms which iduce two icomparable I/O-bouds. The first algorithm cosists of mimickig what is doe i RAM, payig oe I/O per moved item ad thus takig O() I/Os. The secod algorithm cosists of geeratig a proper set of tuples ad the sort them. Precisely, the algorithm creates the sequecepof pairs i,π[i] where the first compoet idicates the positio i where the item S [π[i]] must be stored. The it sorts these pairs accordig to theπ-compoet, ad via a parallel sca of S adpsubstitutesπ[i] with the character S [π[i]], thus creatig the ew pairs i, S [π[i]]. Fially aother sort is executed accordig to the first compoet of these pairs, thus obtaiig a sequece of items correctly permuted. The algorithm uses two sca of the data ad two sorts, so it eeds O( B log M/B M ) I/Os. THEOREM 4.2 Permutig items takes O(mi{, B log M/B /M}) I/Os i a two-level model i which the iteral memory has size M ad the disk page has size B. I what follows we will show that this algorithm, i its simplicity, is I/O-optimal. The two upperbouds for Sortig ad Permutig equal each other wheever =Ω( B log M/B M ). This occurs whe B>log M/B M that holds always i practice because that logarithm term is about 2 or 3 for values of up to may Terabytes. So programmers should ot be afraid to fid sophisticated strategies for movig their data i the presece of a permutatio, just sort them, you caot do better! A lower-boud for Sortig. There are some subtle issues here that we wish to do ot ivestigate too much, so we hereafter give oly the ituitio which uderlies the lower-bouds for both Sortig