I/O-Algorithms Lars Arge

Transcription

1 Fall 2014 August 28, 2014

2 assive Data Pervasive use of computers and sensors Increased ability to acquire/store/process data assive data collected everywhere Society increasingly data driven Access/process data anywhere any time ature/science Obviously not only special in sciences: issues Economist 2/06, 9/08, 02/10: 2/11 Scientific From 150 data illion size growing Gigabytes exponentially, five years ago while to 1200 quality illion and today availability improving anaging data deluge difficult; doing so Paradigm shift: Science will be about mining data will transform business/public life 2

3 assive Data What happens in an internet minute? 3

4 Example: Grid Terrain Data Appalachian ountains (800km x 800km) 100m resolution ~ 64 cells ~128 raw data (~500 when processing) ~ 1.2G at 30m resolution ASA SRT mission acquired 30m data for 80% of the earth land mass ~ 12G at 10m resolution (US available from USGS) ~ 1.2T at 1m resolution (quickly becoming available) points per m 2 already possible 4

5 Example: LIDAR Terrain Data assive (irregular) point sets (~1m resolution) ecoming relatively cheap and easy to collect Sub-meter resolution using mobile mapping 5

6 Example: LIDAR Terrain Data COWI A/S (and others) have scanned Denmark 6

7 Example: LIDAR Terrain Data ~2 million points at 30 meter (<1G) ~18 billion points at 1 meter (>1T) 7

8 Application Example: Flooding Prediction +1 meter +2 meter 8

9 Example: Detailed Data Essential andø with 2 meter sea-level raise 80 meter terrain model 2 meter terrain model 9

10 Random Access achine odel R A Standard theoretical model of computation: Infinite memory Uniform access cost Simple model crucial for success of computer industry 10

11 Hierarchical emory L 1 L 2 R A odern machines have complicated memory hierarchy Levels get larger and slower further away from CPU Data moved between levels using large blocks 11

12 Slow I/O Disk access is 10 6 times slower than main memory access read/write head track read/write The difference arm in speed between modern CPU and disk technologies is analogous to the difference in speed in sharpening a pencil using a sharpener on magnetic surface one s desk or by taking an airplane to the other side of Disk systems try to amortize large access time the transferring world and using large a contiguous blocks of data sharpener on someone else s Important to store/access data to take advantage of desk. blocks (D. (locality) Comer) 12

13 running time I/O-Algorithms Scalability Problems ost programs developed in RA-model Run on large datasets because OS moves blocks as needed oderns OS utilizes sophisticated paging and prefetching strategies ut if program makes scattered accesses even good OS cannot take advantage of block access Scalability problems! data size 13

14 External emory odel lock I/O D = # of items in the problem instance = # of items per disk block = # of items that fit in main memory T = # of items in output I/O: ove block between memory and disk P We assume (for convenience) that > 2 14

15 Scanning: Sorting: Permuting Searching: Fundamental ounds Internal log log 2 External log min{, log log } ote: Linear I/O: O(/) Permuting not linear Permuting and sorting bounds are equal in all practical cases factor VERY important: Cannot sort optimally with search tree log 15

16 Scalability Problems: lock Access atters Example: Traversing linked list (List ranking) Array size = 10 elements Disk block size = 2 elements ain memory size = 4 elements (2 blocks) Algorithm 1: =10 I/Os Algorithm 2: /=5 I/Os Large difference between and / large since block size is large Example: = 256 x 10 6, = 8000, 1ms disk access time I/Os take 256 x 10 3 sec = 4266 min = 71 hr / I/Os take 256/8 sec = 32 sec 16

17 Queue: Queues and Stacks aintain push and pop blocks in main memory Push Pop O(1/) Push/Pop operations Stack: aintain push/pop blocks in main memory O(1/) Push/Pop operations 17

18 Sorting </ sorted lists (queues) can be merged in O(/) I/Os / blocks in main memory Unsorted list (queue) can be distributed using </ split elements in O(/) I/Os 18

19 Sorting erge sort: Create / memory sized sorted lists Repeatedly merge lists together Θ(/) at a time O(log ) phases using O( ) I/Os each I/Os O log ) ( 1 ( ) / ( ( /( ) ) 2 ) 19

20 Sorting Distribution sort (multiway quicksort): Compute Θ(/) splitting elements Distribute unsorted list into Θ(/) unsorted lists of equal size Recursively split lists until fit in memory O(log ) O log ( phases ) I/Os if splitting elements computed in O(/) I/Os 20

21 Computing Splitting Elements In internal memory (deterministic) quicksort split element (median) found using linear time selection Selection algorithm: Finding i th element in sorted order 1) Select median of every group of 5 elements 2) Recursively select median of ~ /5 selected elements 3) Distribute elements into two lists using computed median 4) Recursively select in one of two lists Analysis: Step 1 and 3 performed in O(/) I/Os. 7 Step 4 recursion on at most ~ elements 10 T ) O( ) T( ) T( 7 ) O( ) I/Os (

22 Sorting Distribution sort (multiway quicksort): Computing splitting elements: Θ(/) times linear I/O selection O(/ 2 ) I/O algorithm ut can use selection algorithm to compute splitting 3 elements in O(/) I/Os, partitioning into lists of size < 2 O(log ) O(log ) phases O log ) algorithm ( 22

23 Computing Splitting Elements 4 1) Sample elements: Create / memory sized sorted lists Pick every 1 th element from each sorted list 4 2) Choose split elements from sample: Use selection algorithm times to find every 4 th element 4 Analysis: Step 1 performed in O(/) I/Os Step 2 performed in O ) O( ) I/Os O(/) I/Os ( 23

24 4 1) Sample elements: Computing Splitting Elements Create / memory sized sorted lists Pick every 1 th element from each sorted list 2) Choose split elements from sample: 4 Use selection algorithm times to find every th element sampled elements 24

25 Computing Splitting Elements Elements in range R defined by consecutive split elements 4 Sampled elements in R: 1 etween sampled elements in R: 4 1 1) ( 1) ( 4 etween sampled element in R and outside R: ) ( ( 4 1) sampled elements 25

26 Summary/Conclusion: Sorting External merge or distribution sort takes ( O log ) erge-sort based on /-way merging Distribution sort based on -way distribution and partition elements finding I/Os Optimal As we will prove next time 26

27 References Input/Output Complexity of Sorting and Related Problems A. Aggarwal and J.S. Vitter. CAC 31(9), 1998 External partition element finding Lecture notes by L. Arge and. G. Lagoudakis. 27

28 Project 1: Implementation of erge Sort 28