Stork: State of the Art

Transcription

1 Stork: State of the Art Tevfik Kosar Computer Sciences Department University of Wisconsin-Madison 1

2 The Imminent Data deluge Exponential growth of scientific data : : : : ~0.5 Petabyte ~10 Petabytes ~100 Petabytes ~1000 Petabytes I am terrified by terabytes -- Anonymous I am petrified by petabytes -- Jim Gray Moore s Law outpaced by growth of scientific data! 2

3 High Energy Physics: LHC 2-3 PB/year 11 PB/year Astronomy: Bioinformatics: BLAST LSST 2MASS SDSS DPOSS GSC-II WFCAM VISTA NVSS FIRST GALEX ROSAT OGLE TB - 1 PB/year Educational Technology: WCER EVP 500 TB/year 3

4 How to access and process distributed data? TB TB PB PB 4

5 I/O Management in the History CPU BUS HARDWARE LEVEL DMA I/O PROCESSOR MEMORY CONTROLLER CPU HARDWARE LEVEL I/O PROCESSOR MEMORY DMA BUS DISK CONTROLLER DISK 5

6 I/O Management in the History I/O SUBSYSTEM OPERATING SYSTEMS LEVEL OPERATING SYSTEMS LEVEL I/O SCHEDULER I/O SUBSYSTEM CPU SCHEDULER I/O CPU SCHEDULER I/O CONTROL SYSTEM SCHEDULER I/O CONTROL SYSTEM CPU HARDWARE LEVEL I/O PROCESSOR MEMORY DMA BUS CONTROLLER DISK 6

7 I/O Management in the History BATCH SCHEDULERS DISTRIBUTED SYSTEMS LEVEL I/O SUBSYSTEM OPERATING SYSTEMS LEVEL I/O CPU SCHEDULER SCHEDULER I/O CONTROL SYSTEM CPU HARDWARE LEVEL I/O PROCESSOR MEMORY DMA BUS CONTROLLER DISK 7

8 I/O Management in the History DISTRIBUTED SYSTEMS LEVEL BATCH SCHEDULERS DATA PLACEMENT SCHEDULER I/O SUBSYSTEM OPERATING SYSTEMS LEVEL I/O CPU SCHEDULER SCHEDULER I/O CONTROL SYSTEM CPU HARDWARE LEVEL I/O PROCESSOR MEMORY DMA BUS CONTROLLER DISK 8

9 JOB i JOB i Allocate space for input & output data Stage-in Stage-in Execute job j Execute job j JOB i get JOB j put Stage-out Release input space Stage-out JOB k Release output space JOB k Compute Jobs Individual Jobs Jobs Data placement JOB k 9

10 Separation of Jobs Compute Job C Queue DAG specification Data A A.stork Data B B.stork Job C C.condor.. Parent A child B Parent B child C Parent C child D, E.. A B C D E F E Data Job Queue Workflow Manager 10

11 Stork: Data Placement Scheduler First scheduler specialized for data movement/placement. De-couples data placement from computation. Understands the characteristics and semantics of data placement jobs. Can make smart scheduling decisions for reliable and efficient data placement. A prototype is already implemented and deployed at several sites. Now distributed with Condor Developers Release v

12 [ICDCS 04] Support for Heterogeneity Provides uniform access to different data storage systems and transfer protocols. Acts as an IOCS for distributed systems. Multilevel Policy Support [ Type = Transfer ; Src_Url = srb://ghidorac.sdsc.edu/kosart.condor/x.dat ; Dest_Url = nest://turkey.cs.wisc.edu/kosart/x.dat ; Max_Retry = 10; Restart_in = 2 hours ; ] Protocol translation: using Stork Memory Buffer using Stork Disk Cache 12

13 [ICDCS 04] Dynamic Protocol Selection [ dap_type = transfer ; src_url = drouter://slic04.sdsc.edu/tmp/test.dat ; dest_url = drouter://quest2.ncsa.uiuc.edu/tmp/test.dat ; alt_protocols = gsiftp-gsiftp, nest-nest ; or: src_url = any://slic04.sdsc.edu/tmp/test.dat ; dest_url = any://quest2.ncsa.uiuc.edu/tmp/test.dat ; ] DiskRouter crashes Traditional Scheduler: 48 Mb/s Using Stork: 72 Mb/s 13 DiskRouter resumes

14 [AGridM 03] Run-time Auto-tuning [ link = slic04.sdsc.edu quest2.ncsa.uiuc.edu ; protocol = gsiftp ; bs = 1024KB; // I/O block size tcp_bs = 1024KB; // TCP buffer size p = 4; // number of parallel streams Traditional Scheduler (without tuning) 0.5 MB/s Using Stork (with tuning) 10 MB/s ] Before Tuning: GridFTP parallelism = 1 block_size = 1 MB tcp_bs = 64 KB After Tuning: parallelism = 4 block_size = 1 MB tcp_bs = 256 KB 14

15 [Europar 04] Controlling Throughput Wide Area Local Area Increasing concurrency/parallelism does not always in crease transfer rate Effect on local area and wide are is different Concurrency and parallelism have slightly different impacts on transfer rate 15

16 [Europar 04] Controlling CPU Utilization Client Server Concurrency and parallelism have totally opposite impacts on CPU utilization at the server side. 16

17 [Grid 04] Detecting and Classifying Failures Check DNS Server F DNS Server error Transient No DNS entry Permanent Network Outage Transient Host Down Transient F Protocol Unavailable Transient F Not Authenticated Permanent F Source File Does Not Exist Permanent S Check DNS S Check Network F F S Check Host F S Check Protocol S Check Credentials S CheckS File S Test Transfer F Transfer Failed POLICIES 17

18 [Cluster 04] Detecting Hanging Transfers Collecting job execution time statistics Fit a distribution Detect and avoid Transfer Time (T) vs Probability (t<t) % Probability (t<t) (%) black holes hanging transfers Eg. for normal distribution: 99.7% of job execution times should lie between [(avg-3*stdev), (avg+3*stdev)] Transfer Time (T) (minutes) min 18

19 Stork can also: Allocate/de-allocate (optical) network links Allocate/de-allocate storage space Register/un-register files to Meta Data Catalog Locate physical location of a logical file name Control concurrency levels on storage servers You can refer to [ICDCS 04][JPDC 05][AGridM 03] 19

20 Apply to Real Life Applications 20

21 DPOSS Astronomy Pipeline 21

22 Failure Recovery UniTree not responding SDSC cache reboot & UW CS Network outage Diskrouter reconfigured and restarted Software problem 22

23 End-to-end Processing of 3 TB DPOSS Astronomy Data Traditional Scheduler: 2 weeks Using Stork: 6 days 23

24 Summary Stork provides solutions for the data placement needs of the Grid community. It is ready to fly! Now distributed with Condor developers release v All basic features you will need are included in the initial release. More features coming in the future releases. 24

25 Thank you for listening.. Questions? 25