Extreme scripting and other adventures in data-intensive computing

Size: px

Start display at page:

Download "Extreme scripting and other adventures in data-intensive computing"

Beverly Smith
5 years ago
Views:

1 Extreme scripting and other adventures in data-intensive computing Ian Foster Allan Espinosa, Ioan Raicu, Mike Wilde, Zhao Zhang Computation Institute Argonne National Lab & University of Chicago

2 How data analysis happens at dataintensive computing workshops 2

3 How data analysis really happens in scientific laboratories 3 % foo file1 > file2 % bar file2 > file3 % foo file1 bar > file3 % foreach f (f1 f2 f3 f4 f5 f6 f7 f100) foreach? foo $f.in bar > $f.out foreach? end % % Now where on earth is f98.out, and how did I generate it again? Now: command not found. %

4 Extreme scripting 4 Many activities Numerous files Complex data Data dependencies Many programs Preserving file system semantics, ability to call arbitrary executables Complex scripts Simple scripts Small computers Swift Big computers Many processors Storage hierarchy Failure Heterogeneity

5 Functional magnetic resonance imaging (fmri) data analysis 5

AIRSN program definition 6 (Run snr) functional ( Run r, NormAnat a, } Air shrink ) { Run yrorun = reorientrun( r, "y" ); Run rorun = reorientrun( yrorun, "x" ); Volume std = rorun[0]; Run rndr =

6 AIRSN program definition 6 (Run snr) functional ( Run r, NormAnat a, } Air shrink ) { Run yrorun = reorientrun( r, "y" ); Run rorun = reorientrun( yrorun, "x" ); Volume std = rorun[0]; Run rndr = random_select( rorun, 0.1 ); AirVector rndairvec = align_linearrun( rndr, std, 12, 1000, 1000, "81 3 3" ); Run reslicedrndr = reslicerun( rndr, rndairvec, "o", "k" ); Volume meanrand = softmean( reslicedrndr, "y", "null" ); Air mnqaair = alignlinear( a.nhires, meanrand, 6, 1000, 4, "81 3 3" ); Warp boldnormwarp = combinewarp( shrink, a.awarp, mnqaair ); Run nr = reslice_warp_run( boldnormwarp, rorun ); Volume meanall = strictmean( nr, "y", "null" ) Volume boldmask = binarize( meanall, "y" ); snr = gsmoothrun( nr, boldmask, "6 6 6" ); (Run or) reorientrun (Run ir, string direction) { foreach Volume iv, i in ir.v { or.v[i] = reorient(iv, direction); } }

7 Many many tasks: Identifying potential drug targets 7 Protein target(s) x 2M+ ligands Benoit Roux et al.

8 PDB protein descriptions ZINC 3-D structures 1 protein (1MB) 6 GB 2M structures (6 GB) Manually prep DOCK6 rec file DOCK6 Receptor (1 per protein: defines pocket to bind to) Manually prep FRED rec file FRED Receptor (1 per protein: defines pocket to bind to) NAB Script Template BuildNABScript NAB script parameters (defines flexible residues, #MDsteps) 8 FRED start DOCK6 ~4M x 60s x 1 cpu ~60K cpu-hrs NAB Script Amber prep: 2. AmberizeReceptor 4. perl: gen nabscript Select best ~5K Select best ~5K Amber Select best ~500 ~10K x 20m x 1 cpu ~3K cpu-hrs Amber Score: 1. AmberizeLigand 3. AmberizeComplex 5. RunNABScript report GCMC end ligands ~500 x 10hr x 100 cpu ~500K cpu-hrs complexes For 1 target: 4 million tasks 500,000 cpu-hrs (50 cpu-years)

9 9

10 10 IBM BG/P 570 Teraflop/s, 164,000 cores, 80 TB

11 11 DOCK on BG/P: ~1M tasks on 119,000 CPUs Ioan Raicu et al. Time (sec) cores tasks Elapsed time: 7257 sec Compute time: CPU years Average task: 667 sec Relative efficiency 99.7% (from 16 to 32 racks) Utilization: 99.6% sustained, 78.3% overall

12 Managing 160,000 cores 12 Falkon High-speed local disk Slower shared storage

13 Chirp (multicast) Large dataset Global file system ZOID IFS IFS seg ZOID on I/O node IFS compute node Staging Torus and tree interconnects CN-striped intermediate file system IFS seg IFS compute node Scaling Posix to petascale 13 Intermediate MosaStore (striping) LFS Compute node (local datasets)... LFS Compute node (local datasets) Local

14 Efficiency for 4 second tasks and varying data size (1KB to 1MB) for CIO and GPFS up to 32K processors 14

Provisioning for data-intensive workloads 15 Example: on-demand stacking of arbitrary locations within ~10TB sky survey Challenges Random data

15 Provisioning for data-intensive workloads 15 Example: on-demand stacking of arbitrary locations within ~10TB sky survey Challenges Random data access Much computing Time-varying load Solution Dynamic acquisition of compute & storage Data diffusion S = Sloan Data Ioan Raicu

16 Sine workload, 2M tasks, 10MB:10ms ratio, 100 nodes, GCC policy, 50GB caches/node 16 Ioan Raicu

17 Same scenario, but with dynamic resource provisioning 17

18 GPFS Data diffusion sine-wave workload: Summary 5.70 hrs, ~8Gb/s, 1138 CPU hrs DD+SRP 1.80 hrs, ~25Gb/s, 361 CPU hrs 18 DD+DRP 1.86 hrs, ~24Gb/s, 253 CPU hrs

19 19 Data-intensive Computation Institute: Example applications Astrophysics Cognitive science East Asian studies Economics Environmental science Epidemiology Genomic medicine Neuroscience Political science Sociology Solid state physics

20 20

21 Folker Meyer, Computation Institute BLAST On EC2, US$ $300,000 Sequencing outpaces Moore s law Bioinformatics Sequencing $240,000 $300,000 $600,000 $900, $30,000 $120, Solexa Next-gen Solexa $7,000 $3,000 $3, Gigabases

22 Data-intensive Computation Institute: Hardware PADS: Petascale Active Data Store (NSF MRI) 22 Diverse data sources Data ingest 1000 TB tape backup 500 TB reliable storage (data & metadata) P A D S 180 TB, 180 GB/s 17 Top/s analysis Dynamic provisioning Parallel analysis Remote access Diverse users Offload to remote data centers

23 Data-intensive Computation Institute: Software HPC systems software (MPICH, PVFS, ZeptOS) 23 Collaborative data tagging (GLOSS) Data integration (XDTM) HPC data analytics and visualization Loosely coupled parallelism (Swift, Hadoop) Dynamic provisioning (Falkon) Service authoring (Introduce, cagrid, gravi) Provenance recording and query (Swift) Service composition and workflow (Taverna) Virtualization management (Workspace Service) Distributed data management (GridFTP, etc.)

24 Data-intensive computing is an end-to-end problem 24 Low Chaos Agreement about outcomes Zone of complexity High Plan and control High Certainty about outcomes Ralph Stacey, Complexity and Creativity in Organizations, 1996 Low

25 We need to function in the zone of complexity 25 Low Chaos Agreement about outcomes High Plan and control High Certainty about outcomes Ralph Stacey, Complexity and Creativity in Organizations, 1996 Low

1995 2000 2005 2010 26 The Grid paradigm

services Open software, architecture Computer

26 The Grid paradigm Principles and mechanisms for dynamic virtual organizations Leverage service oriented architecture Loose coupling of data and services Open software, architecture Computer science Physics Astronomy Engineering Biology Biomedicine Healthcare

27 27 As of Oct 19, 2008: 122 participants 105 services 70 data 35 analytical

28 Multi-center clinical cancer trials image capture and review 28 (Center for Health Informatics)

29 29 Summary Extreme scripting offers the potential for easy scaling of proven working practices Interesting technical problems relating to programming and I/O models Many wonderful applications Data-intensive computing is an end-to-end problem Data generation, integration, analysis, etc., is a continuous, loosely coupled process

30 Thank you! Computation Institute

Grid, cloud, and science: Accelerating discovery. A View and Practice from University of Chicago

Grid, cloud, and science: Accelerating discovery A View and Practice from University of Chicago Ian Foster Presented by Ioan Raicu Computation Institute Argonne National Lab & University of Chicago April