Data-Parallel Computing Using Madagascar

Size: px

Start display at page:

Download "Data-Parallel Computing Using Madagascar"

Andrew Shields
5 years ago
Views:

1 PASC16 Data-Parallel Computing Using Madagascar Sergey Fomel The University of Texas at Austin

2 45+ Years of Unix

3 Source: Wikimedia Commons

4 Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. Doug McIlroy

5 Source: Wikimedia Commons

In a nutshell, Madagascar has had 12,757 commits made by 97 contributors representing 1,118,533 lines of code is mostly written in C with an average number of source

7 In a nutshell, Madagascar has had 12,757 commits made by 97 contributors representing 1,118,533 lines of code is mostly written in C with an average number of source code comments has a well established, mature codebase maintained by a large development team took an estimated 328 years of effort starting with its first commit in May, 2003

8 Contributors Salah Al-Hadab, Tariq Alkhalifah, Vladimir Bashkardin, Filippo Broggini, Jules Browaeys, Cody Brown, William Burnett, Yihua Cai, Maria Cameron, Lorenzo Casasanta, Yangkang Chen, Zhonghuan Chen, Jiubing Cheng, Luke Decker, Joseph Dellinger, Esteban Diaz, Yuting Duan, Gang Fang, Mehdi Far, Sergey Fomel, Jeff Godwin, Gilles Hennenfent, Jie Hou, Jingwei Hu, Yin Huang, Trevor Irons, Detchai Ittharat, Jim Jennings, Jun Ji, Long Jin, Parvaneh Karimi, Roman Kazinnik, Alexander Klokov, Siwei Li, Guochang Liu, Yang Liu, Yujin Liu, Xuxin Ma, Douglas McCowan, Dmitrii Merzlikin, Henryk Modzelewski, Jorge Monsegny, Francesco Perrone, Jack Poulson, Kelly Regimbal, James Rickett, Sjoerd de Ridder, Daniel Rocha, Sean Ross-Ross, Colin Russell, Christos Saragiotis, Paul Sava, Karl Schleicher, Reza Shahidi, Yunzhi Shi, Jeffrey Shragge, Eduardo Filpo Silva, Xiaolei Song, Yanadet Sripanich, Junzhe Sun, Ioan Vlad, Hui Wang, Robin Weiss, Zedong Wu, Zhiguang Xue, Jia Yan, Jun Yan, Pengliang Yang, Lexing Ying, Zhendong Zhang, Hejun Zhu + Joseph Dellinger, William Symes, and many others

9 Madagascar Contributors

10 Madagascar Schools

11 Qingdao-2015

12 Research Pyramid Publish Test Implement

13 Research Pyramid Papers Workflows Examples Programs

14 Research Pyramid Ø250 Papers LaTeX Ø1,000 Scripts Ø9,000 Figures Ø1,800 Programs Python SCons Unix C

15 Source: Wikimedia Commons

16 Least-Squares Inversion by Conjugate Gradients mathematics geophysics u sfconjgrad sfmodeling vel=vel.rsf \ mod=image0.rsf niter=100 \ < data.rsf > image.rsf u sfdottest sfmodeling vel=vel.rsf \ mod=image0.rsf dat=data.rsf sfdottest: L[m]*d = sfdottest: L'[d]*m=

17 sfdottest: fork/exec random numbers random numbers sfmodeling adj=n sfmodeling adj=y

18 Data-Parallel Processing u sfomp sfmodeling vel=vel.rsf \ split=3 < image.rsf > data.rsf u mprirun np 128 sfmpi split=3 \ sfmodeling vel=vel.rsf \ --input=image.rsf --output=data.rsf

19 in1.rsf sfmodeling out1.rsf in2.rsf sfmodeling out2.rsf in3.rsf sfmodeling out3.rsf in.rsf in4.rsf in5.rsf sfmodeling sfmodeling out4.rsf out5.rsf out.rsf in6.rsf sfmodeling out6.rsf in7.rsf sfmodeling out7.rsf in8.rsf sfmodeling out8.rsf

20 Submitting Parallel Jobs u pscons: parallel scons (scons j) Flow( data, image, modeling,split=[3, omp ]) Flow( data, image, modeling,split=[3, mpi ]) Flow( data, image, modeling,split=[3,1000]) u sfbatch: jobs on a shared cluster sfbatch exe= scons NP=100 data.rsf scons BATCH=1 data.rsf

21 What Does This Do? sfbatch exe= sfconjgrad mpirun np 128 sfmpi sfomp split=3 sfmodeling vel=vel.rsf niter=100 mod=image0.rsf --input=data.rsf --output=image.rsf

22 (Sripanich et al., 2016; Sun et al., 2016)

23 Conclusions u Madagascar is an open-source package for multidimensional data analysis and reproducible computational experiments 10 years in public existence u Madagascar objects are multidimensional data arrays u Using Unix-style encapsulation for data-parallel computations

Madagascar Open- Source Software Package: Interface to Reproducible Research

Madagascar Open- Source Software Package: Interface to Reproducible Research Sergey Fomel Jackson School of Geosciences The University of Texas at Austin SIAM Geosciences June 30, 2015 45 Years of Unix