Pydron. Semi-Automatic Parallelization for Astronomy Data Processing. Stefan C. Müller Gustavo Alonso Adam Amara André Csillaghy

Size: px

Start display at page:

Download "Pydron. Semi-Automatic Parallelization for Astronomy Data Processing. Stefan C. Müller Gustavo Alonso Adam Amara André Csillaghy"

Cassandra Gaines
6 years ago
Views:

1 Pydron Semi-Automatic Parallelization for Astronomy Data Processing Stefan C. Müller Gustavo Alonso Adam Amara André Csillaghy Pydron 10/8/2014 1

RHESSI Gustavo Alonso Systems Group ETHZ Adam Amara

Csillaghy Institute of 4D Technologies FHNW Stefan

Technologies FHNW Dark Energy Survey Euclid RHESSI:

Lab Solar Orbiter: ESA / AOES Dark Energy Survey: T.

2 RHESSI Gustavo Alonso Systems Group ETHZ Adam Amara Institute of Astronomy ETHZ Solar Orbiter André Csillaghy Institute of 4D Technologies FHNW Stefan C. Müller Systems Group ETHZ Institute of 4D Technologies FHNW Dark Energy Survey Euclid RHESSI: NASA/Goddard Space Flight Center Conceptual Image Lab Solar Orbiter: ESA / AOES Dark Energy Survey: T. Abbott and NOAO/AURA/NSF Euclid: ESA/C. Carreau 10/8/2014 2

3 Overview Astronomy Processing Codes def main(raw_images): calibrated_files = [] for raw_image in raw_images: bias = measure_bias(raw_image) ft= measure_flat(raw_image) Python cal_file = calib(raw_image,bias,ft) calibrated_files.append(cal_file) Semi-Automatic Parallelization Pydron Cloud, Cluster, Multicore Amazon EC2 Pydron 10/8/2014 3

4 Outline Astronomy Data Processing Using Pydron How it works def main(raw_images): calibrated_files = [] for raw_image in raw_images: bias = measure_bias(raw_image) ft= measure_flat(raw_image) Python cal_file = calib(raw_image,bias,ft) calibrated_files.append(cal_file) Pydron 4 Results Amazon EC2 Pydron 10/8/2014 4

5 Why is Astronomy Data Processing Different? Lack of Economies of Scale Pydron 10/8/2014 5

6 # of Users One of a Kind Commercial Systems: Thousands of deployments Millions of users Astronomy Projects: Single Deployment Hundreds of Researchers < 10 that run jobs Innovative Instrument Eperiment... Never seen before.. Kind of data Data volume Data Processing Analysis... Code Reusability is very limited Pydron 10/8/2014 6

7 Code changes constantly Idea Write Code Run Study Results Unproductive Time Astronomers are Developers, not Users Write Paper never run again Pydron 10/8/2014 7

8 Sequential Script Write Code Run Parallel Code Eplain to Developer Write Code Run Pydron Write Code Run Sequential Script + Scaling Time Pydron 10/8/2014 8

9 How is Pydron used? Pydron 10/8/2014 9

10 Sequential Python Code def main(raw_images): calibrated_files = [] for raw_image in raw_images: bias = measure_bias(raw_image) flat = measure_flat(raw_image) cal_file = calibrate(raw_image,bias,flat) calibrated_files.append(cal_file) def measure_bias(raw_image):... def measure_flat(raw_image):... def calibrate(raw_image,bias,bg):... Pydron 10/8/

Look for parallelism in here Minimal code changes required no side-effects

raw_images: bias = measure_bias(raw_image) flat = measure_flat(raw_image)

append(cal_file) @functional def measure_bias(raw_image):.

11 Look for parallelism in here Minimal code changes required no def main(raw_images): calibrated_files = [] for raw_image in raw_images: bias = measure_bias(raw_image) flat = measure_flat(raw_image) cal_file = calibrate(raw_image,bias,flat) def def def calibrate(raw_image,bias,bg):... Pydron 10/8/

12 Just run it Once function is invoked, Pydron will... For the cloud: Start cloud nodes & Start Python processes Transfer code (and libraries) to nodes Schedule tasks Send results back to workstation Shutdown nodes function returns with the result. Pydron 10/8/

13 Machine Learning Random Forest Training Uses scipy native library Multi-Core Cluster Pydron 10/8/

14 How does it work? Pydron 10/8/

Data-flow Graph raw_images @schedule def

raw_images: bias = measure_bias(raw_image) ft =

Value nodes for-each raw_image measure_bias

15 Data-flow Graph def main(raw_images): cal_files = [] for raw_image in raw_images: bias = measure_bias(raw_image) ft = measure_flat(raw_image) cal_file = cal(raw_image,bias,ft) cal_files.append(cal_file) Epressions Task nodes Variables Value nodes for-each raw_image measure_bias measure_flat bias flat cal Static Single Assignment Form Sub-Graphs for compounds cal_file Pydron cal_files 15

16 Python Challenges Data-dependent control-flow No static type information Invoked functions unknown Decorators unknown Adapt Graph with Runtime Information Change graph depending on data Use dynamic type information Detect function decorators Dynamically adapt granularity Pydron 16

17 Data dependent Control-Flow while f(): = g() s += h() Pydron 17

18 Data dependent Control-Flow while f(): = g() s += h() translate s f while-task f g h s + s while s s Pydron 18

19 Data dependent Control-Flow while f(): = g() s += h() translate s f f g while h s + s s s Pydron 19

20 Data dependent Control-Flow while f(): = g() s += h() translate & replace while-task (3) g h g h g h f while s s + s + s + s Pydron 20

21 Data dependent Control-Flow while f(): = g() s += h() translate & replace while-task (3) g h g h g h f while s s + s + s + s Pydron 21

22 Data dependent Control-Flow while f(): = g() s += h() translate & replace while-task (3) g h g h g h f while s s + s + s + s Pydron 22

23 Results Pydron 23

24 Eo-Planet Detection Eecution Time Overhead Pydron 10/8/

25 Future Optimizations Scheduling algorithm Data specific optimizations Pre-fetching Dynamic Resource Allocation Pydron 10/8/

26 Conclusions Pydron lowers the boundary to use parallel infrastructure Non-intrusiveness & low learning curve over ideal performance Pydron 10/8/

27 Q & A Pydron 10/8/

In astroinformatics, the processing and analyzing. Scaling Astroinformatics: Python + Automatic Parallelization COVER FEATURE

In astroinformatics, the processing and analyzing. Scaling Astroinformatics: Python + Automatic Parallelization COVER FEATURE Scaling Astroinormatics: Python + Automatic Parallelization Stean C. Müller, ETH Zürich and University o Applied Sciences Northwestern Switzerland Gustavo Alonso, ETH Zürich André Csillaghy, University