Development Environments for HPC: The View from NCSA

Transcription

1 Development Environments for HPC: The View from NCSA Jay Alameda National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign DEHPC 15 San Francisco, CA 18 October 2015

2 Acknowledgements ª Portions of this material are supported by or based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under its Agreement No. HR , the United States Department of Energy under Contract No. DE-FG02-06ER25752, the Blue Waters sustained petascale computing project, which is supported by the National Science Foundation under award number OCI , and the SI2-SSI Productive and Accessible Development Workbench for HPC Applications, which is supported by the National Science Foundation under award number OCI ª The SI2-SSI team is lead by Jay Alameda (NCSA), Greg Watson (IBM), Steven Brandt (LSU), and Allen Malony (U Oregon). Team members and senior personnel include Beth Tibbitts (IBM), Ralph Johnson (U Illinois), Chris Navarro (NCSA), Sameer Shende (U Oregon), Wyatt Spear (U Oregon), Brian Jewett (U Illinois), Galen Arnold (NCSA), and Rui Liu (NCSA) ª Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation 1

3 Outline ª Overview of Eclipse and the Parallel Tools Platform (PTP) ª Motivation for Workbench for High Performance Computing (WHPC) ª Advantages of Eclipse for HPC Software Developers ª Sustainability, Investment, Collaboration: Building Community ª Challenges with DEHPC ª Opportunities (for discussion) 2

4 Eclipse ª Integrated development environment (IDE) ª Edit code, compile, run, debug without leaving Eclipse ª Graphical user interface 3

5 Eclipse 4

6 Eclipse ª Integrated development environment (IDE) ª Edit code, compile, run, debug without leaving Eclipse ª Graphical user interface ª Plug-ins add new functionality to Eclipse ª Support languages: C/C++, Fortran, Python, ª Support version control systems: Git, Subversion, ª Support issue tracking systems: Bugzilla, Jira, ª Support HPC development (?!) 5

7 Parallel Tools Platform (PTP) ª PTP: a set of plug-ins that extend Eclipse to support HPC development ª Write code on your laptop ª Build it on a remote HPC system using its compilers ª Run it on the remote HPC system (qsub) ª Debug it on the remote system (MPI debugger) ª Tune it for performance on the remote system ª all inside Eclipse! 6

8 Coding & Analysis Eclipse Parallel Tools Platform (PTP) Code completion 7

9 Eclipse Parallel Tools Platform (PTP) Coding & Analysis Launching & Monitoring 8

10 Eclipse Parallel Tools Platform (PTP) Coding & Analysis Launching & Monitoring Debugging 9

11 Eclipse Parallel Tools Platform (PTP) Coding & Analysis Launching & Monitoring Performance Tuning Debugging 10

12 Eclipse Parallel Tools Platform (PTP) Coding & Analysis Launching & Monitoring Performance Tuning Debugging 11

13 Motivation for Workbench for High Performance Computing (WHPC) ª Context ª NSF : SI2-SSI: A Productive and Accessible Development Workbench for HPC Applications Using the Eclipse Parallel Tools Platform ª Stable, portable platform for tool development ª Focus on tool functionality, manage rapid evolution of HPC platforms ª Encourage consistent tool look and feel ª Support for HPC application development practices ª Why Parallel Tools Platform? ª High potential to meet needs of a WHPC. ª Target next generation of HPC developers growing up with IDEs (Eclipse, Visual Studio,...) ª Need to cultivate community of users! 12

14 Improvements ª Work within Eclipse release cycle ª Major (API-breaking) improvements with coordinated June release ª Last major release Eclipse 4.5 Mars released June 2015 ª Minor enhancements and bug-fixes with two coordinated service releases in September and February ª Eclipse 4.5 SR1 Released September, ª Foci of improvements ª Improve usability ª Improve productivity 13

15 Consider 2 possible types of users of Eclipse Parallel Tools ª Science code users/modelers ª Need to build science code ª May need to modify science code (and rebuild) ª Software specialists enabling modeling projects ª Lots of software engineering concerns ª Next set of slides address some of those concerns. 14

16 Science code users/modelers ª Some of the challenges ª Complex codes (eg WRF) ª Codes + HPC architectures can be daunting ª Adding user code not always easy ª WRF: from 15

17 Code navigation Navigating Computational Science Codes Code Outline Syntax-aware editing (navigate to program units and declarations) 16

18 Eclipse aiding in a typical code workflow ª May want to add a model output variable ª Eclipse PTP makes it easy to navigate source, make changes ª Drive remote builds on HPC resources ª Make, autotools ª Run can generate a run configuration for particular system, batch environment 17

19 Software Specialists enabling modeling projects ª Need a wider array of software engineering tools ª Source repository ª Issue tracking ª Documentation ª Performance tuning ª Eclipse Parallel Tools can help with many of these concerns 18

20 Source Code Control: Team Features ª Eclipse supports integration with multiple version control systems (VCS) ª CVS, SVN, Git, and others ª Collectively known as Team services ª Many features are common across VCS ª Compare/merge ª History ª Check-in/check-out ª Some differences ª Version numbers ª Branching 19

21 Issue Tracking ª Mylyn Bridge ª Tracks tasks, links to source and bug repositories Connections to Jira, bugzilla, 20

22 Eclipse Documentation ª Eclipse Help System built in and standalone ( 21

23 Performance Tuning: PTP TAU plug-ins ª TAU (Tuning and Analysis Utilities) ª First implementation of External Tools Framework (ETFw) ª Eclipse plug-ins wrap TAU functions, make them available from Eclipse ª Full GUI support for the TAU command line interface ª Performance analysis integrated with development environment 22

24 Sustainability, Investment, Collaboration: Building Community ª Stakeholders/investors ª Funding agencies à innovative capability ª HPC vendors ª contribute to open source foundational software ª basis for proprietary or open-source add-ons ª Supercomputing centers ª user support and training ª xml documents for batch system and local policy integration 23

25 Sustainability, Investment, Collaboration: Building Community II ª Stakeholders/investors continued ª Supercomputing integrating organizations (eg, XSEDE): ª user support ª single sign-on support (?) ª training ª education support ª Tool providers ª xml documents for integration of command line tools ª extra value-add from tool-specific plugins? ª Anyone else? 24

26 Challenges for DEHPC ª Budgetary pressure on supercomputing centers and integrating organizations ª Challenges successfully competing proposals in open calls (eg, NSF SI2). ª Innovation, impact are paramount here ª User community is large, disperse (eg lots of downloads of PTP, hard to connect to user community) 25

27 Opportunities (for discussion) ª Can we build a community to support a DEHPC? ª PIs proposing to funding agencies for innovative additions? ª Vendor contributions to DEHPC? ª Supercomputing centers and integrating organizations support? ª Eg, User support, training, configuration (xml) documents 26

28 Eclipse Parallel Tools Platform 27