The Prickly Pear Archive: A Portable Hypermedia for Scholarly Publication

Transcription

1 The Prickly Pear Archive: A Portable Hypermedia for Scholarly Publication Dennis Castleberry, Steven Brandt, Frank Löeffler, and Hari Krishnan July 17, 2012

2 1 Problem Specification Traditional Paper System Executable-Based Paper System 2 Hypotheses 3 Design Considerations for an Executable Paper System 4 Case Analysis: The Prickly Pear Archive Evaluation for our Executable Paper System 5 Final Note

3 Problem Specification Traditional Paper System Traditional Paper System The current workflow in the life cycle of a traditional paper may be summarized in the following five steps (Gavish et al.): Store a private copy of the original data to be processed on the local machine Write a script or computer program, with hard-coded tuning parameters, to load the data from the local file, analyze it, and output selected results (a few graphical figures, tables, etc.) to the screen or to local files Withhold the source code that was executed, and the copy of the original data that was used, and keep them in the local file system in a directory called e.g. code-final Copy and paste the results into a publication manuscript containing a textual description of the computational process that presumably took place Submit the manuscript for publication as a package containing the word processor file and the graphical figure files

4 Problem Specification Traditional Paper System Traditional Paper System: Issues There are three issues with this workflow: Communication of the computational methodology used to arrive at the results is based on a non-interactive textual description which is prone to ambiguity Reproducibility of the computational method is difficult without the data and the code; thus, although the result is published, the science is not easy to build upon As a side-effect of the inability to reproduce results, validation of the method is difficult

5 Problem Specification Executable-Based Paper System Executable-Based Paper System Now imagine the following scenario: You have been selected to review a paper on a binary black hole collision. However, instead of having access only to a dense (and exhausting!) textual description of the computational method, you are given easy access to a copy of the paper author s machine. Now, you are free to examine the relevant code, and run the simulation yourself to be sure that the code does what it is supposed to do. You manipulate the parameters to the code (such as the black hole radius) to experiment with it, seeing the visually stunning 3D animations of the simulation for yourself. Now that you clearly understand how the code works and have seen it run, you can give the paper a great review. And, it was fun and exciting to read! ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

6 Problem Specification Executable-Based Paper System Executable-Based Paper System Now imagine the following scenario: You have been selected to review a paper on a binary black hole collision. However, instead of having access only to a dense (and exhausting!) textual description of the computational method, you are given easy access to a copy of the paper author s machine. Now, you are free to examine the relevant code, and run the simulation yourself to be sure that the code does what it is supposed to do. You manipulate the parameters to the code (such as the black hole radius) to experiment with it, seeing the visually stunning 3D animations of the simulation for yourself. Now that you clearly understand how the code works and have seen it run, you can give the paper a great review. And, it was fun and exciting to read! Now imagine, in addition to this, that when you manipulate the parameters to the simulation and re-run it, the paper copy automatically mutates to fit the new results. ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

7 Problem Specification Executable-Based Paper System Executable-Based Paper System Now imagine the following scenario: You have been selected to review a paper on a binary black hole collision. However, instead of having access only to a dense (and exhausting!) textual description of the computational method, you are given easy access to a copy of the paper author s machine. Now, you are free to examine the relevant code, and run the simulation yourself to be sure that the code does what it is supposed to do. You manipulate the parameters to the code (such as the black hole radius) to experiment with it, seeing the visually stunning 3D animations of the simulation for yourself. Now that you clearly understand how the code works and have seen it run, you can give the paper a great review. And, it was fun and exciting to read! Now imagine, in addition to this, that when you manipulate the parameters to the simulation and re-run it, the paper copy automatically mutates to fit the new results. This is an executable paper. ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

8 Problem Specification Executable-Based Paper System Executable-Based Paper System The three essential components of an executable paper are: Data. Includes the data analyzed and generated Code. This includes not only the computational scientific code, but also the code used to generate the media Media. Includes figures, graphs, charts, tables, animations, and any other visual aids to describe the results. Text. Details of the system design is now handled by documentation within the code; thus the text concentrates on the scientific problem and interpretation of the computational results.

9 Hypotheses Hypotheses We hypothesize that the use of an executable paper system: decreases time-to-publication of a computational result increases reviewer-reported verifiability of the results increases reader-reported reproducibility of the results and usability of the publication increases reader-reported overall clarity of the publication

10 Design Considerations for an Executable Paper System Design Considerations In order for a paper to be portable across architectures, the executable paper system must in some way preserve the total coding and execution environment used by the original author For standardization in scientific subcommunities, it is desirable to have a computational framework as the system s centerpiece; or a framework-centric executable archive. Decreases time-to-solution by allowing for a minimum of code development necessary to achieve a computational result Enhances portability, as only a small amount of code needs to be copied from one location to another per paper Each paper added to the archive increases the functionality of the framework and, therefore, the archive If it is a specialized archive, it should be integrated with (numerical) analysis tools used to generate the computational results in such a way that the parameters in the paper may be manipulated and run through the tools

11 The Prickly Pear Archive: Basis Our group wished to build an executable archive centered around Cactus, a computational framework for general problem-solving on regular meshes Though Cactus is cross-platform, we wanted to encapsulate the execution environment, thus we chose to install the system inside of a virtual machine Since Cactus has an automated means to manage simulations (called Simfactory), the remaining work was to segue it with an interface in which a user could manipulate simulation parameters, graphed output, and the equations from which the code is derived

12 The Prickly Pear Archive: Components Kranc is a program that takes as input Mathematica-style structures which represent equations (most notably PDEs) and generates Cactus code which numerically solves them; Kranc is (to be) included in the archive to allow users to enter equations L A TEX is used as the markup language for writing the papers HDF5 is used for data storage of large output files Piraha, a parsing expression grammar, is used for parsing the L A TEX and parameter files

13 The Prickly Pear Archive: How it Works The user is greeted with a web interface to upload parameter files and initial L A TEX code The L A TEX code contains references to parameters, plots, and equations. The Piraha parser runs through the L A TEX file and: Replaces parameter references with values Replaces plot references with plot include statements Replaces Kranc equation references with L A TEX typeset equations Interactive plots are produced and included as figures in the paper Kranc generates code with the equations, which is compiled and run with the parameters given in the parameter file ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

14 The Prickly Pear Archive: How it Works The user is greeted with a web interface to upload parameter files and initial L A TEX code The L A TEX code contains references to parameters, plots, and equations. The Piraha parser runs through the L A TEX file and: Replaces parameter references with values Replaces plot references with plot include statements Replaces Kranc equation references with L A TEX typeset equations Interactive plots are produced and included as figures in the paper Kranc generates code with the equations, which is compiled and run with the parameters given in the parameter file The job is submitted to an XSEDE resource, for example ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

15 Workflow of the Prickly Pear Archive PHP interface XSEDE Kranc Cactus Simfactory LaTeX

16 The Prickly Pear Archive: Paper Results In addition to a modified L A TEX file, an HTML rendering is produced with parameters, plot references, and equation references in editable form fields This allows the user to edit the paper itself and re-compile it, producing a new paper The cycle may continue indefinitely

17 The Prickly Pear Archive: Portability and Reproducibility Reproducibility. Since the virtual machine used to house the archive is large, scripts have been written which build the archive (the Prickly Pear OS and Cactus) from scratch This provides provenance not only for the paper, but for the total execution environment An archive can easily be custom-built to include extra packages required for a paper to run Portability. Since archive copies contain a standard base system, papers are easily able to be ported to another execution environment

18 The Prickly Pear Archive: What Has Been Implemented Feature Interface Parameter Editing Plot/Graph Editing Equation Editing HTML Transcription Reproduction Scripts How Much Complete Mostly Mostly Somewhat Little-IP Little-IP Mostly

19

20

21

22

23

24

25

26

27

28

29

30 Case Analysis: The Prickly Pear Archive Dennis Castleberry, Steven Brandt, Frank Lo effler, The Prickly and Hari PearKrishnan Archive: A Portable Hypermedia for Scholarly July 17, Publication 2012

31 Potential Issues with Executable Archives, Generally Security. The addition of executable code allows for malicious code to be injected into edits to the paper! Authenticate users; accounts should be tied to official contact information ( address, phone number) and approved upon setup House the archive inside of a VM and make regular backups so that, in case lethal code runs accidentally, the archive may be recovered Time to Handle Requests. If the simulation codes are large and time-consuming, then running them with slightly different parameters is infeasible Heroic runs will be archived so that, in the future, when now-large and time-consuming runs become feasible, they will be readily available for experimentation

32 Potential Issues with The Prickly Pear Domain Support. Cactus is a general-purpose framework for numerical problem-solving on regular meshes, and is therefore unsuitable for e.g. computational chemistry Cactus is a growing project which aims to broaden its support for subdomains of computational science The archive aims to be modular enough to adapt to another framework (and job submission system, etc.) Learning Curve. The archive itself is visual, intuitive, and easy to use Cactus essentials can be learned in a day No need to code for AMR; simply write blocks and allow Cactus to handle the parallelism and AMR

33 The Prickly Pear Archive: Future Directions Potentially, the output to the paper including text could be conditioned on the computational results The inclusion of if- and while- constructs in the PPA language could allow for arbitrarily complex I/O The result would be a report that changes fundamentally based on what input is given

34 Evaluation of our Executable Paper System Since our executable paper system is in its infancy, our hypotheses involving human reports are unable to be directly tested ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

35 Evaluation of our Executable Paper System Since our executable paper system is in its infancy, our hypotheses involving human reports are unable to be directly tested But you can help! By downloading the prototype archive, testing it features yourself and providing us with feedback, you can help the archive mature beyond this preliminary stage ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan

36 Final Note You may download the source code for the archive s web component from our Subversion repository at Also, you may download the virtual machine containing the archive at dcastl2/ppa/ Replication scripts available soon!

37 Final Note You may download the source code for the archive s web component from our Subversion repository at Also, you may download the virtual machine containing the archive at dcastl2/ppa/ Replication scripts available soon! Thank you! ennis Castleberry, Steven Brandt, Frank Löeffler, The Prickly and Hari PearKrishnan