Experience in Using a Process Language to Define Scientific Workflow and Generate Dataset Provenance

Transcription

1 Experience in Using a Process Language to Define Scientific Workflow and Generate Dataset Provenance Leon J. Osterweil 1, Lori A. Clarke 1, Aaron Ellison 2, Rodion Podorozhny 3, Alexander Wise 1, Emery Boose 2, and Julian Hadley 2 1 Lab. for Advanced SW Engineering Research (LASER), Dept of Computer Science Univ. of Massachusetts Amherst, MA Harvard Forest, Harvard University, Petersham, MA 3 Dept. of Computer Science, Texas State Univ., San Marcos, TX

2 Or: Why software engineers should be interested in scientific data and how it is processed

3 Or: Why software engineers should be interested in scientific data and how it is processed Scientific data management raises problems in: Configuration Management Abstraction/modularity Exception management Concurrency control Static Analysis --and more

4 Or: Why software engineers should be interested in scientific data and how it is processed Scientific data management raises problems in: Configuration Management Abstraction/modularity Exception management Concurrency control Static Analysis --and more Addressing these Problems can shed light on fundamental issues in software engineering

5 Our approach Evaluate applying the principles of o Modern programming languages o Software engineering o Other CS domains In the form of the Analytic Web concept o Two interrelated types of graphs To see how this helps address the problems of scientists o And what it tells us about the CS domains

6 Our approach Evaluate applying the principles of o Modern programming languages o Software engineering o Other CS domains In the form of the Analytic Web concept o Two interrelated types of graphs To see how this helps address the problems of scientists o And what it tells us about the CS domains

7 You re not a real scientist, you re a Computer Scientist --A Physicist/Astronomer to Lee

8 Computer Scientists do real science, but it is a science of abstract, non-tangible things. And this is a science that you need our help with Lee, back to Physicist/Astronomer

9 So, What do real scientists do (in the abstract)?: The Traditional Structure of Science--The Scientific Method Hypothesis Experimental Design o Define experimentation process Experimentation o Execute the process to produce data Evaluation o Analyze datasets, producing more datasets Reproduction by others o Reapply processes to compare results and increase confidence in the conclusions

10 Focus on Datasets Datasets o Their production o Their reproduction by others o Their use by consumers E.g., In producing subsequent datasets Dataset Provenance o Document where datasets come from o Who, What, When, Where (Typical Metadata) o How What computer programs What cleansing and interpolation algorithms Process Provenance Metadata

11 Real Datasets: Real Complications Is there really any raw data? o Mostly recorded automatically o Often preprocessed by computers Considerable cleaning of datasets o Some by humans o Some by automation Documentation is usually scant or absent Reproduction is hard or impossible o Processes are complex and hard to document Real threat to the foundations of science o Reproducibility is the bedrock of science

12 Perils in Inadequate Dataset Documentation Some datasets are wrong o Incorrectly cleaned o Inappropriate algorithms applied incorrectly o Incompletely processed Statistical data is often misunderstood/misused o Used under assumptions that are unwarranted Incorrect/incorrectly used data can be the basis for gravely important decisions o Clear cut forests o Release of unsafe pharmaceuticals How to know which datasets to trust?

13 The Internet Aggravates These Problems Scientific datasets are published o In the literature o ON THE WEB (NSF mandate) What might be published? o All the data o Analyzed datasets o With all of the fudge factors? What does get published o Highly processed datasets o With important details omitted THE DETAILS MATTER A GREAT DEAL

14 Key Problems to be Addressed Producers need automation help for o Generation o Distribution o Provenance documentation Consumers need help for o Understanding o Reproduction o Regeneration o Reuse These are primarily process issues

15 A Real Ecological Research Problem: The Water Budget Problem Measure and analyze the flow of water through a small forested watershed ds = P - ET- Q Precipitation (P) o Two gauges, P1, P2 used to minimize gaps in data. Surface Discharge (Q) o Measured at a stream gauge o Gaps caused by sensor failure, ice build-up, etc. o Fill as a function of preceding P and Q. Evapotranspiration (ET) o measured at an eddy-flux tower. Photosynthetically active radiation (PAR) o PAR can be used to estimate ET

16 An Eddy Covariance (Flux) Tower Flux source area: = f(canopy structure, wind speed, wind direction, turbulence) CO 2 photosynthesis CO 2 soil respiration CO 2 CO 2 soil respiration Eddy covariance tower

17 Key Features of the Water Budget Process Frequent measurements taken in real-time o Some wirelessly o Considerable manipulation in real-time Datasets published in real-time(?) 30-day retrospective revision of dataset o Maybe revisions at other (more immediate?) intervals Other revisions done at other times o Adjustments for sensor drift o Alternative model evaluation o Use of data from other sites (?) Different scientists want to do different things????

18 DFG of Water Budget Process

19 Some key issues to be dealt with Processes are complex o Intertwined iterations o Exceptions (not shown) interrupt nominal flow o Concurrency Create profusions of artifact instances o How to keep track of multiple instances of a single dataset type? o Need an artifact structure o And configuration management Still other issues o Mixed human and automated agents

20 DFG does not do well with some of these issues Producers generate datasets by executing paths o DFG has some spurious unexecutable paths Consumers need provenance documentation o How datasets were generated o How data items were created o But DFG nodes are types. There are no instances Both need reproducibility o Sufficient details are needed o And can be hard to produce Both need analyses o Which requires rigor and precision o As well as details

21 Our Proposed Approach: Analytic Web Represent scientific datasets as a web structure specifying: o What data were used to derive each new dataset o The process by which each dataset was derived Use two complementary graphs o Process Definition Graph (PDG) (a visual program) Defines process in sufficient detail to support Understanding, Execution, Reproducibility, Scrutiny o Data Derivation Graph (DDGs) (program traces) Documents how each dataset instance was actually derived In terms of actual process steps applied and choices made

22 The Two Graphs Process Derivation Graph (PDG) o Defines the process o Types of tasks that are applied to types of datasets Data Derivation Graph (DDG) o Which instances derived by which others o By which activities PDG execution leads to the creation of the DDG Dataset provenance metadata created from DDG nodes

23 The Process Definition Challenges How to represent processes o Accurately,completely,clearly,reproducibly Real processes are large and complex o Even easy ones How to support execution of these processes How to communicate the processes to working scientists How to validate these processes o Scientific scrutiny o Analysis to assure that composed processes are not anomalous

24 The Little-JIL Process Language Executable,visual process language emphasizing o Use of abstraction o Exception management o Concurrency control o Resource specification and late-binding Use it to define PDGs to determine the characteristics and features needed Execute the PDGs to generate DDGs o That define dataset provenance

25 Value of Hierarchy, Scoping, and Abstraction Process definition is a hierarchical decomposition Think of steps as procedure invocations o They define scopes o Copy and restore argument semantics o Creation of scopes Encourages use of abstraction o Eg. process fragment reuse Use of recursion helps define, propagate, exploit context information to elucidate iteration

26 Elaboration of on key step Augment Realtime Dataset X Get RT Readings ~Data not OK Read Precip Read Others Data not OK Clean Data X ~Dataset Review OK Select Values Clean Data Apply Models Select Best Model Fill Gaps in Data Dataset Review OK

27 Makes use of recursion Apply Models Apply A Model Apply Models Get Model Compute Model Values

28 Use of different types of agents too Apply Models Apply A Model Apply Models Agent is scientist Get Model Compute Model Values

29 The whole process

30 Elaborate this corner

31 Need for complex exception handling Read Precip = Handle No Precip Read Precip 1 Read Precip 2 X Choose Source O X Sensor Down X Read Precip 1 Read Precip 2 Get Airport Put Null

32 The whole process

33 howing process odule reuse

34 re process dule reuse

35 The DDG A DAG of instances o Dataset instances o Process step instances (and their agents) Edges indicate which datasets are o Derived from executing which step instance o Using which dataset instances as inputs

36 Different executions lead to different DDGs Read Precip = Handle No Precip Read Precip 1 Read Precip 2 X Choose Source O X Sensor Down X Read Precip 1 Read Precip 2 Get Airport Put Null

37 DDG if both sensors are read successfully Read Precip 1 Read Precip 2 P1 P2 Read Precip P1 P2 Get RT Readings

38 DDG if Read of Precip 1 Fails twice, then gets value from alternative source (airport) Read Precip 1 Get Airport Read Precip 1 Read Precip 2 P1 P2 P1 Retry Airport P1 Read Precip Handle No Precip P2 P1 Get RT Readings

39 DDG showing consideration of two alternative models P1 P2 Model 1 Model 2 Apply A Model Apply A Model P from Model1 P from Model 2 Add to Dataset P

40 Full DDG: Two Models; Both readings succeeded P1 Read Precip 1 Read Precip Read Precip 2 P2 P1 P2 Get RT Readings P1 P2 Apply Models P1 Model 1 Model 2 P2 Apply A Model Apply A Model

41 Read Precip 1 Get Airport Read Precip 1 Read Precip 2 P1 P2 P1 Retry Airport P1 Read Precip Handle No Precip P2 DDG with two failed Precip readings, then airport read P1 P1 P1 Get RT Readings Apply Models Model 1 Model 2 P2 P2 Apply A Model Apply A Model

42 Read Precip 1 Get Airport Read Precip 1 Read Precip 2 P1 P2 P1 Retry Airport P1 Read Precip Handle No Precip P1 P2 Eliding for Clarity (?) Get RT Readings: Apply Models Model 1 Model 2 Apply A Model Apply A Model

43 Read Precip 1 Read Precip 2 A Further DDG Read Precip; Get RT Readings; Apply Models P1 P2 Model 1 Model 2 Apply A Model Apply A Model P from Model1 P from Model 2 Add to Dataset P

44 sensordatum Read PModelChannel at time 1 Read PModelChannel at time 2 Read PModelChannel at time 3 IdofPModel 1 IdofPModel 2 IdofPModel 3 DDG for Selection from among three Models Auto Select Model at time 1 Auto Select Model at time 2 Auto Select Model at time 3 Cached data IdofPModel 1 After Auto Select Model IdofPModel 2 After Auto Select Model IdofPModel 3 After Auto Select Model Eval Selected Model for Model1 Eval Selected Model for Model2 Eval Selected Model for Model3 SensorDatum.original SensorDatum 1 SensorDatum 2 SensorDatum 3 PModelID SensorDatum.modeled Select Best Model

45 Read try1 time Sensor1 try1 time Read try2 time try2 time Read PModelChannel at time 1 Read PModelChannel at time 2 Read PModelChannel at time 3 IdofPModel 1 IdofPModel 2 IdofPModel 3 Get Airport Data streams Auto Select Model at time 1 Auto Select Model at time 2 Auto Select Model at time 3 Airport airport try time Bind Airport Datum As P1 Cached Data IdofPModel 1 After Auto Select Model Eval Selected Model for Model1 IdofPModel 2 After Auto Select Model Eval Selected Model for Model2 IdofPModel 3 After Auto Select Model Eval Selected Model for Model3 P1 after Get Airport SensorDatum 1 SensorDatum 2 SensorDatum 3 Filter Range Data P1, after Check Ranges Check Ranges Read try1 time P2 Read PModelChannel at time 4 Select Values of P SensorDatum.original Select Best Model PModelID SensorDatum.modeled IdofPModel 4 Auto Select Model at time 4 P set from P2 sensordatum Enhanced Cached Data SensorDatum.original The DDG further along Next PModelID Select Best Model Next SensorDatum.modeled IdofPModel 4 After Auto Select Model Eval Selected Model for Model4 SensorDatum 4

46 Process Provenance Metadata Derived from the DDG o Root of the appropriate DDG subdag o Efficiencies by storing one DDG Provenance metadata stored as pointer Tradeoffs possible o Cache some intermediate values o Rederive the rest

47 Some Related Work in Scientific Workflow The Kepler Project o Largest, best funded o Based on Ptolemy II Data Flow Graph Approach o Oodles of tools Taverna, Teuta, Chimera, JOpera o Different Data Flow Graph projects Provenance Projects o Database approaches o Process history logging approaches

48 Problems with Current Scientific Workflow Languages tend to be semantically weak o Based on box and arrow charts Principal semantic weaknesses o Lack of abstraction facilities o Weak exception management o Non-trivial iteration can be hard to understand Other problems (with some) o Lack of focus on data and data flow o Provenance viewed as a separate issue o Poorly defined semantics Restricts the rigor of analyses Kepler is better than most

49 Kepler Large toolset to support users Highly visual Box and arrow charts are defined by Directors, which provide semantics o But different levels of hierarchy may have different Directors o No standard usage of Directors E.g. for loops and exceptions o Hard to tell what a diagram is saying without looking at Its Directors Director of its parent Programming language experience can inform work in this area

50 Important Clarification DFG-based scientific workflow systems can do (most or) all of the above BUT: Process definitions tend to be ungainly, opaque The Turing Tarpit for process definition Focus here: o What language constructs should be used to make these processes clear, evolvable, analyzable, etc. o Software engineering has much to say about this

51 Observations Creating a PDG makes scientists think carefully The PDG helps scientists create flexible and scalable processes The PDG supports re-execution using different input data and/or different tools. The PDG provides a useful and efficient framework for structuring and generating the DDG. PDGs can be rigorously evaluated for logical, statistical and propagation of measurement errors.

52 Summary Defined PDGs for ecological research processes o Carbon Flux o Water Budget Key semantic features needed o Exception management o Abstraction/modularity/reuse o Agent definition Not favorable for use of DFGs as PDGs Little-JIL interpreter executed simple processes only DDGs done only manually o Automation needed o Application to other process areas and problems?

53 Future work SciWalker 2 o Environment for building and maintaining PDGs and DDGs Focus on DDGs o How to build, maintain, optimize, display them o What else are they good for? (Software) rework? Running benchmarks???? Integrate with Kepler o Use Kepler for low-level tasks o Little-JIL for high-level process definition

54 Questions and Discussion?