Prediction of workflow execution time using provenance traces: practical applications in medical data processing

Size: px

Start display at page:

Download "Prediction of workflow execution time using provenance traces: practical applications in medical data processing"

Felix Simmons
5 years ago
Views:

1 Prediction of workflow execution time using provenance traces: practical applications in medical data processing Hugo Hiden Simon Woodman Paul Watson

2 How long will my program take to run?

3 Part of a bigger picture Can I repeat my results? What are the implications of errors How long will my program take to run? What version of the program ran? How was a result generated?

4 Provenance Research Used to answer these questions Important in scientific research Lots of work done to capture and represent provenance Active research area OPM PROV

5 e-science Central Source of all our provenance data Platform used for many projects Repository of code and data Users can add their own code Well instrumented and understood Used to collect OPM Now PROV Plenty of data sets Diverse projects Large applications Workflows for data processing

6 The workflow model Simple workflow implementation Acyclic directed graph Composed of connected Blocks Deploys at reasonable scale in clouds

7 Modelling performance Execution time for a single block Workflow is some combination of individual block models There should be some predictors: The input data sizes The configuration of the block The machine it is running on The issues are: What types of model are most appropriate How accurate are they

8 Execution time of a block time=f(input-size, block-code, block-settings, random-factors) More data increases execution time Each block has different characteristics, so a model is needed for each block The configuration of the block instance can change behavior Machine load, network traffic, hardware variations, A workflow is a connected pathway of blocks

9 Requirements for a real system Proactively build models In response to more data When more blocks are added Select the most appropriate model Pick based on best error Aim to always return some estimate Mechanisms to return estimate if no models are available

10 Complications Gathering data Collect data non-invasively Model types Different blocks display different characteristics Different algorithms and versions Dynamic environment New blocks being added Block behaviour only becomes apparent as data is collected

11 Data collected via provenance Provenance collection already captures: Data sizes Code versions Algorithm settings Extra instrumentation for Block start and end times Number of concurrent workflows CPU / Memory usage

12 e-sc Architecture Tooling - Maven Plugins - File uploader - Domain specific apps/websites Workflow Engines Workflow Queue External API REST User MGMT - Friends - Groups - Projects - Quotas HTTP e-sc DB Postgres MySQL SQL Server Service/Lib Cache Control Topic Security - ACL - Authentication Migration Queue e-sc Blob Store New e-sc Blob Store Filesystem S3 Postgres Azure Blob Store HDFS Private API RMI Storage - Versioning - Archiving Provenance Store OpenID Shibboleth External Auth REST Processing - Services - Workflows - Libs Provenance Queue Postgres Neo4j SWORD Provenance/ Audit - Capture - Query/Search - Presentation Archive Queues Archive Filesystem AWS Glacier

Data capture architecture Provenance and performance data capture Data / model storage Tooling - Maven Plugins - File uploader - Domain specific apps/websites Workflow Engines Workflow Queue External

13 Data capture architecture Provenance and performance data capture Data / model storage Tooling - Maven Plugins - File uploader - Domain specific apps/websites Workflow Engines Workflow Queue External API REST User MGMT - Friends - Groups - Projects - Quotas HTTP Service/Lib Cache Control Topic Security - ACL - Authentication Data Models Private API RMI Storage - Versioning - Archiving OpenID Shibboleth REST Model building / updating External Auth Processing - Services - Workflows - Libs SWORD Provenance/ Audit - Capture - Query/Search - Presentation

14 Data collected Each execution of a block creates a single data point: ID, Version Setting_1, Setting_2, Memory Use, Input_size Duration, Output_size ID, Version Setting_1, Setting_2, Memory Use, Input_size Duration, Output_size ID, Version Setting_1, Setting_2, Memory Use, Input_size Duration, Output_size Identifying data Model X data Model Y data

15 Block models Execution Time No relationship Blocks may exhibit very different behaviors depending on their implementation details / configuration Execution Time Observed Execution Data Linear relationship Observed Execution Data Execution Time Non-linear relationship Observed Execution Data

16 Selecting the most appropriate model

17 Selecting the most appropriate model

18 Selecting the most appropriate model

19 Selecting the most appropriate model

20 Dynamic model updating Impossible (difficult) to know what the best model will be Gathering more data may change our view Need to implement model updating Models can be rebuilt and replaced on the fly Return best available estimate at a given time This may improve

21 Panel of experts pattern Maintain a suite of different models Rebuild them all when new data arrives Use the best one until the next update Drug modelling project: Quantitative Structure Activity Relationship Activity f( )

22 Model fallbacks What happens if there is no model? Still want to return something We used the following logic: Use version agnostic model Use average execution time of block Use average execution time of all blocks This will always return some prediction as long as a single block of any type has executed

Medical data processing Measure acceleration

Clinicians view results and modify exercise

23 Medical data processing Measure acceleration in 3-axes Typically 100Hz Worn for 2 weeks Analyse sleep patterns, general activity levels etc Data collected and analysed Clinicians view results and modify exercise regime Collections of 100k data sets (24TB) Wrist worn accelerometers

24 Results Physical Activity Classification (PAC1) Prediction Fitted Ideal Prediction Fitted Ideal Predicted (KB) Predicted (seconds) Actual (KB) Actual (seconds) Output size model Duration model

25 Results GGIR GENEActiv processing Prediction Fitted Ideal Prediction (RMSE=34.670, r 2 =0.987) Fitted Ideal Predicted (KB) Predicted (seconds) Actual (KB) Actual (seconds) Output size model Duration model

26 Not always successful Prediction Fitted Ideal 60 Predicted (seconds) Actual (seconds)

27 Predicting Workflow duration Modelling is complicated by connected nature of workflow All data for model readily available not the case here? how big are the intermediate data transfers??????

Data volume produced by a block size=f(input-size, block-code, block-settings, random-factors) More data increases execution time Each block has different

28 Data volume produced by a block size=f(input-size, block-code, block-settings, random-factors) More data increases execution time Each block has different characteristics, so a model is needed for each block The configuration of the block instance can change behavior Machine load, network traffic, hardware variations, phase of moon

29 Modelling total execution time Execution time = Sum(block predictions)

30 Models built for each individual block Prediction generated by propagating size predictions Results Chemical property modelling Predicted (seconds) Training Prediction (RMSE=5.008,r 2 =0.980) Testing Prediction (RMSE=4.698,r 2 =0.981) Fitted Training Ideal Actual (seconds)

31 Modelling workflows: caveats Much harder to model workflow duration Propagation of errors Works for simple workflows Rapidly fails for larger workflows Possible solutions More data collection Model groups of blocks Build models of whole workflows

32 Conclusions Extended provenance capture to build predictive models Asynchronous collection of data and model building Demonstrated it is possible to model block execution time Show it may be possible to combine predictions to estimate workflow execution time Large workflows / poor block models are issues

Microsoft Architecting Microsoft Azure Solutions.

Microsoft Architecting Microsoft Azure Solutions. Microsoft 70-535 Architecting Microsoft Azure Solutions https://killexams.com/pass4sure/exam-detail/70-535 QUESTION: 106 Your organization has developed and deployed several Azure App Service Web and API