DAGMan workflow. Kumaran Baskaran

Transcription

1 DAGMan workflow Kumaran Baskaran NMRbox summer workshop June 26-29,2017

2 what is a workflow?

3 Laboratory Publication

4

5 Workflow management

6 Softwares Workflow management

7 Data Softwares Workflow management

8

9 Manual Train

10 Manual Train Automated Train

11 Introduction DAGMan (Directed Acyclic Graph Manager) is a metascheduler for HTCondor Manages dependencies between jobs DAG can be used to represent workflow Nodes (vertices) represent steps and edges (arcs) identify the dependencies. Maintains event logs and generates rescue DAG in case of one or more failed jobs

12 Workflow and Automation Workflow can be automated if it doesn t require any user interaction Goal of the workflow is to capture every step and automation is a special case of it Workflow management is necessary for reproducible computing

13 Workflow Data Step 1 Step 2 Step 3 Result

14 NMR Workflow FID FT Peak picking Peak assign CS list

15 NMR Workflow FID Format conversion FT Format conversion Peak picking Peak assign CS list

16 NMR Workflow FID Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list

17 NMR Workflow NMRpipe NMRFAM-Sparky FID Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list

18 Back bone assignment NMRpipe NMRFAM-Sparky FID CBCACONH Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list NMRpipe NMRFAM-Sparky FID HNCBCA Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list NMRpipe NMRFAM-Sparky FID NHSQC Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list

19 Back bone assignment NMRpipe NMRFAM-Sparky FID CBCACONH Varian to pipe FT Pipe to Sparky Peak picking Peak assign NMRpipe NMRFAM-Sparky FID HNCBCA Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list NMRpipe NMRFAM-Sparky FID NHSQC Varian to pipe FT Pipe to Sparky Peak picking Peak assign

20 Side chain assignment NMRpipe NMRFAM-Sparky FID CCONH Varian to pipe FT Pipe to Sparky Peak picking Peak assign NMRpipe NMRFAM-Sparky FID HBHACONH Varian to pipe FT Pipe to Sparky Peak picking Peak assign CS list NMRpipe NMRFAM-Sparky FID. Varian to pipe FT Pipe to Sparky Peak picking Peak assign

21 NMR Workflow Back bone CS list All atom CS list Side chain CS list MD NMRpipe NMRFAM-Sparky FID NOESY Varian to pipe FT Pipe to Sparky Peak picking Peak assign Distance restraints

22 DAGMan One Node within a DAG [optional] PRE script HRCondor Job(s) [optional] POST script

23

24 Example JOB A A.condor JOB B B.condor JOB C C.condor JOB D D.condor PARENT A CHILD B C PARENT B C CHILD D B A D C

25 Example Key words HTCondor job description files JOB A A.condor JOB B B.condor JOB C C.condor JOB D D.condor PARENT A CHILD B C PARENT B C CHILD D B A D C Job dependencies

26 Complex workflow

27 An error in the workflow

28 Complex workflow

29 Rescue DAG

30 Job done!

31 NMR Example File: NMRWorkflow.dag FID Job Varian2Pipe Varian2Pipe.sub Job FT FT.sub Job Pipe2Sparky Pipe2Sparky.sub Job Sparky Sparky.sub PARENT Varian2Pine CHILD FT PARENT FT CHILD Pine2Sparky PARENT Pine2Sparky CHILD Sparky NMRpipe script to convert Bruker/Varian to pipe NMRpipe script to FT NMRpipe script to convert pipe to UCSF NMRFAM-Sparky

32 File: Varian2Pipe.sub NMR Example universe = local notify_user = kbaskaran@bmrb.wisc.edu notification = Error getenv = True executable = <path>/varian2pipe.com arguments = $(OPTIONAL) log = logs/varian2pipe.log output = logs/varian2pipe.out error = logs/varian2pipe.err copy_to_spool = False priority = 0 queue

33 File:NMRWorkflow.dag NMR Example Job Varian2Pipe(CBCACONH) cbcaconhvarian2pipe.sub Job FT(CBCACONH) cbcaconhft.sub Job Pipe2Sparky(CBCACONH) cbcaconhpipe2sparky.sub Job Varian2Pipe(HNCACB) hncacbvarian2pipe.sub Job FT(HNCACB) hncacbft.sub Job Pipe2Sparky(HNCACB) hncacbpipe2sparky.sub Job Varian2Pipe(NHSQC) nhsqcvarian2pipe.sub Job FT(NHSQC) nhsqcft.sub Job Pipe2Sparky(NHSQC) nhsqcpipe2sparky.sub Job Sparky Sparky.sub VARS Sparky OPTIONAL="/home/nmrbox/kbaskaran/Desktop/ubiquitin/Original_data/CBCACONH.fid/CBCACONH.ucsf / home/nmrbox/kbaskaran/desktop/ubiquitin/original_data/hncacb.fid/hncacb.ucsf /home/nmrbox/kbaskaran/desktop/ ubiquitin/original_data/nhsqc.fid/nhsqc.ucsf" PARENT Varian2Pipe(CBCACONH) CHILD FT(CBCACONH) PARENT FT(CBCACONH) CHILD Pipe2Sparky(CBCACONH) PARENT Varian2Pipe(HNCACB) CHILD FT(HNCACB) PARENT FT(HNCACB) CHILD Pipe2Sparky(HNCACB) PARENT Varian2Pipe(NHSQC) CHILD FT(NHSQC) PARENT FT(NHSQC) CHILD Pipe2Sparky(NHSQC) PARENT Pipe2Sparky(CBCACONH) Pipe2Sparky(HNCACB) Pipe2Sparky(NHSQC) CHILD Sparky

34 File:NMRWorkflow.dag NMR Example DOT dag.dot UPDATE Job Varian2Pipe(CBCACONH) cbcaconhvarian2pipe.sub Job FT(CBCACONH) cbcaconhft.sub Job Pipe2Sparky(CBCACONH) cbcaconhpipe2sparky.sub Job Varian2Pipe(HNCACB) hncacbvarian2pipe.sub Job FT(HNCACB) hncacbft.sub Job Pipe2Sparky(HNCACB) hncacbpipe2sparky.sub Job Varian2Pipe(NHSQC) nhsqcvarian2pipe.sub Job FT(NHSQC) nhsqcft.sub Job Pipe2Sparky(NHSQC) nhsqcpipe2sparky.sub Job Sparky Sparky.sub VARS Sparky OPTIONAL="/home/nmrbox/kbaskaran/Desktop/ubiquitin/Original_data/CBCACONH.fid/CBCACONH.ucsf / home/nmrbox/kbaskaran/desktop/ubiquitin/original_data/hncacb.fid/hncacb.ucsf /home/nmrbox/kbaskaran/desktop/ ubiquitin/original_data/nhsqc.fid/nhsqc.ucsf" PARENT Varian2Pipe(CBCACONH) CHILD FT(CBCACONH) PARENT FT(CBCACONH) CHILD Pipe2Sparky(CBCACONH) PARENT Varian2Pipe(HNCACB) CHILD FT(HNCACB) PARENT FT(HNCACB) CHILD Pipe2Sparky(HNCACB) PARENT Varian2Pipe(NHSQC) CHILD FT(NHSQC) PARENT FT(NHSQC) CHILD Pipe2Sparky(NHSQC) PARENT Pipe2Sparky(CBCACONH) Pipe2Sparky(HNCACB) Pipe2Sparky(NHSQC) CHILD Sparky

35 File:NMRWorkflow.dag NMR Example DOT dag.dot UPDATE Job Varian2Pipe(CBCACONH) cbcaconhvarian2pipe.sub Job FT(CBCACONH) cbcaconhft.sub Job Pipe2Sparky(CBCACONH) cbcaconhpipe2sparky.sub dot -Tps dag.dot -o dag.ps Job Varian2Pipe(HNCACB) hncacbvarian2pipe.sub Job FT(HNCACB) hncacbft.sub will produce dag image Job Pipe2Sparky(HNCACB) hncacbpipe2sparky.sub Job Varian2Pipe(NHSQC) nhsqcvarian2pipe.sub Job FT(NHSQC) nhsqcft.sub Job Pipe2Sparky(NHSQC) nhsqcpipe2sparky.sub Job Sparky Sparky.sub VARS Sparky OPTIONAL="/home/nmrbox/kbaskaran/Desktop/ubiquitin/Original_data/CBCACONH.fid/CBCACONH.ucsf / home/nmrbox/kbaskaran/desktop/ubiquitin/original_data/hncacb.fid/hncacb.ucsf /home/nmrbox/kbaskaran/desktop/ ubiquitin/original_data/nhsqc.fid/nhsqc.ucsf" PARENT Varian2Pipe(CBCACONH) CHILD FT(CBCACONH) PARENT FT(CBCACONH) CHILD Pipe2Sparky(CBCACONH) PARENT Varian2Pipe(HNCACB) CHILD FT(HNCACB) PARENT FT(HNCACB) CHILD Pipe2Sparky(HNCACB) PARENT Varian2Pipe(NHSQC) CHILD FT(NHSQC) PARENT FT(NHSQC) CHILD Pipe2Sparky(NHSQC) PARENT Pipe2Sparky(CBCACONH) Pipe2Sparky(HNCACB) Pipe2Sparky(NHSQC) CHILD Sparky

36 DAGMan commands Submit a dag: condor_submit_dag NMRWorkflow.dag Check the condor jobs : condor_q Delete a job : condor_rm <job id> Generate a DAG image : dot -Tps dag.dot -o dag.ps

37 NMR Example DAG image (all job successful)

38 NMR Example DAG image (some running and one error)

39 NMR Example DAG image (completed with an error)

40 Rescue DAG A rescue DAG is automatically generated whenever one or more jobs fail User may correct the error and submit the rescue DAG using the following command condor_submit_dag -doresucefrom <rescue id> <original DAG file>

41 NMR Example DAG image (Rescue DAG) won't run

42 Advantages of DAGMan Captures every step with extensive logging Easily scaleable from personal condor to high performance cloud based clusters basically it is the same DAG you submit to different condor pool Rescue DAGs are quite useful for debugging Easy to recover from an intermediate stage

43 Disadvantages of DAGMan No GUI, user need to edit the DAG file to make changes Since it is based on acyclic architecture, handling a recursion is not straight forward

44 Current issues There are no well defined steps in structure determination using NMR There are multiple ways to achieve the goal Users may prefer their own choice of softwares tools and protocols It s not easy to define an end point for a particular step Example : In Sparky peak picking is a non trivial process. User may add/delete peaks at anytime. Similarly user may correct an assignment manually anytime

45 Future plan A generic DAG for most common workflow and users may add flavors to it Create a workflow library for different tasks workflow for peak assign workflow for refinement PRE and POST script options could be used to validate the data before and after every job

46 Tutorial Data : 3 time domain data sets available in summer_workshop/dagman/ubiquitin/original_data CBCA(CO)NH HNCACB NHSQC NMRPipe scripts already in the folders Workflow : Condor submit files and a DAG file is available in summer_workshop/dagman/ubiquitin/workflow

47 Tutorial Go to workflow' folder and open the NMRWorkflow.dag in a test editor This contains job definitions and dependencies. Open any one of the.sub file using a text editor This contains condor job descriptions

48 Submitting a DAG Open terminal and go summer_workshop/ DAGMan/ubiquitin/workflow Command tp submit a DAG condor_submit_dag NMRWorkflow.dag

49 Submitting a DAG

50 Checking jobs

51 Generating a DAG image

52 Generating a DAG image

53 Opens Sparky session at the end

54 Rescue dag Rename one of the experimental data folder to create an error Submit the dag and one of the spectra will fail Now you can correct the error and run a rescue DAG to complete only missing jobs

55 Creating an error

56 Submitting DAG with an error to overwrite logs

57 Check the DAG image

58 DAG image Other jobs are running Error

59 Wait for the DAG to complete and generate DAG image

60 DAG completed with an error

61 Rescue file generated

62 Correct the error and run the rescue DAG

63 Correct the error and run the rescue DAG

64 Check the DAG image

65 Done!