Running Applications On The Grid. Jon Bednasz & Steve Gallo Center for Computational Research University at Buffalo

Transcription

1 Running Applications On The Grid Jon Bednasz & Steve Gallo Center for Computational Research University at Buffalo

2 Applications Autodock/AutoGrid 4 Suite of automated docking tools designed to predict how small molecules, such as substrates or drug candidates, bind to a receptor of known 3D structures Autodock/AutoGrid 4 is Free Software

3 Applications Who? Dr. Barbara Poliks and Graduate Student Colby Chiauzzi Research Associate Professor Dept of Physics, Applied Physics and Astronomy Binghamton University Why? Autodock/Autogrid was used to determine how the 6C-inhibitor would position itself in each of the five binding sites of pentameric Lumazine Synthase.

4 Flow Chart - Gather Gather files and information Assume we have a valid DOE certificate What machine do we want to run on? Push files to grid machine Need correct autodock/autogrid executables Need input files Discover VO's directory on grid machine run_directory=`globus-job-run ${host} /bin/sh -c 'echo $OSG_DATA'` Globus-job-submit Pull results back

5 Flow Chart - Push Create working directories within $OSG_DATA Gather files and information globus-job-run ${host} -dir ${run_directory} /bin/sh -c "mkdir binghamton" globus-job-run ${host} -dir ${run_directory} /bin/sh -c "mkdir binghamton/${experiment}" Stage executables to grid machine Push files to grid machine globus-url-copy file://`pwd`/${experiment}/autodock4 gsiftp://${host}/${run_directory}/binghamton/${experiment}/autodock4 globus-url-copy file://`pwd`/${experiment}/autogrid4 gsiftp://${host}/${run_directory}/binghamton/${experiment}/autogrid4 globus-url-copy file://`pwd`/${experiment}/grid.sh gsiftp://${host}/${run_directory}/binghamton/${experiment}/grid.sh Change perms on the files Globus-job-submit globus-job-run ${host} -dir ${run_directory}/binghamton/${experiment} /bin/sh -c "chmod u+x autodock4" globus-job-run ${host} -dir ${run_directory}/binghamton/${experiment} /bin/sh -c "chmod u+x autogrid4" globus-job-run ${host} -dir ${run_directory}/binghamton/${experiment} /bin/sh -c "chmod u+x grid.sh" Stage the input files to grid machine Pull results back globus-url-copy file://`pwd`/${experiment}/${input_file1} gsiftp://${host}/${run_directory}/binghamton/${experiment}/${input_file1} globus-url-copy file://`pwd`/${experiment}/${input_file2} gsiftp://${host}/${run_directory}/binghamton/${experiment}/${input_file2}

6 Flow Chart Submit Gather files and information Globus Command Push files to grid machine globus-job-submit ${host}/jobmanager-pbs -dir ${run_directory}/binghamton/${experiment}/ -maxtime 4300 ${run_directory}/binghamton/${experiment}/grid.sh What is grid.sh Globus-job-submit cd $OSG_DATA/binghamton/3_sa0GTDPColc4/./autogrid4 -p./sa0gtdp4.gpf -l sa0gtdp4.glg &&./autodock4 -p sa0gtdp4.dpf -l sa0gtdp4.dlg Pull results back

7 Flow Chart - Pull Gather files and information Push files to grid machine globus-url-copy gsiftp://${host}/${run_directory}/ binghamton/${experiment}/ file://`pwd`/${experiment}/ Globus-job-submit Pull results back

8 Lesson's Learned Write a wrapper script run_grid_job.sh Checks inputs Helps to keep experiments/files organized Use globus-job-status Find someone, like Barbara and Colby, who are in need and highly motivated

9 Results Figure 2.4 Distances between the 6C-inhibitor ( 31 P) and selected 15 N atoms of Lumazine Synathase. The NZ of lysine residue is located ~7.6Å away from the phosphorus. Three best docked inhibitors from AutoDock runs are shown. The green ribbon is traced through the backbone of the loop Ile 91-Lys 92-Gly 93-Ser 94-Thr 95-Met 96-His 97-Phe 98 in the vicinity of the phosphonate group of the inhibitor.

10 Lumazine Synthase Results

11 Commercial Applications COBALT CFD Computational Fluid Dynamics Licensed application License server resides at a remote location Source code not available

12 Commercial Challenges Source code typically not available License issues Node-locked licensing model Remote license servers Many use MPI Applications sensitive to various MPI versions Grid job managers handle MPI differently than traditional command line submissions

13 Basic Grid Job Workflow Identify required files and gather in one location Stage data and executables to grid resource Submit job Query resource for job completion Retrieve results Clean up

14 Running: Gather Your Files Gather your files into a single directory Application files Configuration files Data files Why a single directory? Easier to manage a single directory Staging Easier to retrieve results Simpler cleanup

15 Running: Stage To Resource Select a working directory for your job ($OSG_DATA) globus-job-run localhost /bin/env fgrep OSG_DATA OSG_DATA=/san/scratch/grid/grid-tmp/grid-data Stage files to the resource Use globus-url-copy file:// specifies a local file or directory gsiftp:// specifies a GSI FTP server Gotchyas Local paths need 3 slashes ( file:///path/to/file ) Directories must end with a slash ( gsiftp://path/to/dir/ ) Data on the resource will be owned by the VO user and accessible to other VO users Must use fully qualified paths File permissions are NOT maintained on copy!

16 Running: Submit Job Submit using globus-job-submit Use the correct job manager (pbs, sge, etc.) Returns a job id URL Stores standard output and standard error in your Globus cache Use this URL to identify job for checking status and cleaning up Don t use globus-job-run since it will wait for the job to complete #> globus-job-submit u2-grid.ccr.buffalo.edu/jobmanager-pbs -np 1 -maxtime 5 \ -dir /san/scratch/grid/grid-tmp/grid-data/nysgrid/gallo-1 \ aracne -i arraydata10x336.exp -j arraydata10x336.adj -o arraydata10x336.out

17 Running: Get Job Status Query job status using globus-job-status Use URL from job submission Don t query status constantly, this loads the gatekeeper Intermediate status is up to the application Valid job states: UNSUBMITTED, PENDING, ACTIVE, DONE, FAILED #> globus-job-status PENDING #> globus-job-status DONE

18 Running: Retrieve Results Retrieve results using globus-url-copy Can retrieve entire directory or specific files #> globus-url-copy -v -cd \ gsiftp://u2-grid.ccr.buffalo.edu/san/scratch/grid/grid-tmp/grid-data/nysgrid/gallo-1/ \ file:///san/user/smgallo/u2/projects/nysgrid/gallo-1-results/ Source: gsiftp://u2-grid.ccr.buffalo.edu/san/scratch/grid/grid-tmp/grid-data/nysgrid/gallo- 1/ Dest: file:///san/user/smgallo/u2/projects/nysgrid/gallo-1-results/ aracne arraydata10x336.adj arraydata10x336.exp arraydata10x336.out

19 Running: Cleanup Remove cached job output and any job files from resource Remove job output globus-job-clean Manually remove job working directory Don t use wildcards for filenames! You risk deleting everything if there is an error in your script. #> globus-job-clean force Cleanup successful. #> globus-job-run u2-grid.ccr.buffalo.edu \ /bin/sh c cd $OSG_DATA/GRASE; rm -rf gallo-1

20 COBALT: Specific Challenges Licensed product MPI FORTRAN application Single submission script creates input files and runs MPI task launcher

21 COBALT: Licensing Licensed to Syracuse University for 16 nodes Needed permission from the vendor to run remotely Ensure there were no license domain restrictions Firewall rule modifications on license server Compute nodes require outbound internet connectivity

22 COBALT: Workflow Three files used to run a job case.job Create an input file for Cobalt and runs the MPI wrapper. User modifies this file. CoMPIRUN Cobalt MPI wrapper installed with application COBALT executable case.job Create input file Set Environment Run MPI Wrapper License Server CoMPIRUN Set license Server info Set up working directory Prepare input files for COBALT Execute COBALT using MPI task launcher Clean Up

23 COBALT: Modifications Modifying COBALT for the Grid Create a Job Package to send to grid host Modify COBALT scripts MPI handled by grid job manager Cannot simply send your submission script Job submission and status polling Results retrieval and cleanup

24 Traditional MPI #> qsub case.job Generate configuration files Copy/generate input files case.job Resource manager allocates nodes mpirun cobalt.linux.dp Node 1 CoMPIRUN Node 2 Node n Cleanup

25 Grid MPI #> globus-job-submit jobmanager-pbs -x &(jobtype=mpi) cobalt.linux.dp Resource manager allocates nodes mpirun cobalt.linux.dp Grid Job Manager Node 1 Node 2 Node n

26 COBALT: Modified Workflow create_job_package Create a job package including data files. Much of the work of the CoMPIRUN script is now done here. submit_job Grid submission wrapper create_job_package Create input file Set up working directory Prepare data files for COBALT Job Package submit_job Stage Job Package Execute COBALT using grid job manager Poll for Status Retrieve Results Clean Up License Server

27 Applications Amber Amber is used for simulation of biomolecules

28 Applications Who? Dr. Barbara Poliks Research Associate Professor Dept of Physics, Applied Physics and Astronomy Binghamton University Why? Needed to run long term molecular dynamics (up to 20ns) on large systems of protein + water (Lumazine Synthase + water and tubulin + water both simulated systems had more then 100,000 atoms) 1ns dynamics took about 3.5 hours CPU with parallel processing using 32 nodes.

29 Flow Chart - Gather Gather files and information Assume we have a valid DOE certificate What machine do we want to run on? Push files to grid machine Need correct amber path Need input files Discover VO's directory on grid machine run_directory=`globus-job-run ${host} /bin/sh -c 'echo $OSG_DATA'` Globus-job-submit Pull results back

30 Flow Chart - Push Gather files and information Create working directories within $OSG_DATA globus-job-run ${host} -dir ${run_directory} /bin/sh -c "mkdir binghamton" Push files to grid machine globus-job-run ${host} -dir ${run_directory} /bin/sh -c "mkdir binghamton/${experiment}" Stage inputs to grid machine globus-url-copy file://`pwd`/${experiment}/dyn.in gsiftp://${host}/${run_directory}/binghamton/${experiment}/dyn.in Globus-job-submit globus-url-copy file://`pwd`/${experiment}/sa0_ggafdyn1.rst gsiftp://${host}/${run_directory}/binghamton/${experiment}/sa0_ggafdyn1.rst globus-url-copy file://`pwd`/${experiment}/sa0_gg.prmtop gsiftp://${host}/${run_directory}/binghamton/${experiment}/sa0_gg.prmtop Pull results back

31 Flow Chart Submit Gather files and information Globus Command Push files to grid machine globus-job-submit ${host}/jobmanager-pbs -dir ${run_directory}/binghamton/${experiment}/ -maxtime np 4 -x '(jobtype=mpi)' /util/amber/amber9/exe/pmemd -O -i dyn.in -o sa0_ggafdyn1.out -p sa0_gg.prmtop -c sa0_ggafdyn1.rst -r sa0_ggafdyn1.rst -x sa0_ggafdyn1.mdcrd Globus-job-submit Pull results back

32 Flow Chart - Pull Gather files and information Push files to grid machine globus-url-copy gsiftp://${host}/${run_directory} /binghamton/${experiment}/ file://`pwd`/${experiment}/ Globus-job-submit Pull results back

33 Lesson's Learned Mpi jobs are significantly harder Needed to change pbs.pm Define $mpirun Change $remote_shell (rsh from ssh) Execute submitted script directly Find someone, like Barbara and Colby, who are in need and highly motivated

34 Results - 1 Paper 1 (accepted to Biochemistry October ): 15N{31P} REDOR NMR Studies of the Binding of Phosphonate Reaction Intermediate Analogues to Saccharomyces cerevisiae Lumazine Synthase Tsyr-Yan Yu, Robert D. O'Connor, Astrid C. Sivertsen, Colby Chiauzzi, Barbara Poliks, Markus Fischer, Adelbert Bacher, Ilka Haase, Mark Cushman, * and Jacob Schaefer * Department of Chemistry, Washington University, Saint Louis, Missouri 63130, Department of Physics,State University of New York at Binghamton, Binghamton, New York 13902, Universität Hamburg,Institute of Food Chemistry, Grindelallee 117, D Hamburg, Germany,Lehrstuhl für Organische Chemie und Biochemie, Technische Universität München, D Garching, Germany,and Department of Medicinal Chemistry and Molecular Pharmacology, School of Pharmacy, Purdue University,West Lafayette, IN ABSTRACT : Lumazine synthase catalyzes the reaction of 5-amino-6-D-ribitylamino-2,4(1 H,3 H)- pyrimidinedione(1) with (S)-3,4-dihydroxybutanone 4-phosphate (2) to afford 6,7-dimethyl-8-D-ribityllumazine(3), the immediate biosynthetic precursor of riboflavin. The overall reaction implies a series of intermediates that are incompletely understood. The 15N{31P} REDOR NMR spectra of three metabolically stable phosphonate reaction intermediate analogues complexed to Saccharomyces cerevisiae lumazine synthase have been obtained at 7 and 12 T. Distances from the phosphorus atoms of the ligands to the side chain nitrogens of Lys92, His97, Arg136, and His148 have been determined. These distances were used in combination with the X-ray crystal coordinates of one of the intermediate analogues complexed with the enzyme in a series of distance-restrained molecular dynamics simulations. The resulting models indicate mobility of the Lys92 side chain, which could facilitate the exchange of inorganic phosphate eliminated from the substrate in one reaction, with the organic phosphatecontaining substrate necessary for the next reaction.

35 Results - 2 Paper 2 - to be submitted to Biochemistry Dec. 2008: Dissecting the Paclitaxel-Microtubule Association: Quantitative Assessment of the 2 -OH Group Shubhada Sharma, Chandraiah Lagisetti, Colby Chiauzzi, Barbara Poliks, Robert M. Coates, and Susan Bane*, Department of Chemistry, Binghamton University, State University of New York, Binghamton, New York 13902, Department of Chemistry, University of Illinois, Urbana-Champaign, Illinois 61801, Department of Physics, Binghamton University, State University of New York, Binghamton, New York ABSTRACT: Molecular interactions of paclitaxel and colchicine with tubulin have been studied in detail. The quantitative contribution of 2 -hydroxyl group and the role of N-benzoyl group in the association of paclitaxel with microtubules have been determined by studying the effect of selected paclitaxel derivatives on tubulin assembly. The affinities of these taxanes for microtubules and their cytotoxicities were also measured. The 2 -hydroxyl group was found to contribute a quarter of the total free energy change and more than three quarters of free change attributed to the C-13 side chain for the association of paclitaxel with microtubules. Molecular modeling analysis suggests that the 2 - hydroxyl group forms a hydrogen bond with the D26 of β-tubulin. The N-benzoyl group therefore may serve as an anchor for appropriate conformation of C-13 side chain, facilitating the formation of the hydrogen binding interaction between the 2 -hydroxyl group and protein. These findings help define the structural requirements for high affinity binding to microtubules and are consequently critical towards the development of structurally simpler paclitaxel analogs.

36 Miscellany Standard grid job managers allow a single MPI task launcher to be specified. Many applications require a specific flavour or version of MPI Not all task launchers compatible TODO Rewrite task launcher to accept generic scripts Create a jobmanager-pbs-mpi that allows any installed MPI task launcher to be written Allow users to submit their standard MPI submission scripts

37 Conclusion Questions?