Grid Experiment and Job Management

Transcription

1 Grid Experiment and Job Management Week #6 Basics of Grid and Cloud computing University of Tartu March 20th 2013 Hardi Teder

2 Overview Grid Jobs Simple Jobs Pilot Jobs Workflows Job management tools UI, GUI, Portal, API Basics of Grid and Cloud computing 2/24

3 Grid Experiment Computing exercise you have to calculate in pieces Big data analyses (CMS, ATLAS etc) Simulating environment Weather prediction Movie rendering Basics of Grid and Cloud computing 3/24

4 Grid experiments Running an experiment can spend thousands of CPU years The results may not be relevant when you get the results Sometimes experiment doesn't fit in a HPC centre Then you need Grid Basics of Grid and Cloud computing 4/24

5 Grid experiment stages Grid experiment stages Pre-processing Running the experiment Computing grid jobs Post-processing Rendering 3D movie Preparing input data for rendering jobs and generating job descriptions Sending the the jobs tracking them and collecting the results Glueing the frames to a movie Basics of Grid and Cloud computing 5/24

6 a Grid job Grid job is an independent peace of experiment Submitting Job description and input files Preparing the environment Checking status Resubmit / Cancel / Kill Downloading results Output files Sending big files to storage Basics of Grid and Cloud computing 6/24

7 Grid Job stages ACCEPTING job has reached the site ACCEPTED job submitted but not yet processed PREPARING input files are being retreived PREPARED input files are retreived SUBMITTING interaction with LRMS ongoing INLRMS:Q job is queued by LRMS INLRMS:R job is running INLRMS:S job is suspended INLRMS:E job is Finishing in LRMS INLRMS:O job is in any other LRMS state KILLING job is being cancelled by user request EXECUTED job is completed in LRMS FINISHING output Files are being transferred FINISHED job is Finished FAILED job is Finished with an error KILLED job is cancelled by user request DELETED job is removed due to expiration time Basics of Grid and Cloud computing 7/24

8 Job Descriptions.xrsl Extended Eesource Specification Language.jsdl - Job Submission Description Language.jdl Job Description Language (glite) & (executable="/bin/echo") (arguments="hello ARC!") (jobname="arc-test-job") (stdout="stdout") (stderr="stderr") (gmlog="gmlog") (CPUTime="5") Basics of Grid and Cloud computing 8/24

9 Grid Job management UI GUI Portals / Gateways APIs Basics of Grid and Cloud computing 9/24

10 Regular grid job You can describe your exercise in one grid job description Or generating similar job descriptions for your experiment Running Job 1 Preprocessing Running Job 2 Postprocessing Running Job N Basics of Grid and Cloud computing 10/24

11 Pilot jobs Sometimes you can't: get enough information from the middleware describe your jobs enough Basics of Grid and Cloud computing 11/24

12 Pilot jobs policy From Policy on Grid Pilot Jobs document Before submitting pilot jobs to a Site the VO must have approval from the Grid and from that Site The pilot job must only execute user jobs belonging to registered and authorised members of the VO. The pilot job framework must meet the fine-grained monitoring and control requirements defined in the Grid Security Traceability and Logging policy. The pilot job must respect the result of any local authorisation and/or policy decisions, e.g. blocking the running of the user job. etc Basics of Grid and Cloud computing 12/24

13 Workflows Scientific Workflows are networks of analytical steps that may involve, e.g., database access and querying, data analysis and mining, and many other steps including computationally intensive jobs submitted to high performance clusters and Grids Bertram Ludäscher Basics of Grid and Cloud computing 13/24

14 Grid Job workflows Automation of complex experiments stages need different resources Basics of Grid and Cloud computing 14/24

15 Simple Parallelism Basics of Grid and Cloud computing 15/24

16 Data Parallelism Basics of Grid and Cloud computing 16/24

17 Workflows save time Workflows allow the reuse of software, applications and simulations improve the repeatability of scientific calculations become more efficient Basics of Grid and Cloud computing 17/24

18 EGI workflow tools web-based gateways desktop application-based gateways Basics of Grid and Cloud computing 18/24

19 Command Line tools arcsub arcstat arccat Also workflow management command line tools are available arcget arcsync arcinfo Basics of Grid and Cloud computing 19/24

20 Graphical Grid UI Science gateways for grid users P-grade Portal Migrating desktop ArcGui Client beta Basics of Grid and Cloud computing 20/24

21 APIs and tools The SWIG wrapped ARC Python API More info API.pdf Basics of Grid and Cloud computing 21/24

22 Specific tools for experiments CMS Remote Analysis Builder (CRAB) Basics of Grid and Cloud computing 22/24

23 References More information about ARC: EGI Workflows page EGI SPG documents Basics of Grid and Cloud computing 23/24

24 More information from: Thank you Hardi Teder Basics of Grid and Cloud computing 24/24