Batch system usage arm euthen F azo he Z J. B T

Transcription

1 Batch system usage

2 General stuff Computing wikipage: Central address for questions & requests: Data storage: AFS ( /afs/ifh.de/group/amanda/scratch/ your scratch area) Lustre: fast parallel storage, use in batch jobs dcache (acs): mass storage, files on tape 1

3 Resources: Batch farm: ~700 cores, all Intel Xeon GHz, RAM: 2-4 GB/core All nodes run 64-bit SL5 SUN Grid Engine (SGE) 6.2 Shared between all groups (nic, that, amanda, z_nuastr, cta, pitz, atlas) Cluster: 1024 cores, (accessible for theory groups) Other farms: GRIDZN: atlas, lhcb, cta (9%), icecube (6%), etc (you need a grid certificate) NAF (National Analysis Facility: Physics at the Terascale: LHC experiments: cms, atlas, lhcb and ILC 2

4 Using the SGE Batch Farm Commands: WorkingGroupServers only for small jobs. If more computing power is needed, use the farm. qrsh : Interactive access, for heavy ROOT sessions, moving data,... qsub: Batch jobs, for simulation, data processing,... Usage: 1. Split task into small jobs 2. Script them 3. Submit the job scripts (qsub your_farm_script.sh) #$... #$... #$... #$... script to submit ordinary shell script interpreted by batch system 3

5 What every script needs #!/bin/zsh Batch Job #$ -S /bin/zsh shell to be used #$ -l h_cpu=00:29:00 cpu time for this job Comment #$ -l h_vmem=850m maximum memory usage of this job #$ -j y stderr and stdout are merged #$ -m bea send mail on job's begin, end and abort (bea) #$ -o /afs/ifh.de/.../farmmessages redirect output -o #$ -cwd execute job from current directory #$ -l os=sl5 operating system #$ -P amanda project name discouraged, only for testing regularly delete files If directory is full, job will crash obsolete since all systems have SL5 (64bit) amanda has less priority than z_nuastr 4

6 What every script needs... advices Requesting resources CPU time, 3 queues: Short: 30min Medium: 12 hours Long: 48 hours There is no difference for time given inside time range: always give upper limit! (e.g. 29min) If job last longer that requested, it will crash. Time your scripts prior to sending! Use the short queue, it is usually empty! Memory: Less memory requested will give higher priority to your job. If you request less memory than needed, your job will crash. Always test! 5

7 Shell script part Shell script Comment SIG_EVT=$1... Input parameters hostname; date some info you want in the stdout file cd $TMPDIR always $TMPDIR, NOT /tmp! cp.../infile./ fetch input file(s) your_program run the actual job, output to $TMPDIR cp outfile /lustre/... store output file(s) your_program example: ${WORK}/IceRec_v03/build64/env-shell.sh my_analysis $START $END $SIG_EVT $TIME 6

8 Batch commands qsub : submit a job qstat : shows running/waiting jobs qstat -u jlbazo [oreade30] ~ % qstat -u jlbazo job-id prior name user state submit/start at queue slots ja-task-id JobAnalysi jlbazo r 11/08/ :38:32 2d.q@blade5c.ifh.de 1 qstat -ext -u jlbazo (extended information: e.g. project ) qhost : shows status of execution hosts qdel : delete submitted jobs qdel job_id (delete job) qdel -u jlbazo (delete all jobs from user) 7

9 Monitoring the farm activity Useful script from Adam: myjobs.awk (you can copy it from: alias myjobs="qstat -g d awk -f ~/myjobs.awk" Fast look at the farm usage from others and your own jobs 8

10 Monitoring the farm activity Monitoring and ACcounting in the SGE BATch Farm In October In

11 Batch Scheduler Strategy Scheduling order SGE assigns tickets to each job. Job with most tickets is sent first. If requested resources are not available, next job in turn is tried. Project has big influence on scheduling policy. Number of tickets depends on the resources requested (mem, cpu). cpu parameter has a much bigger weight than mem parameter. Request less resources for a faster scheduling (short queue, less mem) When in need, use project z_nuastr, but think about others 11

12 Batch final recommendations Always send a few test jobs first Make sure you have sufficient filesystem quota for all job output Avoid: jobs writing same file too many jobs working in same directory writing too much to stdout/err Usually, transfer data at beginning/end of job only Most of the time, work on the local disk ($TMPDIR) Avoid mass failures, they cause mail storms Most common mistake: failure to request resources 10

13 Resources Farm info: Introduction to DESY-computing: