Advanced Bash Scripting

Transcription

1 for HPC SysAdmins Piero Calucci Scuola Internazionale Superiore di Studi Avanzati Trieste 2 Democritos Modeling Center for Research in Atomistic Simulation INFM November 2008 Advanced School in High Performance and Grid Computing

2 Master Foo once said to a visiting programmer: «There is more Unix-nature in one line of shell script than there is in ten thousand lines of C.» (... ) «Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?» (... ) «And how many hours would you require to implement and debug that C program?» asked Nubi. «Many,» admitted the visiting programmer. «But only a fool would spend the time to do that when so many more worthy tasks await him.» «And who better understands the Unix-nature?» Master Foo asked. «Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?» Upon hearing this, the programmer was enlightened. from Master Foo and the Ten Thousand Lines

3 Outline 1 File System Users: 2 Find Users on a Node 3 Check for Dead and Restart When Needed

4 Example 1: File System Users Beyond Background: you need to know who is doing what with some file system object (file, directory, whole file system... ) the command can tell the PIDs of involved process(es) you want more: process owner complete command line

5 Beyond Interactive solution Who is using the /tmp file system? -m /tmp /tmp: Then we need to know what each PID is: ps -p 531 PID TTY TIME CMD 531? 00:00:00 screen So it is a screen process, it has no attached tty, and has used no time.

6 Beyond Interactive solution /2... or even better ps -p 531 -f UID PID PPID C STIME TTY TIME CMD john Oct19? 00:00:00 SCREEN (complete commandline is available adding the -l switch, but output is too long to fit on a slide) We would then need to repeat for each of the PIDs listed in the output of.

7 Beyond Step by Step Shell Function /2 1 first explode the output into tokens: -m /tmp tr -s \n /tmp: c... tr simply converts each occurrence of a given character into a different one: here we convert spaces in newlines; the -s option collapses repeated characters. There can be a few letters attached to PIDs: here c means «current directory».

8 Beyond Step by Step Shell Function /3 2 then take only PIDs out of the list: -m /tmp tr -s \n \ egrep -o [0-9] egrep selects only digits ([0-9]), optionally repeated (+), then outputs only the matching string (-o).

9 Beyond Step by Step Shell Function /4 3 finally pass the resulting list to ps with any appropriate option: -m /tmp tr -s \n \ egrep -o [0-9]+ xargs ps -f -p UID PID PPID C STIME TTY STAT TIME CMD john Oct19? Ss 0:00 SCREEN eve Sep29? Ss 0:00 /usr/bin/lamd xargs takes its STDIN and appends it to a given command line (here ps -f -p).

10 Beyond Step by Step Shell Function /5 As a last step we wrap everything in a shell function: function 2 () { $* \ tr -s " " "\n" \ egrep -o [0-9]+ \ xargs ps -fl -p } where $* is used so that any command line parameter given to our little function is passed to the inner command.

11 Example 2: Background: HPC cluster with the Maui job scheduler we need to know the user list for a given node nodes are not job-exclusive there can be jobs from different users sharing a node available commands: checknode reports information on a given node, including the list of job assigned to the node checkjob reports information on a given job, including the list of assigned nodes and the job owner

12 Interactive Solution Step by Step /1 ] checknode a206 checking node a206 State: Running (general information on node status omitted) Reservations: Job (x1) -1:18:00:36 -> 2:05:58:24 (3:23:59:00) Job (x1) -22:07:38 -> 3:01:51:22 (3:23:59:00) JobList: , Jobs and are running on node a206.

13 Interactive Solution Step by Step /2 ] checkjob checking job State: Running Creds: user:joe group:phys class:smp2 qos:slow WallTime: 1:18:39:39 of 4:00:00:00 SubmitTime: Tue Oct 21 15:43:36 (more information omitted) Job owner is user joe; job has run for 1 day, 18 hours and 39 minutes out of 4 days requested. Now we would need to repeat for job If we wanted to know current users of a dozen more nodes, the whole process is not exactly exciting...

14 Step by Step Shell Function /1 We are going to define a shell function node_user that is going to accept a single argument (a node name) and output the list of users on a node, along with their remaining time. 1 Let s start by extracting the list of running jobs on the node: checknode a206 grep JobList JobList: ,115848

15 Step by Step Shell Function /2 2 then explode the list to single components: checknode a206 grep JobList egrep -o [0-9] The regular expression [0-9]+ means «any digit, repeated one or more times»; the -o option tells egrep to output only the matching strings (not whole lines).

16 Step by Step Shell Function /3 3 then pass each of the list elements to the checkjob command: checknode a206 grep JobList egrep -o [0-9]+ xargs -i checkjob {} (very long output, not shown) xargs takes its standard input and uses it to build a new command line the -i option tells xargs to iterate over input elements and build a command line for each one {} is a placeholder in the command line where xargs will put each list element

17 Step by Step Shell Function /4 4 finally filter the insanely long output to get only user and WallTime lines: checknode a206 grep JobList egrep -o [0-9]+ xargs -i checkjob {} egrep -o user:[ˆ ]+ WallTime:.+ Another regular expression where is logical «OR» [ˆ ] means «everything that is not a space»

18 Step by Step Shell Function /5 The final magic: function node_user () { [ $# -ne 1 ] && { echo "Usage: $FUNCNAME nodename" 1>&2; return 1 }; checknode $1 grep JobList \ egrep -o [0-9]+ xargs -i checkjob {} \ egrep -o user:[^ ]+ WallTime:.+ }

19 Background: Example 3: you have some slightly unstable service that sometimes crashes and needs to be restarted; or maybe it is sometimes killed due to resource exaustion you don t want to deploy a full-featured monitoring system and/or need a quick-and-dirty solution you want a basic level of configurability, e.g. you want to be able to shut down a service and not have it restarted monitored services are important enough that you want them restarted without human intervention, but not so critical you need them to be restarted «now»

20 Interactive Solution Step by Step At some point node m001 starts looking funny... 1 go to node m001: ssh m001 2 check for service availability: service ntpd status ntpd is stopped 3 you known for sure that ntpd should be there, so you restart it: service ntpd restart ntpd: Synchronizing with time server: [OK] Starting ntpd: [OK]

21 Step by Step Shell Script /1 Version 0 script could be #!/bin/bash service ntpd status \ service ntpd restart The first part of the pipeline reports the service status both on STDOUT and in the exit code we are only interested in the exit code actually.

22 Step by Step Shell Script /2 A more advanced version of our script would check if ntpd is supposed to be there before deciding it is thime to (re)start it. This can be done with chkconfig: chkconfig --list ntpd ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off But what runlevel are we currently in? runlevel N 3 So this will tell us what we really need to know: RL=$( runlevel cut -d -f2 ) chkconfig --list ntpd grep -q ${RL}:on echo $? 0

23 Step by Step Shell Script /3 We can make a shell function with our little «check if service should be there» piece of script: function is_active() { [ "x$1" = "x" ] && return 1 local SERVICE=$1 local RL=$( runlevel cut -d -f2 ) chkconfig --list $SERVICE \ grep -q "${RL}:on" return $? }

24 Step by Step Shell Script /4 The new script will then be: #!/bin/bash (here goes the definition of is_active) is_active ntpd && \ { service ntpd status >/dev/null \ service ntpd restart ; }

25 Step by Step Shell Script /5 As a final step the script needs to be executed periodically. On RedHat-based distros, making the script executable and moving it to the /etc/cron.hourly directory is enough to have it executed once per hour. If your system doesn t support RedHat-style cron directories, or you want some different time interval you will need to work with the slightly old-fashioned crontab.

26