Deniz Savas and Michael Griffiths

Transcription

1 Introduction to Linux on Iceberg Deniz Savas and Michael Griffiths Corporate Information and Computing Services The University of Sheffield 2012

2 Introduction to Linux on Iceberg Working with Directories Help! Running Programs Editors The Shell Secure Shell Grid Security and Applying to Use Grid Resources Exercise

3 / (root) Directory Structure usr home data cs4un1 cs4un2 Home directory of user cs4un1 : /home/cs4un1 When you log in you are positioned in your home directory. The environment variable $HOME is also set to contain this directory name.

4 Format of Unix commands command [option...] [filename...] eg: ls ls -l tutorial more tutorial

5 Searching in files grep string file This command finds and prints out the lines in the file(s) containing the specified string string file = word or phrase = file or list of files (wild_card can be used) Note: We strongly advise that the string is quoted. Examples: grep Green Man england.dat grep Zodiac t*.dat grep Zone[a-z] security.fil

6 Finding files and information about them find :Finds a file in the directory hierarchy Example : find. name myprog.* -print Note: it is safe to enclose strings containing wild characters in quotas which : Shows in which directory a command is located. l Syntax: which command_name file : Can be used to see what type of data a particular file contains. For example, script, program, library, executable binary etc Syntax: file command_name

7 Using find Searches recursively for specified file find [-H] [-L] [-P] path_list options action -H L P control the treatment of symbolic links ( not of concern in an average user's directory) Options: -name filename ( e.g. -name diary.txt ) -size nnn ( e.g. -size +10M ) -atime nnn (time last accessed) (e.g. -atime -3 ) -mtime nnn (time last modified) (e.g. -mtime -7 ) Note: number can be minus (-) to indicate less then, plus(+) to indicate more than or without a sign to indicate exactly.

8 Find Examples Find a file called mystery in /bin and /usr and print the result find /bin /usr name libgl* print find. user myusername print Find files accessed in the last n days find. atime n print Find files modified within the last n days find. mtime -n -print (e.g. find. -ntime -7 -print Note: -print is usually the default and can be omitted.

9 Issuing System Commands on Files Found Use the exec option of the find command Copy found file to a specified directory, curly braces instruct find to substitute the name of the file in this location find. name *.doc exec cp {} document \; File is copied to directory document ; is needed to terminate the execute command \ escape character to take away special meaning of ; in find Reaffirm execution of system commands using ok option find. -name *.doc -ok rm {} \; Search for a string in all the find files -find. name *.doc exec grep Iceberg {} \; -print

10 Resources 1: Filestore Three areas of filestore available on Iceberg: 1. /home : A permanent, secure, backed up area directory belonging to you only under: /home/your_username Limit: 5 GBytes 2. /data :A permanent storage area which is not backed up belonging to you under: /data/your_username Limit: 50 GBytes 3. /fastdata : A semi-temporary storage area where files older than 60 days will be deleted. Create directory /fastdata/your_username and use it. Currently no limit.

11 Resources2: Scratch area Temporary data storage on local compute nodes I/O much faster than NFS mounted /home and /data Data not visible to other worker nodes and not backed up Create a directory using your username in /scratch on a worker and work from this directory The data in the /scratch area is deleted periodically when the worker is not accessed by any processor or jobs

12 Resources 4: Important Notes The data area is not backed up. Check quota regularly if you go over quota the account will become frozen you ll need to contact iceberg-admins Check quota using the command quota If you exceed your quota using the RM command Note upper case

13 Resources 5 : Transferring Data Command line tools such as scp, sftp Use sftp tools such as winscp for windows

14 Running programs Two modes of operation foreground and background Foreground Interact with program via keyboard/screen Background No connection with keyboard/screen Submit to backbround by Appending & EG: myprog >& myfile & The symbols >& redirect output and any errors to the file myfile Although the above method of running jobs on the background is feasible, the prio we recommend that you submit your background into the batch queue via the qsub command.

15 Redirection Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues. Standard Input (default=>keyboard) Standard Output (default=>screen) Redirection symbols <,>,>> can be used to specify files as the source/destination of the read/write operations to override the above defaults.

16 Redirection continued Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues. To redirect the output to a file use the > symbol. Example: ls -l > dirlist The > symbol should be used with care as it may over-write an existing file. >> symbol can be used instead if the output should be appended to the end of an existing file rather than over-writing it. Example: ls -l >> logfile If nothing is directed to a file then a zero size file is created, or if the file already existed then the contents of the file is removed Example: > afile The file /dev/null is a special symbol to indicate a black-hole Example : ls l > /dev/null

17 Redirection continued If any program expects any of it s input from the standard-input-channel, i.e. the keyboard, it can also read the same information from a file by redirection. To read input from a file use the < symbol. Example: write cs1xyz < message.fil Here any text input {write} program expects from the keyboard will be simply read from a file named message.fil. Each end-of-line will be treated as an <ENTER> on the keyboard.

18 Piping Feeding the output of one command into the input of another command The symbol is called a pipe command command eg: ls -al more ls -la grep Nov

19 Examples of re-direction and piping ls l grep Jun ls l grep `Jun` > june_files ls l grep `Jun` cut c > june_files cut c 1-10 < test_files aspell l < message.txt > report.txt grep fluent < news.dat grep fluent < news.dat cut c 1-72 (grep fluent < news.dat ) cut c 1-72 (grep fluent < news.dat ) cut c 1-72 > fluent.news

20 Foreground Program Control Kill a program Stop a program Ctrl C Ctrl Z Note a stopped program still exists in the system and hence can be re-started by the commands; bg : in the background fg : in the foreground

21 Program control within current shell jobs Lists jobs (programs) bg %job_id Place a job in the background fg %job_id Return a job to the foreground stop %job_id Stop a job kill %job_id Kill a job Process_id can be used in place of %job_id for more definitive way of identification. jobs [1] + Running time.sh > out stop %1 [1] + Stopped (signal) time.sh > out bg %1 [1] + time.sh > out & kill %1 Terminated

22 Program control using ps and kill ps Report process status ps -f -u username UID PID PPID C STIME TTY TIME CMD username :23:04 pts/50 0:00 sleep 2 username :05:45 pts/50 0:01 -csh username :35:07 pts/50 0:00 /bin/sh time.sh kill Terminate process Find its process ID (PID) using the ps command Kill the process using the kill command kill Sometimes kill on its own does not work so try kill -KILL or kill -9 pid

23 Running programs on iceberg Iceberg is the gateway to the cluster of worker nodes and the only one where direct logging in is allowed. Iceberg s main purpose is to allow access to the worker nodes but NOT to run cpu intensive programs. All cpu intensive computations must be performed on the worker nodes. This is achieved by the qsh command for the interactive jobs and qsub command for the batch jobs. Once you log into iceberg, taking advantage of the power of a worker node for interactive work is done simply by typing qsh and working in the new shell window that is opened. This what appears to be a trivial task has would in fact have queried all the worker nodes for you and started a session on the least loaded worker in the cluster. The next set of slides assume that you are already working on one of the worker nodes (qsh session).

24 Filename Completion Complete a partially typed filename Operation Type enough characters to uniquely identify the name Press the Tab key (for C shell use Esc key) If the response is a bleep press Ctrl-d to list possible matches

25 Operation Repeating Previous Commands history List previous commands!! re-run last command!n re-run the nth command!str last command starting with str eg:!vi Setup Add the following to your.cshrc file set history=40

26 Command Aliasing Definitions alias ll=' ls -lf ' alias h=' history 24 ' alias dir=' ls ' Setup Add alias definitions to your.bashrc file

27 UNIX Utilities tar compress sed awk

28 Secure Shell What is ssh? How to use it? Single sign on using ssh Digital certificates Accessing and Managing Grid Resources Wrgrid commands Further Information

29 Secure Shell Program to log into another computer over a network Execute commands on a remote machine Move files from one machine to another Provides strong authentication and secure communications over insecure channels. Intended as a replacement for rlogin, rsh, rcp, and rdist.

30 The Secure shell protocol SSH is a new method of communications over the Internet that encrypts data end-to-end. Replaces telnet, ftp, rsh and rcp Components Secure shell ssh Secure ftp sftp Secure copy scp

31 ssh You only need the SSH client. The server is unnecessary, unless you wish to connect back to your home machine via the Internet using SSH. Connecting to a WRG node ssh -l wrsmg maxima.leeds.ac.uk To use X-windows add the "-X" flag SSH will then carry Xwindows traffic over the Internet to connect Range of options for changing ports, specifying authentication files, encryption algorithms etc. Use man ssh for help with options

32 Examples Running X Windows Apps File manager and NAG Iris explorer on Maxima After using telnet or rsh to access a remote host setenv DISPLAY workstation_address:1.0 Workstation address can be an ip number of the workstation Check ip number using nslookup (on linux), ipconfig (pc)

33 Transferring files to/from iceberg Summary of file transfer methods as well as links to downloadable tools for file transfers to iceberg are published at: Command line tools such as scp, sftp and gftp are available on most platforms. Can not use ftp ( non-secure ) to iceberg. Graphical tools that transfer files by dragging and dropping files between windows are available winscp, coreftp, filezilla, cyberduck

34 Pitfalls when transferring files ftp is not allowed to by iceberg. Only sftp is accepted. Do not use spaces in filenames. Linux do not like it. Secure file transfer programs sftp classify all files to be transferred as either ASCII_TEXT or BINARY. All SFTP clients attempt to detect the type of a file automatically before a transfer starts but also provide advanced options to manually declare the type of the file to be transferred. Wrong classification can cause problems particularly when transfers take place between different operating systems such as between Linux and Windows. If you are transferring ASCII_TEXT files to/from windows/linux, to check that transfers worked correctly while on iceberg, type; cat v journal_file If you see a ^M at the end of each line you are in trouble!!! CURE: dos2unix wrong_file on iceberg

35 Working with files To copy a file: cp my_file my_new_file To move ( or rename ) a file : mv my_file my_new_file To delete a file : rm my_file To list the contents of a file : less file_name To make a new directory( i.e. folder) : mkdir new_directory To copy a file into another directory: cp my_file other_directory To move a file into another directory: mv my_file other_directory To copy a directory to another location: cp R directory other_directory To remove a directory and all its contents!!!: rm R directory ( use with care ) Wildcards : * means matching any sequence of characters. For example: cp *.dat my_directory

36 Your filestore Three areas of filestore always available on iceberg These areas are available from all headnode and worker nodes. 1. home directory: /home/username 5 GBytes allocations Permanent, secure, backed up area ( deleted files can be recovered ) 2. data directory /data/username 50 GBytes of ollocation Not backed but mirrored on another server 3. /fastdata area /fastdata Much faster access from MPI jobs No storage limits but no backup, or mirroring Files older than 90 days gets deleted automatically Always make a directory under /fastdata and work there.

37 Storage allocations Storage allocations for each area are as follows: On /home 5 GBytes On /data 50 GBytes No limits on /fastdata Check your usage and allocation often to avoid exceeding the quota by typing quota If you exceed your quota, you get frozen and the only way out of it is by reducing your filestorage usage by deleting unwanted files via the RM command ( note this is in CAPITALS ). Requesting more storage: iceberg-admins@sheffield.ac.uk to request for more storage. Excepting the /scratch areas on worker nodes, the view of the filestore is the same on every worker.

38 Secure ftp (sftp) Establishes an FTP-style file transfer session between the Unix systems sftp command always used in the form: sftp e.g.from titania sftp

39 Transferring Files Using sftp From the SFTP prompt (sftp>) can do the following: get command to retrieve a file from the remote Unix server. get test.txt put command to transfer a file from your Unix system to the remote Unix system you are connected to. put file2.txt

40 Navigating file systems using sftp From the SFTP prompt (sftp>) can do the following: ls command to display the contents of a directory on the remote Unix system you are connected to. ls /home/user. Will display the contents of the directory /home/user on the remote Unix system. cd and lcd commands change current remote directory, or current local directory. e.g. cd /home/user. Will change the current remote directory to /home/user.

41 Summary of sftp commands 1 mget Retrieve multiple files from server mput Transfer multiple files to server pwd Display remote working directory quit or exit Quit sftp rename oldpath newpath Rename remote file rmdir path Remove remote directory rm path Delete remote file version Show SFTP version? Synonym for help

42 Summary Listing of SFTP commands 2 ascii Use text transfer mode help Display the help text image Use binary transfer mode lls [ls-options [path]] Display local directory listing lmkdir path Create local directory ls [path] Display remote directory listing mkdir path Create remote directory put local-path [remote-path] Upload file

43 Secure copy - scp Using SCP Fast, easy method to copy single files from your Unix system to a remote Unix system.

44 Retrieving a file using SCP To retrieve a file from a remote Unix system, the syntax is: scp username@server:file local-file username= username on the remote system server= the name of the remote Unix system file= the file to retrieve from the remote system local-file= the location you wish to save the file to on your local Unix system

45 Transferring a file to a remote Unix system using SCP The syntax is: scp local-file username@server:file local-file= the file to transfer from the local system username= username on the remote system server= the name of the remote Unix system file= the location you wish to save the file to on the remote Unix system

46 Transferring Multiple Files mget and mput with sftp Use tar and compress to package a directory tree scp transfers the packaged directory tree uncompress and extract directory tree using tar

47 Security Authentication Are you who you say you are? Authorisation What are you permitted to do? Message protection Integrity Confidentiality Single sign-on Delegation

48 SSH Authentication SSH allows to perform authentication based on what we know (our key pass-phrase) and what we have (our private key).

49 SSH Authentication Enable single sign on to remote resources that use ssh. Authentication Utilities ssh-keygen Authentication key pair generation ssh-agent Authentication agent ssh-add Adds identities for authentication agent

50 authorized_keys ssh files and directories Contains public keys of hosts and users authorised to access this host known_hosts List of hosts from which ssh authentication is allowed. Contains public key for remote host

51 Enabling Single Sign On Using SSH Enable using the Utility ssh-keygen ssh-keygen Generates and manages authentication keys for ssh

52 Steps for setting up single sign on General steps for enabling your local host to access an account on a remote host without providing a password Generate a key pair from your local host Copy public key to remote host you require access Edit authorized_keys file on the remote host

53 Running ssh-keygen Generate a key pair of type rsa ssh-keygen t rsa At the prompt for a file name press return the default filename will be generated for the pair At the prompt for a pass phrase and the prompt to re-enter the pass phrase just hit return. A pair of keys has now been generated the private key must be protected.

54 Preparing the Remote Host for Single Sign On Copy the public key you created using ssh-keygen to the.ssh directory on the remote host Append your public key to the authorised_keys file in the.ssh directory Single sign on using secure shell is now enabled.

55 Comments Enables distributed application shell scripts that request applications and transfers resources between different systems on which you have accounts. Approach does not require logon to access each node

56 Grid Security Secure shell and PKI Grid Security Infratstructure X509 certifcates Grid proxy My Proxy

57 Digital Certificates Electronic passport that allows you to verify your identity online. Uses the techniques of public key encryption - you sign your certificate with a private key and broadcast a public key. Allows remote computers to verify that you are who you claim to be.

58 How to Obtain an X509v3 Certificate Go to and follow the Request certificate" link (remember PIN) Get request approved by local registration authority (ID required) After notification return to get certificate, install it and test it. Install certificate

59 Further Help USPACE Linux Group GNU/Linux Command-Line Tools Summary Bash Guide for Beginners Advanced Bash-Scripting Guide The Linux Documentation Project