Introduction to Linux/Unix. Xiaoge Wang, ICER Jan. 14, 2016

Transcription

1 Introduction to Linux/Unix Xiaoge Wang, ICER Jan. 14, 2016

2 How does this class work We are going to cover some basics with hands on examples. Exercises are denoted by the following icon: Bold means commands which I expect you type on your terminal in most cases.

3 Green and Red Sticky Use the sjcky notes provided to help me help you. No S%cky = I am working Green = I am done and ready to move on (yea!) Red = I am stuck and need more Jme and/or some help

4 Agenda IntroducJon Linux Commands NavigaJon Get or create files Organizing files Closer look into files Searching for files DistribuJng files File permission Learn new commands Scripts Pipeline Make you own command Environment of a shell Summary

5 Agenda Linux Commands NavigaJon Get or create files Organizing files Closer look into files Searching for files DistribuJng files File permission Learn new commands Scripts Pipeline Make you own command Environment of a shell Summary

6 Introduction Get ready for adventure? Ticket(account)? Big map Linux/Unix Shell Overview of trail Commands Simple Shell script Get ready for HPC.

7 Exercise 0: Get ready Connect to HPCC (gateway) $ ssh username@hpcc.msu.edu Windows users MobaXterm Mac users Terminal Linux users? Read important message $ mod Go to a development node $ ssh dev-nodename

8 Mac Message of the Day

9 Show screen

10 Big Picture Shell Shell

11 Big Picture Shell CLI GUI

12 OS Shell Example

13 Overview of the Trail Commands Simple Shell script Get ready for HPC

14 Linux/Shell Tour Ready? GO!

15 Agenda IntroducJon Scripts Pipeline Make you own command Environment of a shell Summary

16 Example 1: Navigation Task: Wander around on the node. Change from one directory to another directory and see the list of files in a directory. Must see directories: home, /, /bin. Commands echo $HOME ls, ls l, ls a, cd, cd.., cd ~, cd pwd Concepts learned: Short cuts: ~,..,. Directories: current, parent, home, hidden Command format ( opjons, parameters )

17 Exercise 1: Task: Wander around on scratch space. 1. Go to /mnt/ls15/scratch/users/class0 2. Go to each directory and take look. 3. Go to $SCRATCH directory and take look. We Learned: what is scratch space?

18 Example 2.1: Create a files Task: make a directory ex1 that contains the file Done. Write the following in the file Done : This is what I have done. List of command you have done... Commands mkdir editor: nano history output using redirect > We Learned: Standard output redirect editor

19 Example 2.2: Get files Task: Make a directory workshop that contains: 1. Cheat sheet of linux shell commands from hjp://cli.learncodethehardway.org/ bash_cheat_sheet.pdf 2. Copy from /bin Commands wget cp, cp r We Learned: Copy files and directory.

20 Exercise 2: Task: Make a directory Workshop that contains: 1. Cheat sheet of linux shell commands from hjp://cli.learncodethehardway.org/ bash_cheat_sheet.pdf 2. Copy from /mnt/ls15/scratch/users/class0/ Intro2linux 3. A file Done same as example2.1 Hint: Follow steps in example2.1 and 2.2

21 Example 3: Organizing Files Task: Reorganize files. Create new directory Move a file from one directory to another directory (both up and down). Rename a file Commands cp mv Rm mkdir We Learned: absolute/relajve file name (path) /bin, /lib, /doc, /src/

22 Exercise 3: Task: Organize files in Intro2linux by copying data files into a directory DATA and moving pdf files into the directory Intro2linux/DOC. 1. Make directories DATA and DOC 2. copy data files to DATA 3. move pdf files to DOC Note: 1. DATA and DOC are at different levels. DOC is inside Intro2linux but DATA is outside. 2. Intro2linux contains same files as before.

23 Example 4: Closer look Task: Looking at the contents of the file Done. Find out how many Jmes ls -l command is executed. Commands: cat, less, less n, more, more -n grep, grep r, grep -i wc, We Learned: Get more informajon from a file content.

24 Exercise 4: Closer look Task: View the contents of the data file polls.csv. Find out how many polls were recorded from Michigan. Go to the DATA directory and view the file Use MI to find the data recorded from Michigan.

25 Example 5: Searching for a file Task: Find all the files with *.csv extension located in your home directory Command: find ~ name *.csv We Learned: Find a file on a Linux system Wild card *

26 Exercise 5: Searching for a file Find all CSV data files with *.csv extension in your scratch space.

27 Example 6: Distributing files Task: Put a set of files into an archive file, compress it and make it ready for distribujon Commands: tar, gzip, gunzip We Learned: tar/untar compress/uncompress (extract)

28 Exercise 6: Task: Make an archive file of ~/Examples in your home directory, compress it, copy it to your scratch space, open it. Pack a tar file Compress Copy to scratch space Open it

29 Example 7: File Permissions Task: Protect a file from the damage of write/copy/mv. Commands: chmod We Learned: Ownership of a file (u, g, o, a) Change permission (r,w, x)

30 Exercise 7: Task: Make sure the data files permission is right to allow people in the group to read and write. If it is not, then change it.

31 Example 8 :Self Learning Task: Find out the command to shuffle lines of a data file pet_store.csv, then create a file pet_shuffled.csv. Commands man, -help, Google search how to shuffle lines in a file linux shell We Learned: How to find command How to get details of a command Command shuf

32 Exercise 8: Self learning Task: Find out how to sort lines of a data file, then sort the file polls.csv to polls_sort.csv. 1. Find out the command to sort the file. 2. Try to sort the data file polls.csv. Note: Do not sort the first line!

33 Agenda IntroducJon Linux Commands NavigaJon Get or create files Organizing files Closer look into files Search files Distribute files File permission Learn new commands Summary

34 From Command to Script

35 Shell Script Combine commands together for more sophisjcated/complicated task. Grammar Environment and scope Parameters, expressions, expansion Control flow (branch, loops) ExecuJon (source vs. direct run)

36 Examples : Pipeline Task: count number of files under /bin. SoluJon 1: ls /bin > bin_list wc bin_list SoluJon 2: ls /bin wc We Learned: If the standard output from one command can be used as standard input of next command, a pipeline can be built. Commonly used: grep, wc, sort, less, more, uniq, head, tail,

37 Examples of pipeline Filtering output showq grep BatchHold SorJng the output ls l sort CounJng ls wc Bejer viewing ps more

38 Exercise 9: Pipeline Task: Find out how many jobs are in job queue. Among them, how many are running. 1. Count total jobs in job queue. Use showq to get list of jobs in queue. 2. Count only the running jobs. Use keyword Running to filter the jobs

39 Example 10:Making a command Task: Get ready for running jobs. Need to Copy data files to workspace Update a work log file Make sure it is ready: data files Make all tasks above into a command Commands $ cp./data.csv $SCRATCH/BuildModel/Data.csv $ date >> $SCRATCH/BuildModel/log $ ls -l $SCRATCH/BuildModel/Data.csv We Learned: If a task needs to be implemented by more than a single line command/pipeline and needs to be executed repeatedly, then consider making a command. Environment variable $SCRATCH >> vs. > File permissions need to be set to executable Add it to $PATH

40 Exercise 10: Making a command Task: Follow example 10 to make a command data_on_scratch

41 Example 11 : Environment Task: In addijon to example 10, we would like to Load a module Go to scratch space Make sure it is ready: module and scratch space Commands $ module load PETSc $ echo $PATH $ cd $SCRATCH/BuildModel $ pwd We Learned: Shell variable scope Environment variable $PATH, $PWD Source run vs. direct run

42 Exercise 11: Task: 1. Follow example 11 to make your command run_on_scratch.

43 Example 12 :Expressions Task: Same as Example 10 except ONLY copy the data file to workspace if data file is newer. Need to: Copy data files to scratch if newer Update a work log file Make sure it is ready: data files Make the whole task as a command Commands $ cp./data.csv $SCRATCH/BuildModel/Data.csv $ date >> $SCRATCH/BuildModel/log $ ls -l $SCRATCH/BuildModel/Data.csv We Learned: Expression

44 Exercise 12: Expression Task: follow example 12 to make a command data_update_on_scratch

45 Example 13 : Using Variables Task: There are 3 models similar to Example 11 under development in BuildModel. But we d like to go to a parjcular directory when working on one of the models. Use variables to specify the parjcular module to load and directory to go to. Take two variables Commands Similar as example 11 We Learned: PosiJon variable and its usage.

46 Exercise 13: Task: 1. Follow example 13 to make your command run_model_on_scratch with variable.

47 Summary Commands Cheat sheet for quick reference Very simple shell script Get examples from Internet & HPCC Class material: Intro2linux

48 Q & A Thank You!