Brief Introduction on Linux Related Technology. KONG, Lei

Transcription

1 Brief Introduction on Linux Related Technology KONG, Lei

2 Contents 1. Useful clients under Windows & Linux 2. Command line utilitie 3. Bash scripts

3 Section I Useful Tools under Windows & Linux

4 Clients under Windows for Remote Login Use PuTTY/SecureCRT as ssh client to login to the remote server. PuTTY: ham/putty/ SecureCRT: crt/index.html

5 Clients under Windows for File Transfer Use WinSCP to do file transfer between local and remote computer WinSCP SFTP (SSH File Transfer Protocol), mostly used with SSH-2 SCP (Secure Copy Protocol), mostly used with SSH-1

6 Clients under Windows for X Use Xmanager as Xwindow Server for X11 display. Xmanager:

7 Other Useful Tools GNU Screen lftp sophisticated command line ftp/http client GNU wget

8 Section II BASIC Useful Linux/Unix Tools (Particular Thanks: ZHAO, Shuqi)

9 GNU Text Utilities dos, unix, mac text file dos2unix unix2dos mac2dos time, estimate the time for the jobs time someprog progparam economize on disk run_certain_program -o /dev/stdout gzip -c > result.gz

10 GNU Text Utilities Is the program with correct format? cat -A filename edit the original file sed -i -e /badline/d *.fa perl -i.bak -npe s/old/new/g *.fa Input special characters in bash(eg. tab) ctrl+v special_character

11 sort -b --ignore-leading-blanks -d --dictionary-order -f --ignore-case -i --ignore-nonprinting -n --numeric-sort -r --reverse -k POS1[,POS2] --key=pos1[,pos2] -s --stable -t SEPARATOR --field-separator=separator -u --unique

12 sort example A sort key position may also have any of the option letters `Mbdfinr appended to it, in which case the global ordering options are not used for that particular field. sort sort -k5,6nr -k7,7

13 tar <options> for tar x to extract from an archive. c to Create an archive. t to list the contents of an archive. z to work with a gzipped (ending with.gz) archive. (GNU Tar only) j to work with a bzip2 (ending with.bz2) archive. (GNU Tar only) more general for -z and -j bash $ tar cf - path/to/dir/ gzip -c > xxx.tar.gz bash $ tar cf - path/to/dir/ bzip2 -c > xxx.tar.bz2

14 find Syntax: find pathname-lists option expression; Examples: To print out the current pathname for file myfile : bash $ find. -name myfile -print To print out the yes* files accessed in the last two days: bash $ find. -name yes* -atime +2 -print

15 find 如何将查询的结果作为命令行参数 tmp]$ cd find-example/ find-example]$ >a >b >c >d >e >f find-example]$ ls a b c d e f [kongl@soft find-example]$ find. -type f -exec mv {} {}.1 \; [kongl@soft find-example]$ ls a.1 b.1 c.1 d.1 e.1 f.1

16 Difference between.bash_profile and.bashrc Login with shell (ssh /etc/profile -> ~/.bash_profile -> ~/.bash_login -> ~/.profile #append to.bash_profile test -f ~/.bashrc &&. ~/.bashrc Login without shell (ssh cmd) /etc/bash.bashrc -> ~/.bashrc

17 Special Shell Variables $# The number of arguments. All arguments, as separate words. $* All arguments, as one word. $$ ID of the current process. $? Exit status of the last command. $0,$1,..$9,${10},${11}...${N} Positional parameters. cbimgmt $ cat wrapper.sh exec program.real $@ cbimgmt $ cat wrapper.sh exec program.real $@ WHAT S THE DIFFERENCE

18 Bash test and comparison functions test, [ ], [[ ]], (( )), or if-then-else-fi cbimgmt $ test expr #= [ expr ], return $? cbimgmt $ test 3 -gt 4 && echo true echo false cbimgmt $ [[ ( -d "$HOME" ) && ( -w "$HOME" ) ]] && \ echo "home is a writable directory" arithmetic comparisons -eq, -ne, -lt, -le, -gt, or -ge, String comparisons =,!=, <, or >

19 SSH Related Tips always type the user and host name? tmp]$ cat ~/.ssh/config Host soft Hostname User kongl

20 SSH Public Key Authentication ~]$ ls ~/.ssh/ config known_hosts ~]$ ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/kongl/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/kongl/.ssh/id_dsa. Your public key has been saved in /home/kongl/.ssh/id_dsa.pub. The key fingerprint is: e4:b4:8d:98:f5:6f:39:5e:f7:96:f8:0c:c0:d6:cf:73 ~]$ scp ~/.ssh/id_dsa.pub soft:.ssh/authorized_keys password: id_dsa.pub 100% KB/s 00:00 ~]$ ssh soft Last login: Sat Nov 28 15:34: from

21 SSH Copy without SCP I [kongl@soft tmp]$ dd if=/home/kongl/tmp/c ssh soft 'dd of=/home/kongl/tmp/e' 0+1 records in 0+1 records out 123 bytes (123 B) copied, 2.1e-05 seconds, 5.9 MB/s kongl@ 's password: 0+1 records in 0+1 records out 123 bytes (123 B) copied, seconds, 58.4 kb/s [kongl@soft tmp]$ ls a c d e find-example sort_example.txt test.sh [kongl@soft tmp]$ cat e Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy.

22 SSH Copy Without SCP II tmp]$ cat c ssh soft 'cat > /home/kongl/tmp/f' kongl@ 's password: [kongl@soft tmp]$ ls a c d e f find-example sort_example.txt test.sh [kongl@soft tmp]$ cat f Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy.

23 lftp lftp [ftp:// lftp -c command lftp -e cmd lftp -f script_file lftp commands! shell command debug level off cd, lcd get, put, mget, mput, pget mirror

24 Vim command mode operations :g/pattern/d (Delete lines with the specified pattern) :%!sed /pattern/d (same as above) :1,10s/old/new/g (Replace old with new in line 1-10 globally.) :r!date (Read current date.) :set nu, :set nonu (Display line number or not.) :set ai, :set noai (auto indent) :set hlsearch, :set nohlsearch (high light search results) n+yy (copy n lines),p (paste),dd (delete/cut the current line), u (undo) w (move cursor forward a word),b (move cursor backward a word) shift+d (delete from cursor to line end) ctrl+d (half page down),ctrl+u (half page up) :w (save),:q! (force quit without saving),:wq (save & quit)

25 Vim insert mode i (insert mode) esc (quit insert mode) ctrl+v special_character (input a special character in insert mode)

26 Task Manager Tips crontab crontab e # use /bin/sh to run commands, # no matter what /etc/passwd says SHELL=/bin/sh # run five minutes after midnight, every day 5 0 * * * $HOME/bin/daily.job &>> $HOME/tmp/out # run at 2:15pm on the first of every month * * $HOME/bin/monthly crontab filename

27 Four device file /dev/stdin /dev/stdout /dev/stderr /dev/null cbimgmt $ cat /etc/passwd do_job -i /dev/stdin -o /dev/stdout cbimgmt $ ln -s /dev/stdout o.png && convert in.ps o.png display cbimgmt $ find /etc name * -type f -exec ls -l {} \; 2>/dev/null

28 Bash shortcuts moving the cursor Ctrl + a Go to the beginning of the line (Home) Ctrl + e Go to the End of the line (End) Ctrl + p Previous command (Up arrow) Ctrl + n Next command (Down arrow) Alt + b Back (left) one word Alt + f Forward (right) one word Ctrl + f Forward one character Ctrl + b Backward one character Ctrl + xx Toggle between the start of line and current cursor position

29 Bash shortcuts - editing Ctrl + L Clear the Screen, similar to the clear command Ctrl + u Cut/delete the line before the cursor position. Alt + Del Delete the Word before the cursor. Alt + d Delete the Word after the cursor. Ctrl + d Delete character under the cursor Ctrl + h Delete character before the cursor (Backspace) Ctrl + w Cut the Word before the cursor to the clipboard. Ctrl + k Cut the Line after the cursor to the clipboard. Alt + t Swap current word with previous Ctrl + t Swap the last two characters before the cursor (typo). Esc + t Swap the last two words before the cursor. ctrl + y Paste the last thing to be cut (yank) Alt + u UPPER capitalize every character from the cursor to the end of the current word. Alt + l Lower the case of every character from the cursor to the end of the current word. Alt + c Capitalize the character under the cursor and move to the end of the word. Alt + r Cancel the changes and put back the line as it was in the history (revert). ctrl + _ Undo TAB Tab completion for file/directory names

30 Bash shortcuts - history Ctrl + r Recall the last command including the specified character(s) searches the command history as you type. Equivalent to : vim ~/.bash_history. Ctrl + p Previous command in history (i.e. walk back through the command history) Ctrl + n Next command in history (i.e. walk forward through the command history) Ctrl + s Go back to the next most recent command. (beware to not execute it from a terminal because this will also launch its XOFF). Ctrl + o Execute the command found via Ctrl+r or Ctrl+s Ctrl + g Escape from history searching mode!! Repeat last command!abc Run last command starting with abc!abc:p Print last command starting with abc!$ Last argument of previous command ALT +. Last argument of previous command!* All arguments of previous command ^abc^def Run previous command, replacing abc with def

31 Bash shortcuts process control Ctrl + C Interrupt/Kill whatever you are running (SIGINT) Ctrl + l Clear the screen Ctrl + s Stop output to the screen (for long running verbose commands) Ctrl + q Allow output to the screen (if previously stopped using command above) Ctrl + D Send an EOF marker, unless disabled by an option, this will close the current shell (EXIT) Ctrl + Z Send the signal SIGTSTP to the current task, which suspends it. To resume the task enter fg.

32 screen related tips create a screen: #screen [ENTER] fork a new shell #screen [ENTER] or C-a c name current window: C-a A switch to certain window: C-a Present a list of all windows: C-a Send the command character (C-a): C-a a detach screen: C-a d Toggle to previously window: C-a C-a To split screens: C-a S (Shift-s) To unsplit screens: C-a Q (shift-q) To switch between screens: C-a TAB resumes a detached session: screen -r Attach to a not detached session: screen -x

33 Section III Bash Script Programming

34 Readings Advanced Bash-Scripting Guide BASH Programming - Introduction HOW-TO Intro-HOWTO.html

35 UNIX Philosophy KISS: Keep It Simple and Stupid OT2: One Time, One Thing Collaboration makes the world better

36 Very Simple Scripts Hello World tmp]$ cat hello.sh #!/bin/bash echo Hello World tmp]$./hello.sh Hello World A very simple backup script #!/bin/bash tar -czf /var/my-backup.tgz /home/me/

37 Input & Output Standard input (stdin): keyboard; Standard output (stdout): monitor; A process has 3 special file descriptor.

38 Process file descriptor $ ls -l /proc/14649/fd total 0 lrwx lion lion 64 Aug 16 07:32 0 -> /dev/pts/8 lrwx lion lion 64 Aug 16 07:32 1 -> /dev/pts/8 lrwx lion lion 64 Aug 16 07:32 2 -> /dev/pts/8 Why did we look at directory /proc/14649/fd? /proc is for "process information" /14649 is for "process #14649", the running bash shell /fd is for "file descriptors", the list of the process' open file descriptors Now we see that we have three open file descriptors: 0,1,2. 0 is stdin 1 is stdout 2 is stderr

39 IO Redirection 在 shell 中, 使用者可以利用 > 和 < 来进行输入输出重定向 command >file Redirect command s stdout to a file command &>file Redirect command s stdout to a file along with the stderr command >>file Append the command s stdout to the file command &>> file Append the command s stdout to the file along with the stderr command 2>&1 Redirect stderr to stdout. command i>&j Redirect file descriptor i to j. j<>filename Open file filename, and assign file descriptor j to it.

40 IO Redirection examples ~]$ > a ls [kongl@soft ~]$ cat a a Desktop F-7-i386-DVD.iso fedora7 gentoo new-svap surg _40u4_linux64.tar.gz tmp zimbra

41 The order bash deal with io redirection: left to right tmp]$ ls tmp]$ >a tmp]$ >c tmp]$ ls a c [kongl@soft tmp]$ rm b > d 2>&1 [kongl@soft tmp]$ cat d rm: cannot remove `b': No such file or directory [kongl@soft tmp]$ rm b 2>&1 >d rm: cannot remove `b': No such file or directory [kongl@soft tmp]$ cat d [kongl@soft tmp]$ [kongl@soft tmp]$ 2>&1 rm b > d rm: cannot remove `b': No such file or directory [kongl@soft tmp]$ >d 2>&1 rm b [kongl@soft tmp]$ >d rm b 2>& 我是分隔符 [kongl@soft tmp]$ rm b 2>&1 2>a [kongl@soft tmp]$ cat a rm: cannot remove `b': No such file or directory [kongl@soft tmp]$ rm b 2>a 2>&1 rm: cannot remove `b': No such file or directory [kongl@soft tmp]$ cat a [kongl@soft tmp]$

42 Redirection order test tmp]$ cat c Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy. [kongl@soft tmp]$ cat c > a > d > e > f [kongl@soft tmp]$ cat a [kongl@soft tmp]$ cat d [kongl@soft tmp]$ cat e [kongl@soft tmp]$ cat f Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy.

43 More Funny tmp]$ cat test.sh #!/bin/bash cat c rm b [kongl@soft tmp]$ ls a c d test.sh [kongl@soft tmp]$./test.sh > a rm: cannot remove `b': No such file or directory [kongl@soft tmp]$ cat a Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy. [kongl@soft tmp]$./test.sh > a 3>&1 1>&2 2>&3 3<&- Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy. [kongl@soft tmp]$ cat a rm: cannot remove `b': No such file or directory

44 File descriptor tmp]$ ls c d test.sh [kongl@soft tmp]$ cat c Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy. [kongl@soft tmp]$ 3<>a cat c>&3 [kongl@soft tmp]$ cat a Greg Sanders: All work and no play makes Greg a dull boy. Gil Grissom: All play and no work makes Greg an UNEMPLOYED boy.

45 PIPE Process A STDOUT STDIN Process B STDOUT STDERR STDERR

46 Sample: simple pipe with sed tmp]$ ls a c d e f find-example hello.sh sort_example.txt test.sh [kongl@soft tmp]$ ls sed -e "s/[aeio]/u/g" u c d u f fund-uxumplu hullu.sh surt_uxumplu.txt tust.sh

47 Sample: an alternative to ls -l *.txt tmp]$ ls -l *.txt -rw-r--r-- 1 kongl cbi 381 Nov 28 11:02 sort_example.txt [kongl@soft tmp]$ ls -l grep "\.txt$" -rw-r--r-- 1 kongl cbi 381 Nov 28 11:02 sort_example.txt

48 Variables Sample: Hello World! using variables #!/bin/bash STR="Hello World!" echo $STR Sample: A very simple backup script (little bit better) #!/bin/bash OF=/var/my-backup-$(date +%Y%m%d).tgz tar -czf $OF /home/me/

49 Local variables tmp]$ cat local_hello.sh #!/bin/bash HELLO=Hello function hello { local HELLO=World echo $HELLO } echo $HELLO hello echo $HELLO [kongl@soft tmp]$./local_hello.sh Hello World Hello

50 Conditionals if expression then statement if expression then statement1 else statement2 if expression1 then statement1 else if expression2 then statement2 else statement3

51 Examples Basic conditional example if.. then #!/bin/bash if [ "foo" = "foo" ]; then echo expression evaluated as true fi Basic conditional example if.. then... else #!/bin/bash if [ "foo" = "foo" ]; then echo expression evaluated as true else echo expression evaluated as false fi Conditionals with variables #!/bin/bash T1="foo" T2="bar" if [ "$T1" = "$T2" ]; then echo expression evaluated as true else echo expression evaluated as false fi

52 Loops - for [kongl@soft tmp]$ cat for_example.sh #!/bin/bash for i in $( ls *.*); do echo item: $i done echo "============separater==========" for i in `seq 1 5`; do echo $i done [kongl@soft tmp]$./for_example.sh item: for_example.sh item: hello.sh item: local_hello.sh item: sort_example.txt item: test.sh ============separater==========

53 Loops - while [kongl@soft tmp]$ cat while_example.sh #!/bin/bash COUNTER=0 while [ $COUNTER -lt 5 ]; do echo The counter is $COUNTER let COUNTER=COUNTER+1 done [kongl@soft tmp]$./while_example.sh The counter is 0 The counter is 1 The counter is 2 The counter is 3 The counter is 4

54 Loops - until [kongl@soft tmp]$ cat until_example.sh #!/bin/bash COUNTER=10 until [ $COUNTER -lt 5 ]; do echo COUNTER $COUNTER COUNTER=`expr $COUNTER - 1` done [kongl@soft tmp]$./until_example.sh COUNTER 10 COUNTER 9 COUNTER 8 COUNTER 7 COUNTER 6 COUNTER 5

55 Functions without parameter tmp]$ cat fun_no_param.sh #!/bin/bash function quit { exit } function hello { echo "Hello World!" } hello quit echo "It should not reach here!" [kongl@soft tmp]$./fun_no_param.sh Hello World!

56 Functions with parameters tmp]$ cat fun_with_param.sh #!/bin/bash function quit { exit } function my_echo { echo $1 } my_echo Hello World my_echo "Hello World" quit echo "It should not reach here, either!" [kongl@soft tmp]$./fun_with_param.sh Hello Hello World

57 Using select to make simple menus tmp]$ cat select_menu.sh #!/bin/bash OPTIONS="Hello Test Quit" select opt in $OPTIONS; do if [ "$opt" = "Quit" ]; then echo done exit elif [ "$opt" = "Test" ]; then echo "This is a test." elif [ "$opt" = "Hello" ]; then echo "Hello World!" else clear echo "bad options!!!" fi done [kongl@soft tmp]$./select_menu.sh 1) Hello 2) Test 3) Quit #? 1 Hello World! #? 2 This is a test. #? 3 done

58 Using the command line parameters tmp]$ cat cmd_param.sh #!/bin/bash if [ -z "$1" ]; then echo usage: $0 directory exit fi SRCD=$1 TGTD="$HOME/tmp/" BASENAME=`basename $SRCD` OF=$BASENAME-$(date +%Y%m%d).tgz echo tar czf "$TGTD/$OF" $SRCD tar czf "$TGTD/$OF" $SRCD [kongl@soft tmp]$./cmd_param.sh usage:./cmd_param.sh directory [kongl@soft tmp]$./cmd_param.sh find-example/ tar czf /home/kongl/tmp//find-example tgz find-example/ [kongl@soft tmp]$ ls a d find-example fun_no_param.sh local_hello.sh test.sh c e find-example tgz fun_with_param.sh select_menu.sh until_example.sh cmd_param.sh f for_example.sh hello.sh sort_example.txt while_example.sh

59 Reading user input with read tmp]$ cat read_example.sh #!/bin/bash echo Please, enter your name read NAME echo "Hi $NAME!" echo Please, enter your firstname and lastname read FN LN echo "Hi! $LN, $FN!" tmp]$./read_example.sh Please, enter your name Kidding Hi Kidding! Please, enter your firstname and lastname Kidding Serious Hi! Serious, Kidding!

60 Arithmetic evaluation tmp]$ cat arithmetic.sh #!/bin/bash echo "echo 1 + 1" echo echo "echo \$((1+1))" echo $((1+1)) echo "echo \$[1+1]" echo $[1+1] echo "echo \$[3/4]" echo $[3/4] echo "echo 3/4 bc -l" echo 3/4 bc -l [kongl@soft tmp]$./arithmetic.sh echo echo $((1+1)) 2 echo $[1+1] 2 echo $[3/4] 0 echo 3/4 bc -l

61 Getting the return value of a program [kongl@soft tmp]$ cat return_val.sh #!/bin/bash cd /dada &> /dev/null echo return value: $? cd $(pwd) &> /dev/null echo return value: $? [kongl@soft tmp]$./return_val.sh return value: 1 return value: 0

62 Capturing a commands output [kongl@soft tmp]$ cat capture_output.sh #!/bin/bash LIST=`ls find-example` for f in $LIST do echo FILE: $f done [kongl@soft tmp]$./capture_output.sh FILE: a.1 FILE: b.1 FILE: c.1 FILE: d.1 FILE: e.1 FILE: f.1

63 DEBUG The first line like this will produce some intresting output information #!/bin/bash -x Example: tmp]$ cat debug.sh #!/bin/bash -x PARAM=$1 echo $0 echo $1 tmp]$./debug.sh test.sh + PARAM=test.sh + echo./debug.sh./debug.sh + echo test.sh test.sh

64 Acknownlegement Dr. ZHAO, Shuqi

65 REFERENCE tml

66 Thanks

67

68 Some Linux Tips mkfifo, a program to create named pipe

69 GNU Text Utilities dos, unix, mac text file dos2unix unix2dos mac2dos time, estimate the time for the jobs time someprog progparam economize on disk run_certain_program -o /dev/stdout gzip -c > result.gz

70 /proc directory organization /proc/cpuinfo /proc/meminfo /proc/filesystems /proc/version /proc/[0-9]+ /proc/[0-9]+/cmdline /proc/[0-9]+/environ /proc/[0-9]+/fd /proc/[0-9]+/cwd

71 Some Linux Tips export hash -r reset wc #the \n number, not the line number ldd cbimgmt $ ldd /bin/bash linux-gate.so.1 => (0xffffe000) libncurses.so.5 => /lib/libncurses.so.5 (0xb7f6e000) libdl.so.2 => /lib/libdl.so.2 (0xb7f6a000) libc.so.6 => /lib/libc.so.6 (0xb7e50000) /lib/ld-linux.so.2 (0xb7fb8000)

72 file copy related Tips copy certain file to a new directory cbimgmt $ mkdir target cbimgmt $ find src -name *.class grep -v secret \ cpio -pd target Compress certain files cbimgmt $find../dir1/ cpio -o --format=tar > test.tar or cbimgmt $find../dir1/ cpio -o -H tar gzip> test.tar.gz work with rsync cbimgmt $ mkdir target cbimgmt $ rsync --exclude="*.java" -av src target/

73 Some Linux Tips Remember the work path cbimgmt $ cd some_path. cbimgmt $ do some work cbimgmt $ forget where to return? cbimgmt $ pushd some_path cbimgmt $ do some work cbimgmt $ popd Reorder the file column and row cbimgmt $ cat data cbimgmt $ cat data tr ' ' '\n' xargs -l5 cbimgmt $ cat data tr ' ' '\n' paste cbimgmt $ cat data tr ' ' '\n' xargs -l3 tr ' ' '+' bc 15 40

74 Some Linux Tips look up or signal process pgrep [-flvx] [-d delimiter] [-n -o] [-P ppid,...] [-g pgrp,...] [-s sid,...] [-u euid,...] [-U uid,...] [-G gid,...] [-t term,...] [pattern] pkill [-signal] [-fvx] [-n -o] [-P ppid,...] [-g pgrp,...] [-s sid,...] [-u euid,...] [-U uid,...] [-G gid,...] [-t term,...] [pattern] cbimgmt $ pgrep -u root #= ps -ef egrep '^root ' awk '{print $2}' cbimgmt $ pkill -HUP syslogd display a tree of processes pstree [-a] [-c] [-h -Hpid] [-l] [-n] [-p] [-u] [-Z] [-A -G -U] [pid user]

75 SSH Related Tips ## Linux Laptop.ssh/config ## Host work HostName User sporkey LocalForward :80 LocalForward :22 LocalForward :139 Host workssh HostName localhost User donkey Port HostKeyAlias workssh ## Linux Laptop.ssh/config ## Host workssh HostName ProxyCommand ssh /usr/bin/nc -w 1 %h 22