Intro to Linux (fun with UNIX) (( and the command line interface!))

Transcription

1 Intro to Linux (fun with UNIX) (( and the command line interface!))

2 Overview Why Linux? Your environment Using ThinLinc The command line interface (CLI/terminal/shell) Linux directory structure: in the File system Working with files Pipes and Where to find help? SSH + FTP

3 Why use Linux? The biggest part of the so/ware used in the field of bioinforma7cs is largely wri:en for POSIX systems (such as Linux and UNIX). Why use Command Line Interface (CLI)? Many tools do not come with a Graphical User Interface (GUI) High effort to develop good GUIs. Displaying of GUI oten not possible (e.g. via SSH clients).

4 CentOS 7 Open Linux distribu@on built upon Red Hat Enterprise Linux (RHEL). Linux is a mul@- access, mul@- tasking system: mul@ple users may be logged in and run mul@ple tasks on one machine at the sharing resources (CPUs, memory, disk space). This is what you will be doing during the course.

5 Your environment In this class you will be using a IGEL thin client. A thin client refers to either a sotware program or to an actual computer that relies heavily on another computer to do most of its work. Think of it as a dumb computer that allows you to access our server (much bigger computer) stored in the data center. You get access to computa@onal resources that would not be available in a normal computer, such as the one you use at work or at home. The server we will be using is a HP DL 580 G7 4 CPUs x 8 cores + hyperthreading = 64 threads 1 TB RAM access to 100 TB storage

6 Logging in to our server It s easy! You will start with a fairly familiar- looking desktop. Double click on highlighted icon.

7 Enter your TU Graz the you use to login to online.tugraz.at

8 ATer you ve entered your click Connect.

9 Welcome to your Linux desktop

10 Logging out of your session At the end of each class please log out of your session. To log out simply click this icon at the top let of your screen. A dialog like the one on the right will appear, make sure you have selected Save this session for future logins and then click the Log Out bujon. You will be returned to the ini@al desktop of the thin client. Try it now!

11 Session control: F8 Pressing F8 during your ThinLinc remote desktop session gives you access to a few more op@ons. Disconnect session: Preserves your session! à programs stay running Full Screen mode: Switches on windowed mode. à allows you to toggle between local and remote environment.

12 Before we begin

13 The command line interface (CLI) The CLI is a text- based interface that allows computer users to type commands that get executed by the opera@ng system. You call programs, modify files and control your system just by wri@ng a bit of text. We will be using the CLI to do most of the lab tasks in this class. There are different types of Unix shells or CLIs available, the one we are using is called Bash. Other good examples are tcsh, zsh or ksh. A command shell is characterized by the available commands.

14 How to open a shell window

15

16 Linux directory structure You won t find any Windows, Program Files, or Users folders if you start browsing around the file system on your Linux computer. No drive lejers like C:\ Lowest level of the File- Tree is the root symbol / External storage devices are mounted to the file tree. For example our external storage is mounted at /export/ 2 most important folders we will be using in this course: /export/home/username your user home directory similar to the Windows C:\Users\User /export/programs contains all the bioinforma7cs so/ware used in the course.

17 in the file system: Basic commands First: Open a file- browser (Thunar) for visual aid. You start at your home directory. The symbol ~ references your home directory! Change directory: cd Type: cd /export/programs/ List the content of the directory: ls Switch back to your home directory: cd ~ cd /export/home/username

18 GUI CLI Linux directory structure at the root - / Graphical User Interface (GUI) versus command line view

19 Rela7ve and absolute paths: Absolute: /export/programs/samtools/ Rela7ve: ~ your home directory. this directory (where you currently are).. parent directory (upper directory) To see where you are at the moment: pwd command: (print working directory).

20 in the file system: Directories and Files Make a new directory: mkdir Make sure you are in your home directory Type: mkdir test Type: ls Type: cd test/ Type: ls Type: cd.. Type: ls

21 The manual: Every UNIX and Linux machine comes with built- in help for most of its commands and programs. Type man followed by the command name To leave: q For Scrolling: Arrow Up or Down for one line up or down Page Down or Ctrl- F for one page down Page Up or Ctrl- B for one page up To search A slash, then your search term, then enter. e.g. /list n to go to the next occurrence of the search term Even man itself has a manpage. Type man man into the terminal.

22 The manual: List Type: man ls Use ls alh in the current directory. - a - - all - l use a long lis@ng format - h, - - human- readable: print sizes in human readable format (e.g., 1K 234M 2G)

23 directory content Use ls alh in the current directory. permissions Owner user and group size file last accessed name blue - directory

24 Tab every single character of a command is tedious. But your shell can help you! Press the Tab key while wri@ng commands, directories or file names. Press once to complete un@l the last unambiguous character (you will hear a beep if there are no sugges@ons or you have ambiguous op@ons). Press twice to show all comple@on op@ons.

25 Tab More examples: bow<tab> -> bowtie and you hear a beep bowtie<tab><tab> :

26 Tab Examples: Commands: cuffl<tab> -> cufflinks Directories: ls /ex<tab> -> ls /export Files: cp /export/data/mol923/c<tab> -> cp /export/data/mol923/completionexample.txt

27 Useful things to know May not work with all remote clients! Use Up/Down arrow keys this will cycle through recently executed commands. Use the TAB key this will oten present you with a list of choices ater typing a part of a command (more on this in a moment) history command: list all recently used commands history - list all remembered commands history more - list all remembered commands page by page history grep i example (list all remembered commands containing string example, case- insensi@ve)

28 Working with files Types of files There are many types of files. Here are the most important: Text files (human- readable; can be viewed and modified using a text editor) Text documents (e.g., README files) Data in text format (e.g., FASTA, FASTQ, ) Scripts: Shell scripts (*.sh) Perl scripts (*.pl) Python scripts (*.py) and many others Binary files (not human- readable; cannot be viewed using a text editor) Executables (e.g., samtools, blastn, bow@e) Data in binary format (e.g, BAM files, index files for Bow@e, formajed BLAST databases) Compressed files - archives (*.gz, *.zip, *.bz2) oten text files re- formajed to save space on disk or packaged directory trees Directories are files too!

29 The file command: file [FILENAME] Tells you the type and of a file Examples: [amgrecu@cbt01 UNIX_examples]$ file example.txt example.txt: UTF-8 Unicode English text, with very long lines [amgrecu@cbt01 UNIX_examples]$ file read read: ASCII text [amgrecu@cbt01 UNIX_examples]$ file first-script.sh first-script.sh: Bourne-Again shell script text executable [amgrecu@cbt01 UNIX_examples]$ file blastn-example.fasta blastn-example.fasta: ASCII text [amgrecu@cbt01 velvet]$ file velvetg velvetg: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux , not stripped

30 The touch command: touch [FILENAME] Update the access and modifica7on 7mes of a file. Creates a new file if the specified file does not exist. Enter your test directory. Type: touch testfile List the content of the directory.

31 The mv command: Move! mv [target] [destination] Moves a file to a new loca@on. Also renames the file if desired. Create a new directory test2 Type: mv testfile test2/ Enter test2 and list the content of the directory. Type: mv testfile mytestfile List again

32 The cp command: Copy! cp [target] [destination] Creates an exact copy of a target file. Type: cp mytestfile.. Switch to the parent directory and list the content.

33 The rm command: Remove! rm [Filename] Removes the file. Once and for all! Be especially carful with the rm command Type: rm mytestfile

34 Removing directories rmdir /export/home/amgrecu/new_dir - Remove directory called new_dir in /export/home/amgrecu will fail if the directory is not empty rm rf /export/home/amgrecu/new_dir - Remove directory called new_dir in /export/home/amgrecu with all its content (i.e. all files and subdirectories will be gone) rm rf new_dir - Remove directory called new_dir in current directory with all its content (i.e. all files and subdirectories will be gone) rm rf is a very dangerous command! Once you have deleted something it is GONE! There is no Recycle Bin! There are no backups! Check carefully what you are dele7ng!

35 Naming Files Best Names should be meaningful, e.g. sample19_blastn_out.fasta, NOT blastn.fasta Names are case- sensi7ve - MyFile, myfile, myfile are all different Use only le:ers (upper- and lower- case), numbers from 0 to 9, a dot (.), and an underscore (_) e.g this_is_an_example.ext, NOT this &ismy example^.ext Avoid other characters! They may have special meaning in Linux No spaces in filenames! While working in command line terminal you always explicitly specify a program which is supposed to open this file.

36 File permissions: The chmod command (change modifier) is used to change file permissions: 4= read 2= write 1= execute Myself, Group, World chmod 700 filename 7 = read + write + execute.

37 programs: Explicit: /usr/local/bin/samtools From directory: cd /usr/local/bin/./samtools From environment (Path variable!): samtools

38 Working with files - compression To save disk space, we can compress large files if we do not intend to use them for a while. A lot of files downloaded from the web are compressed and usually need to be uncompressed before processing can take place. Compression works best on text files, such as FASTA or FASTQ. Common compressed formats: gzip (gz) gzip my_file - compresses file my_file, producing my_file.gz gzip d my_file.gz - decompress my_file.gz, producing my_file bzip2 bzip2 my_file - compresses to my_file.bz2 bzip2 my_file.bz2 decompress to my_file

39 Working with files - compression zip zip my_file.zip my_file1 my_file2 my_file3 - create a compressed archive called my_files.zip, containing three files: my_file1, my_file2, my_file3 unzip my_files.zip - decompress the archive into the cons@tuent files and directories tar tar -cvf my_file.tar my_file1 my_file2 my_dir - create a compressed archive called my_files.tar, containing files my_file1, my_file2 and the directory my_dir with all its content tar -xvf my_files.tar - decompress the archive into the cons@tuent files and directories Help for compression tools: gzip - h bzip2 - - help zip - - help tar - - help

40 text files User- friendly (GUI): gedit Kate More advanced nano You can call them from CLI: nano filename gedit filename kate filename In the course we will generally be using gedit

41 Working with text files more example.txt - display the content of the file README.txt in the current directory dividing the file into pages; press SPACE bar to go to the next page head -5 big.fasta - display first 5 lines of the file big.fasta in the current directory tail -5 big.fasta - display last 5 lines of the file big.fasta in the current directory head big.fasta tail display lines 901 through 1000 of the file big.fasta Note the character - it pipes the output from one command as input to another.

42 Working with text files Looking for a string in a text file: grep Error: lane error.log - display all lines of the file error.log in the current directory which contain the string Error: lane Looking for a string in a group of text files: grep Error: lane *.out - display all files *.out in the current directory which contain the string Error: lane ; also display the lines containing that string

43 Where to find help No.2? Not every command has a man page. However most commands have built- in help output: cp --help blastn -help cufflinks -h It s usually one of - h, - - help, - help or similar You can pipe the output into a pager (use q to leave; up/ down arrow Ctrl- F/Ctrl- B to navigate) cp --help less

44 Where to find help No.3? Also, there is plenty of help on the Internet hjp:// Most problems can be solved by Googling. Basically, just type all related words into the search field. hjp://stackoverflow.com/ Ask specific here. General are usually already answered. You can find them through the search field, but usually they re the top hits on Google anyway. hjps:// Stackoverflow for Biologists hjp://seqanswers.com/ Forum about tools and omics data processing tools

45 UNIX Tutorials hjp://people.ischool.berkeley.edu/~kevin/unix- tutorial/ toc.html hjp:// unixintro.html

46 Gezng files in and out of my home computer Filezilla or WinSCP hjps://filezilla- project.org/ Connect to cbt01.cbt.tugraz.at Use Port 22

47

48 from Home to the UNIX machines at TUGraz ThinLinc Client hjps:// Connect to cbt01.cbt.tugraz.at We have 50 licenses, if you aren t allowed in, please try again later.

49 Book Only available as an ebook: The Linux Command Line A Complete Introduc7on By William E. Shojs Jr. hjp://shop.oreilly.com/product/ do# An alterna@ve: Learning the bash Shell, 3rd Edi7on Unix Shell Programming By Cameron Newham

50 Your assignment for today Download Filezilla and ThinLinc client and install them on your home computer. Connect to cbt01.cbt.tugraz.at and play through a UNIX tutorial some more.

51 Emergency Assistance Max Malek