Short Read Sequencing Analysis Workshop Day 2 Learning the Linux Compute Environment In-class Slides Matt Hynes-Grace Manager of IT Operations, BioFrontiers Institute
Review of Day 2 Videos Video 1 Introduction to linux Linux is a command-based OS; we connect and communicate with the server via typed/written commands ssh command (Mac) and MobaXterm(Windows) Video 2 Working on a linux system Navigate the linux file system; full and relative path designations Commands and tricks for navigating linux file system Video 3 Linux Commands for File Manipulation Learn to use additional linux commands aimed at manipulating file and directory content
Review of Day 2 Videos Video 4 Remote RSYNC, reading files Access command manuals, use options to customize command functions Command line I/O, capture STDOUT and STDERR, piping Video 5 More file manipulation Access command manuals, use options to customize command functions Command line I/O, capture STDOUT and STDERR, piping Video 6 - Permissions Exploring Unix Permissions Video 7 VIM Tutorial
Outline For Today Questions???? Brief review of command line user interface Log into the server Discuss the directories we will work with Copy files to home Execute commands from videos; introduce new commands VIM practice Create and edit README file
Help Requests bit-help@colorado.edu, Subject: SR2017 This should be used for anything that isn t working as expected on the systems or access issues on campus and off. We re going avoid giving you the answers to the homework questions
We Use the Shell to Communicate With the OS Keyboard Input Device Shell and Terminal (Sh, Bash) Enter a command Operating System (Linux) Hardware (CPU, RAM, Hard Drive) Execute the command
We Use the Shell to Communicate With the OS Keyboard Input Device Shell and Terminal (Sh, Bash) Local Machine (laptop/desktop Computer) Operating System (Linux) Hardware (CPU, RAM, Hard Drive) Remote Machine Tortuga
Understanding the Command Line Keyboard Input Device $ <command> <arguments>
How to Access A Server Via the Terminal Open a terminal session MobaXterm(PC users) Terminal App (Macs) Terminal VPN connection required for off-campus access SSH command $ ssh <username>@tortuga.colorado.edu Enter password when prompted (it will be hidden as you type)
The hostname command The hostname command shows you the current system for the terminal you are interacting with. Type: $ hostname
The Home Directory Current working directory when you login in your home directory Home directory is your personal directory Home: /mnt/scratch_nobackup/<username> pwd = print work directory usage: $ pwd cd or- cd ~ to get home
Linux Uses A Hierarchical File System Your directory: /mnt/scratch_nobackup/<username> Root Directory /mnt
Linux Uses A Hierarchical File System Image Credit: linuxacademy.com
Linux Uses A Hierarchical File System Your directory: /mnt/scratch_nobackup/<username> 2017 Workshop directory: /mnt/scratch_nobackup/sread2017/ Root Directory /mnt
Basic Linux Commands For Navigating the File System cd = change directory usage: $ cd <path> will take you to the location specified in path Type: $ cd /mnt/scratch_nobackup/sread2017/ ls = list contents of directory usage: $ ls list contents of current working directory Type: $ ls
Filesystem navigation cd = change directory without a path or with ~ usage: $ cd ~ will take you to your home directory Type: $ cd ~ ls = list contents of directory usage: $ ls list contents of current working directory Type: $ ls
Create basic organization Make a Workshop-Day-2 directory in your home usage: $ mkdir <name> Will create a new directory of <name> Type $ mkdir Workshop-Day-2 Make a Genomes directory in Workshop-Day-2 Type $ mkdir Workshop-Day-2/Genomes Directories can be created from any location as long as the full path is provided.
Add structure to your folders Change directory to your Workshop-Day-2/ List contents of your current directory mkdir = make directory usage: $ mkdir <directory> Type: $ mkdir Bowtie Make new directories: ChIP-seq RNA-seq Genomes/Bowtie2Index make a new directory
The tree command Change directory to your home directory List contents of your current directory tree= list contents of directories in a tree-like format usage: $ tree <path> show directory/file hierarchy Type: $ tree Workshop-Day-2 If your terminal shows odd characters try appending a A parameter to the command ex tree A <dir>
Break Time
Linux Commands For File Manipulation cp = copy a file or directory usage: $ cp <source> <destination> rsync = copy a directory or file remotely or locally usage: $ rsync <source> <destination> mv = move directory or file usage: $ mv <source> <destination> Can be used to rename a file or directory
The man command This will show you the options and parameters for most programs on the linux system Usage: $ man <name of program> Type: $ man rsync Type q to escape the manual
Use rsync to copy SGDv4/ to home $ rsync -av --progress /mnt/scratch_nobackup/sread2017/workshop-day2/genomes/sgdv4 /mnt/scratch_nobackup/<username>/workshop-day-2/genomes/ ^ Note the single space after the character 4 above Replace <username> with your username No slash after SGDv4 is important to preserve directory structure $ ls /mnt/scratch_nobackup/<username>/workshop-day-2/ $ ls /mnt/scratch_nobackup/<username>/workshop-day-2/genomes
Prepare Genome Files Change directory to your Genomes/SGDv4, list contents What you should see: SGDv4.chrom.tar SGDv4.gff Tar files: Tape Archive - collection of files in a single file, allows easier distribution and data backup tar = Tape Archive usage: $ tar create or extract an archive
The tar Command tar = Tape Archive usage: $ tar create or extract an archive Open the tar manual: Type: $ man tar List contents of this archive Type: $ tar -tvf SGDv4.chrom.tar Extract content of the archive Type: $ tar xf SGDv4.chrom.tar
Preparing the SGDv4 Genome List the contents of SGDv4/ Concatenate the individual chromosome fasta files into a single fasta file named SGDv4.fa cat = concatenate usage: $ cat <file1> <file2> etc. cat command prints to the screen (STDOUT) capture the output with > Type: $ cat chr01.fa chr02.fa > SGDv4.fa
What is Vim? Vim is a free, open source terminal based text editor Edit content of human-readable files Several options exist for terminal text editors Emacs Nano
Using Vim vim = Vi Improved $ vim <file> open <file> for editing in Vim In your home directory create a new file: Type: $ touch vim_example.txt Open the file with vim Type: $ vim vim_example.txt
Using Vim Vim has 2 operating modes: Command mode (default) (Vim commands only) Insert mode (editing) i/i o/o a/a insert open new line append Switch back to command mode esc
Using Vim In vim: type i you are now in insert mode Hit esc Yank/copy the line: shift + y Paste the line: p Insert Mode: i/i, o/o Command Mode: esc
Using Vim In vim: Type gg you will be taken to first line in file Insert Mode: i/i, o/o Command Mode: esc Type shift + O you have now opened a new line and are in insert mode Return to command mode Type shift + g you will be taken to the last line of the file
In vim: Using Vim Insert Mode: i/i, o/o Command Mode: esc Type o you have now opened a new line and are in insert mode Return to command mode Type dd this will delete the current line Type u to undo this change
Save and Exit Vim Insert Mode: i/i, o/o Command Mode: esc In vim: Type :w and hit enter this will save changes to the file Type :wq! and hit enter this will save changes and exit
Linux Commands For Viewing Files head usage: $ head <file> view the first 10 lines of a <file> Type: $ head vim_example.txt tail usage: $ tail <file> view the last 5 lines of a <file> Type: $ tail n 5 vim_example.txt
Linux Commands For Viewing Files more usage: $ more <file> Use Enter and spacebar to progress through file Only unidirectional movement Press q button to escape less usage: $ less <file> Use up and down arrows to navigate through file Bidirectional movement Press q button to escape
Documenting Your Work Is Important Create a README README file list information about files, archives, and directories Things to include: Brief description of file content Date and method of data generation, data generator While in your home directory Type: $ ls -l Workshop-Day-2/* > README
Edit Your README File In vim Type: $ vim README Start editing/annotating your README Bowtie/ FASTQ/ Genomes/ Bowtie2Index Insert Mode: i/i, o/o Command Mode: esc For example: Bowtie/ # This directory will contain all the output from Bowtie Save changes and exit (:wq!)
The End Questions?? Don t forget the homework. Help session in A104 1-3PM Today Watch videos for Day 3
Acknowledgements Workshop Coordinators: Mary Allen, Amber Scott, Meghan Sloan Funding: BioFrontiers Institute and Colorado Office of Economic Development and International Trade Additional Acknowledgments Compute Resources: BioFrontiers IT Staff Robin Dowell and Dowell Lab 2016