Essential Skills for Bioinformatics: Unix/Linux
|
|
- August Mills
- 5 years ago
- Views:
Transcription
1 Essential Skills for Bioinformatics: Unix/Linux
2 SHELL SCRIPTING
3 Overview Bash, the shell we have used interactively in this course, is a full-fledged scripting language. Unlike Python, Bash is not a general-purpose language. Bash is explicitly designed to make running and interfacing command-line programs as simple as possible. For these reason, Bash often takes the role as the glue language of bioinformatics, as it s used to glue many commands together into a cohesive workflow.
4 Overview Note that Python is a more suitable language for commonly reused or advanced pipelines. Python is a more modern, fully featured scripting language than Bash. Compared to Python, Bash lacks several nice features useful for data-processing scripts: better numeric type support, useful data structures, better string processing, refined option parsing, availability of a large number of libraries, and powerful functions that help with structuring your programs. However, there s more overhead when calling command-line programs from a Python script compared to Bash. Bash is often the best and quickest glue solution.
5 Writing and running bash scripts Most Bash scripts in bioinformatics are simply commands organized into a re-runnable script with some features to check that files exist and ensuring any error causes the script to abort. We will learn the basics of writing and executing Bash scripts, paying particular attention to how create robust Bash scripts.
6 A robust Bash header By convention, Bash scripts have the extension.sh. You can create them in your favorite text editor (e.g. emacs or vi). Anytime you write a Bash script, you should use the following Bash script header, which sets some Bash options that lead to more robust scripts. #!/bin/bash set e set u set o pipefail
7 A robust Bash header #!/bin/bash This is called the shebang, and it indicates the path to the interpreter used to execute this script. set e By default, a shell script containing a command that fails will not cause the entire shell script to exit: the shell script will just continue on to the next line. We always want errors to be loud and noticeable. This option prevents this, by terminating the script if any command exited with a nonzero exit status.
8 A robust Bash header Note that this option ignores nonzero statuses in if conditionals. Also, it ignores all exit statuses in Unix pipes except the last one. set u This option fixes another default behavior of Bash scripts: any command containing a reference to an unset variable name will still run. It prevents this type of error by aborting the script if a variable s value is unset
9 A robust Bash header set o pipefail set e will cause a script to abort if a nonzero exit status is encountered, with some exceptions. One such exception is if a program runs in a Unix pipe exited unsuccessfully. Including set o pipefail will prevent this undesirable behavior: any program that returns a nonzero exit status in the pipe will cause the entire pipe to return a nonzero status. With set e enabled, this will lead the script to abort.
10 Running bash scripts Running Bash scripts can be done one of two ways: 1. bash script.sh 2../script.sh While we can run any script, calling the script as an executable requires that it has executable permissions. We can set these using: chmod u+x script.sh This adds executable permissions for the user who owns the file. Then, the script can be run with./script.sh.
11 Variables Processing pipelines having numerous settings that should be stored in variables. Storing these settings in a variable defined at the top of the file makes adjusting settings and rerunning your pipelines much easier. Rather than having to changes numerous hardcoded values in your scripts, using variables to store settings means you only have to change one value. Bash also reads command-line arguments into variables.
12 Variables Bash s variables don t have data types. It s helpful to think of Bash s variables as strings. We can create a variable and assign it a value with. results_dir= results/ Note that spaces matter when setting Bash variables. Do not use spaces around the equal sign.
13 Variables To access a variable s value, we use a dollar sign in front of the variable s name. Suppose we want to create a directory for a sample s alignment data, called <sample>_aln/, where <sample> is replaced by the sample s name. sample= CNTRL01A mkdir ${sample}_aln/
14 Command-line arguments The variable $0 stores the name of the script, and commandline arguments are assigned to the value $1, $2, $3, etc. Bash assigns the number of command-line arguments to $#.
15 Command-line arguments If you find your script requires numerous or complicated options, it might be easier to use Python instead of Bash. Python s argparse module is much easier to use. Variables created in your Bash script will only be available for the duration of the Bash process running that script.
16 if statement Bash supports the standard if conditional statement. The basic syntax is: if [commands] then [if-statements] else [else-statements] fi
17 if statement A command s exit status provides the true and false. Remember that 0 represents true/success and anything else if false/failure. if [commands] [commands] could be any command, set of commands, pipeline, or test condition. If the exit status of these commands is 0, execution continues to the block after then. Otherwise execution continues to the block after else.
18 if statement [if-statements] is a placeholder for all statements executed if [commands] evaluates to true (0). [else-statements] is a placeholder for all statements executed if [commands] evaluates to false. The else block is optional.
19 if statement Bash is primarily designed to stitch together other commands. This is an advantage Bash has over Python when writing pipelines. Bash allows your scripts to directly work with command-line programs without requiring any overhead to call programs. Although it can be unpleasant to write complicated programs in Bash, writing simple programs is exceedingly easy because Unix tools and Bash harmonize well.
20 if statement Suppose we wanted to run a set of commands only if a file contains a certain string. Because grep returns 0 only if it matches a pattern in a file and 1 otherwise. The redirection is to tidy the output of this script such that grep s output is redirected to /dev/null and not to the script s standard out.
21 test Like other programs, test exits with either 0 or 1. However test s exit status indicates the return value of the test specified through its arguments, rather than exit success or error. test supports numerous standard comparison operators.
22 test String/integer Description -z str String str is null str1 = str2 str1 and str2 are identical str1!= str2 str1 and str2 are different int1 eq int2 Integers int1 and int2 are equal int1 ne int2 int1 and int2 are not equal int1 lt int2 int1 is less than int2 int1 gt int2 int1 is greater than int2 int1 le int2 int1 is less than or equal to int2 int1 ge int2 int1 is greater than or equal to int2
23 test In practice, the most common conditions you ll be checking are to see if files or directories exist and whether you can write to them. test supports numerous file- and directoryrelated test operations.
24 test File/directory expression Description -d dir dir is a directory -f file file is a file -e file file exists -r file file is readable -w file file is writable -x file file is executable
25 test Combining test with if statements is simple: if test f some_file.txt then [ ] fi Bash provides a simpler syntactic alternative: if [ f some_file.txt ] then [ ] fi Note the spaces around and within the brackets: these are required.
26 test When using this syntax, we can chain test expression with a as logical AND, -o as logical OR,! as negation. Our familiar && and operators won t work in test, because these are shell operators. if [ $# ne 1 o! r $1 ] then fi echo usage: script.sh file_in.txt
27 for loop In bioinformatics, most of our data is split across multiple files. At the heart of any processing pipeline is some way to apply the same workflow to each of these files, taking care to keep track of sample names. Looping over files with Bash s for loop is the simplest way to accomplish this. There are three essential parts to creating a pipeline to process a set of files: 1. Selecting which files to apply the commands to 2. Looping over the data and applying the commands 3. Keeping track of the names of any output files created
28 for loop Suppose we have a file called samples.txt that tells you basic information about your raw data: sample name, read pair, and where the file is.
29 for loop Suppose we want to loop over every file, gather quality statistics on each and every file, and save this information to an output file. First, we load our filenames into a Bash array, which we can then loop over. Bash arrays can be created manually using:
30 for loop But creating Bash arrays by hand is tedious and error prone. The beauty of Bash is that we can use a command substitution to construct Bash arrays. We can strip the path and extension from each filename using basename.
31 for loop
32 Learning Unix
33 Sequence Alignments
34 DATA FORMATS
35 Overview Nucleotide (and protein) sequences are stored in two plaintext formats widespread in bioinformatics: FASTA and FASTQ. We will discuss each format and their limitations, and then see some tools for working with data in these formats.
36 FASTA The FASTA format originates from the FASTA alignment suites, created by William R. Pearson and David J. Lipman. The FASTA format is used to store any sort of sequence data not requiring per-base pair quality scores. This includes: reference genome files, protein sequences, coding DNA sequences (CDS), transcript sequences, and so on.
37 FASTA FASTA files are composed of sequence entries, each containing two parts: a description and the sequence data. The description line begins with a greater than symbol (>) and contains the sequence identifier and other optional information The sequence data begins on the next line after the description, and continues until there s another description line.
38 FASTA An example FASTA file: The FASTA format s simplicity and flexibility comes with an unfortunate downside: the FASTA format is a loosely defined ad hoc format.
39 FASTA In general, the following rules should be observed: 1. Sequence lines should not be too long. While a FASTA file that contains the sequence of the entire human chromosome 1 on a single line is a valid FASTA file, most tools that run on such a file would fail. 2. Some tools may accept data containing alphabets beyond those that they know how to deal with. For example, the standard alphabet for nucleotides would contain ATGC. An extended alphabet may also contain 1) N: A, T, G, or C 2) W: A or T 3) Search the web for IUPAC nucleotides to get a list of all such symbols.
40 FASTA 3. The sequence lines should always at the same width with the exception of the last line. Some tools will fail to operate correctly and may not even warn the users if this condition is not satisfied. The following is technically a valid FASTA but it may cause various problems: It should be reformatted to:
41 FASTA 4. Use upper-case letters. Whereas both lower-case and upper-case letters are allowed by the specification, the different capitalization may carry additional meaning and some tools and methods will operate differently when encoutering upper- or lower-case letters. Some communities (e.g. Ensembl) chose to designate the lowercase letters as all repeats and low complexity regions.
42 FASTQ The FASTQ format extends FASTA by including a numeric quality score to each base in the sequence. The FASTQ format is widely used to store high-throughput sequencing data, which is reported with a per-base quality score indicating the confidence of each base call. It is the de facto standard by which all sequencing instruments represent data.
43 FASTQ The FASTQ format looks like: Line1: The description line beginning This contains the record identifier and other information. Line2: Sequence data, which can be on one or many lines. Line3: The line beginning with + indicates the end of the sequence. Line4: Quality data, which can also be on one or many lines, but must be the same length as the sequence. Each numeric base quality is encoded with ASCII characters.
44 FASTQ The FASTQ format is a multi-line format just as the FASTA format is. In the early days of high-throughput sequencing, instruments always produced the entire FASTQ sequence on a single line. The FASTQ format suffers from the unexpected flaw that sign is both a FASTQ record separator and a valid value of the quality string. For that reason it is a little more difficult to design a correct FASTQ parsing program.
Lecture 5. Essential skills for bioinformatics: Unix/Linux
Lecture 5 Essential skills for bioinformatics: Unix/Linux UNIX DATA TOOLS Text processing with awk We have illustrated two ways awk can come in handy: Filtering data using rules that can combine regular
More informationEssentials for Scientific Computing: Bash Shell Scripting Day 3
Essentials for Scientific Computing: Bash Shell Scripting Day 3 Ershaad Ahamed TUE-CMS, JNCASR May 2012 1 Introduction In the previous sessions, you have been using basic commands in the shell. The bash
More informationBash scripting basics
Bash scripting basics prepared by Anatoliy Antonov for ESSReS community September 2012 1 Outline Definitions Foundations Flow control References and exercises 2 Definitions 3 Definitions Script - [small]
More informationCOMP 4/6262: Programming UNIX
COMP 4/6262: Programming UNIX Lecture 12 shells, shell programming: passing arguments, if, debug March 13, 2006 Outline shells shell programming passing arguments (KW Ch.7) exit status if (KW Ch.8) test
More information9.2 Linux Essentials Exam Objectives
9.2 Linux Essentials Exam Objectives This chapter will cover the topics for the following Linux Essentials exam objectives: Topic 3: The Power of the Command Line (weight: 10) 3.3: Turning Commands into
More informationShell Programming (bash)
Shell Programming Shell Programming (bash) Commands run from a file in a subshell A great way to automate a repeated sequence of commands. File starts with #!/bin/bash absolute path to the shell program
More informationOPERATING SYSTEMS LAB LAB # 6. I/O Redirection and Shell Programming. Shell Programming( I/O Redirection and if-else Statement)
P a g e 1 OPERATING SYSTEMS LAB LAB 6 I/O Redirection and Shell Programming Lab 6 Shell Programming( I/O Redirection and if-else Statement) P a g e 2 Redirection of Standard output/input i.e. Input - Output
More informationLinux Shell Scripting. Linux System Administration COMP2018 Summer 2017
Linux Shell Scripting Linux System Administration COMP2018 Summer 2017 What is Scripting? Commands can be given to a computer by entering them into a command interpreter program, commonly called a shell
More informationShell scripting and system variables. HORT Lecture 5 Instructor: Kranthi Varala
Shell scripting and system variables HORT 59000 Lecture 5 Instructor: Kranthi Varala Text editors Programs built to assist creation and manipulation of text files, typically scripts. nano : easy-to-learn,
More informationCSCI 211 UNIX Lab. Shell Programming. Dr. Jiang Li. Jiang Li, Ph.D. Department of Computer Science
CSCI 211 UNIX Lab Shell Programming Dr. Jiang Li Why Shell Scripting Saves a lot of typing A shell script can run many commands at once A shell script can repeatedly run commands Help avoid mistakes Once
More informationCS Unix Tools & Scripting
Cornell University, Spring 2014 1 February 24, 2014 1 Slides evolved from previous versions by Hussam Abu-Libdeh and David Slater A note on awk for (item in array) The order in which items are returned
More informationCommand Interpreters. command-line (e.g. Unix shell) On Unix/Linux, bash has become defacto standard shell.
Command Interpreters A command interpreter is a program that executes other programs. Aim: allow users to execute the commands provided on a computer system. Command interpreters come in two flavours:
More informationShell Start-up and Configuration Files
ULI101 Week 10 Lesson Overview Shell Start-up and Configuration Files Shell History Alias Statement Shell Variables Introduction to Shell Scripting Positional Parameters echo and read Commands if and test
More informationbash Execution Control COMP2101 Winter 2019
bash Execution Control COMP2101 Winter 2019 Bash Execution Control Scripts commonly can evaluate situations and make simple decisions about actions to take Simple evaluations and actions can be accomplished
More informationScripting. More Shell Scripts. Adapted from Practical Unix and Programming Hunter College
Scripting More Shell Scripts Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss Back to shell scripts Now that you've learned a few commands and can edit files,
More informationContents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
More informationA shell can be used in one of two ways:
Shell Scripting 1 A shell can be used in one of two ways: A command interpreter, used interactively A programming language, to write shell scripts (your own custom commands) 2 If we have a set of commands
More informationShells & Shell Programming (Part B)
Shells & Shell Programming (Part B) Software Tools EECS2031 Winter 2018 Manos Papagelis Thanks to Karen Reid and Alan J Rosenthal for material in these slides CONTROL STATEMENTS 2 Control Statements Conditional
More informationCSC 2500: Unix Lab Fall 2016
CSC 2500: Unix Lab Fall 2016 Control Statements in Shell Scripts: Decision Mohammad Ashiqur Rahman Department of Computer Science College of Engineering Tennessee Tech University Agenda User Input Special
More informationShells and Shell Programming
Shells and Shell Programming 1 Shells A shell is a command line interpreter that is the interface between the user and the OS. The shell: analyzes each command determines what actions are to be performed
More informationA Big Step. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers
A Big Step Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Copyright 2006 2009 Stewart Weiss What a shell really does Here is the scoop on shells. A shell is a program
More informationShell programming. Introduction to Operating Systems
Shell programming Introduction to Operating Systems Environment variables Predened variables $* all parameters $# number of parameters $? result of last command $$ process identier $i parameter number
More informationCOMP 2718: Shell Scripts: Part 1. By: Dr. Andrew Vardy
COMP 2718: Shell Scripts: Part 1 By: Dr. Andrew Vardy Outline Shell Scripts: Part 1 Hello World Shebang! Example Project Introducing Variables Variable Names Variable Facts Arguments Exit Status Branching:
More informationShells and Shell Programming
Shells and Shell Programming Shells A shell is a command line interpreter that is the interface between the user and the OS. The shell: analyzes each command determines what actions are to be performed
More informationScripting. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers
Scripting Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss What a shell
More informationLING 408/508: Computational Techniques for Linguists. Lecture 5
LING 408/508: Computational Techniques for Linguists Lecture 5 Last Time Installing Ubuntu 18.04 LTS on top of VirtualBox Your Homework 2: did everyone succeed? Ubuntu VirtualBox Host OS: MacOS or Windows
More informationBash scripting Tutorial. Hello World Bash Shell Script. Super User Programming & Scripting 22 March 2013
Bash scripting Tutorial Super User Programming & Scripting 22 March 2013 Hello World Bash Shell Script First you need to find out where is your bash interpreter located. Enter the following into your command
More informationSHELL SCRIPT BASIC. UNIX Programming 2014 Fall by Euiseong Seo
SHELL SCRIPT BASIC UNIX Programming 2014 Fall by Euiseong Seo Shell Script Interactive shell sequentially executes a series of commands Some tasks are repetitive and automatable They are what programs
More informationLecture 3. Essential skills for bioinformatics: Unix/Linux
Lecture 3 Essential skills for bioinformatics: Unix/Linux RETRIEVING DATA Overview Whether downloading large sequencing datasets or accessing a web application hundreds of times to download specific files,
More informationexample: name1=jan name2=mike export name1 In this example, name1 is an environmental variable while name2 is a local variable.
Bourne Shell Programming Variables - creating and assigning variables Bourne shell use the set and unset to create and assign values to variables or typing the variable name, an equal sign and the value
More informationControl Structures. CIS 118 Intro to LINUX
Control Structures CIS 118 Intro to LINUX Basic Control Structures TEST The test utility, has many formats for evaluating expressions. For example, when given three arguments, will return the value true
More informationUNIX shell scripting
UNIX shell scripting EECS 2031 Summer 2014 Przemyslaw Pawluk June 17, 2014 What we will discuss today Introduction Control Structures User Input Homework Table of Contents Introduction Control Structures
More informationComputer Systems and Architecture
Computer Systems and Architecture Stephen Pauwels UNIX Scripting Academic Year 2018-2019 Outline Basics Conditionals Loops Advanced Exercises Shell Scripts Grouping commands into a single file Reusability
More informationShell Scripting. Todd Kelley CST8207 Todd Kelley 1
Shell Scripting Todd Kelley kelleyt@algonquincollege.com CST8207 Todd Kelley 1 If we have a set of commands that we want to run on a regular basis, we could write a script A script acts as a Linux command,
More informationSHELL SCRIPT BASIC. UNIX Programming 2015 Fall by Euiseong Seo
SHELL SCRIPT BASIC UNIX Programming 2015 Fall by Euiseong Seo Shell Script! Interactive shell sequentially executes a series of commands! Some tasks are repetitive and automatable! They are what programs
More informationLinux Bash Shell Scripting
University of Chicago Initiative in Biomedical Informatics Computation Institute Linux Bash Shell Scripting Present by: Mohammad Reza Gerami gerami@ipm.ir Day 2 Outline Support Review of Day 1 exercise
More informationUnix Shell scripting. Dr Alun Moon 7th October Introduction. Notation. Spaces
Unix Shell scripting Dr Alun Moon 7th October 2017 Introduction Shell scripts in Unix are a very powerfull tool, they form much of the standard system as installed. What are these good for? So many file
More informationComputer Systems and Architecture
Computer Systems and Architecture UNIX Scripting Bart Meyers University of Antwerp August 29, 2012 Outline Basics Conditionals Loops Advanced Exercises Shell scripts Grouping commands into a single file
More informationLinux shell & shell scripting - II
IBS 574 - Computational Biology & Bioinformatics Spring 2018, Tuesday (02/01), 2:00-4:00PM Linux shell & shell scripting - II Ashok R. Dinasarapu Ph.D Scientist, Bioinformatics Dept. of Human Genetics,
More informationProgramming Languages and Uses in Bioinformatics
Programming in Perl Programming Languages and Uses in Bioinformatics Perl, Python Pros: reformatting data files reading, writing and parsing files building web pages and database access building work flow
More informationbash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017
bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017 Command Lists A command is a sequence of commands separated by the operators ; & && and ; is used to simply execute commands in
More informationGrep and Shell Programming
Grep and Shell Programming Comp-206 : Introduction to Software Systems Lecture 7 Alexandre Denault Computer Science McGill University Fall 2006 Teacher's Assistants Michael Hawker Monday, 14h30 to 16h30
More informationShell Script Example. Here is a hello world shell script: $ ls -l -rwxr-xr-x 1 horner 48 Feb 19 11:50 hello* $ cat hello #!/bin/sh
Shell Programming Shells A shell can be used in one of two ways: A command interpreter, used interactively A programming language, to write shell scripts (your own custom commands) Shell Scripts A shell
More informationhttp://xkcd.com/208/ 1. Review of pipes 2. Regular expressions 3. sed 4. Editing Files 5. Shell loops 6. Shell scripts cat seqs.fa >0! TGCAGGTATATCTATTAGCAGGTTTAATTTTGCCTGCACTTGGTTGGGTACATTATTTTAAGTGTATTTGACAAG!
More informationAssignment 3, Due October 4
Assignment 3, Due October 4 1 Summary This assignment gives you practice with writing shell scripts. Shell scripting is also known as bash programming. Your shell is bash, and when you write a shell script
More informationVi & Shell Scripting
Vi & Shell Scripting Comp-206 : Introduction to Week 3 Joseph Vybihal Computer Science McGill University Announcements Sina Meraji's office hours Trottier 3rd floor open area Tuesday 1:30 2:30 PM Thursday
More informationsottotitolo A.A. 2016/17 Federico Reghenzani, Alessandro Barenghi
Titolo presentazione Piattaforme Software per la Rete sottotitolo BASH Scripting Milano, XX mese 20XX A.A. 2016/17, Alessandro Barenghi Outline 1) Introduction to BASH 2) Helper commands 3) Control Flow
More informationTitle:[ Variables Comparison Operators If Else Statements ]
[Color Codes] Environmental Variables: PATH What is path? PATH=$PATH:/MyFolder/YourStuff?Scripts ENV HOME PWD SHELL PS1 EDITOR Showing default text editor #!/bin/bash a=375 hello=$a #No space permitted
More informationShell script. Shell Scripts. A shell script contains a sequence of commands in a text file. Shell is an command language interpreter.
Shell Scripts A shell script contains a sequence of commands in a text file. Shell is an command language interpreter. Shell executes commands read from a file. Shell is a powerful programming available
More informationПрограмиранев UNIX среда
Програмиранев UNIX среда Използванена команден шел и създаванена скриптове: tcsh, bash, awk, python Shell programming As well as using the shell to run commands you can use its built-in programming language
More informationBourne Shell Reference
> Linux Reviews > Beginners: Learn Linux > Bourne Shell Reference Bourne Shell Reference found at Br. David Carlson, O.S.B. pages, cis.stvincent.edu/carlsond/cs330/unix/bshellref - Converted to txt2tags
More informationAdvanced Unix Programming Module 03 Raju Alluri spurthi.com
Advanced Unix Programming Module 03 Raju Alluri askraju @ spurthi.com Advanced Unix Programming: Module 3 Shells & Shell Programming Environment Variables Writing Simple Shell Programs (shell scripts)
More informationWindshield. Language Reference Manual. Columbia University COMS W4115 Programming Languages and Translators Spring Prof. Stephen A.
Windshield Language Reference Manual Columbia University COMS W4115 Programming Languages and Translators Spring 2007 Prof. Stephen A. Edwards Team members Wei-Yun Ma wm2174 wm2174@columbia.edu Tony Wang
More informationScripting. More Shell Scripts Loops. Adapted from Practical Unix and Programming Hunter College
Scripting More Shell Scripts Loops Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss Loops: for The for script version 1: 1 #!/bin/bash 2 for i in 1 2 3 3 do
More informationEECS 470 Lab 5. Linux Shell Scripting. Friday, 1 st February, 2018
EECS 470 Lab 5 Linux Shell Scripting Department of Electrical Engineering and Computer Science College of Engineering University of Michigan Friday, 1 st February, 2018 (University of Michigan) Lab 5:
More informationChapter 4. Unix Tutorial. Unix Shell
Chapter 4 Unix Tutorial Users and applications interact with hardware through an operating system (OS). Unix is a very basic operating system in that it has just the essentials. Many operating systems,
More informationhttp://xkcd.com/208/ 1. Review of pipes 2. Regular expressions 3. sed 4. awk 5. Editing Files 6. Shell loops 7. Shell scripts cat seqs.fa >0! TGCAGGTATATCTATTAGCAGGTTTAATTTTGCCTGCACTTGGTTGGGTACATTATTTTAAGTGTATTTGACAAG!
More informationCOMS 6100 Class Notes 3
COMS 6100 Class Notes 3 Daniel Solus September 1, 2016 1 General Remarks The class was split into two main sections. We finished our introduction to Linux commands by reviewing Linux commands I and II
More informationSTP, Unix, SAC tutorial, Ge167 Winter 2014
STP, Unix, SAC tutorial, Ge167 Winter 2014 Asaf Inbal 1 Downloading waveforms In this tutorial we ll learn how to download waveforms using a tool called STP (Seismic Transfer Program) and manipulate them
More informationSimple Shell Scripting for Scientists
Simple Shell Scripting for Scientists Day Three Julian King Bruce Beckles University of Cambridge Computing Service 1 Introduction Who:! Julian King, Unix Support, UCS! Bruce Beckles, e-science Specialist,
More informationIntroduction: What is Unix?
Introduction Introduction: What is Unix? An operating system Developed at AT&T Bell Labs in the 1960 s Command Line Interpreter GUIs (Window systems) are now available Introduction: Unix vs. Linux Unix
More informationUseful Unix Commands Cheat Sheet
Useful Unix Commands Cheat Sheet The Chinese University of Hong Kong SIGSC Training (Fall 2016) FILE AND DIRECTORY pwd Return path to current directory. ls List directories and files here. ls dir List
More informationBashed One Too Many Times. Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009
Bashed One Too Many Times Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009 What is a Shell? The shell interprets commands and executes them It provides you with an environment
More informationBasic Linux (Bash) Commands
Basic Linux (Bash) Commands Hint: Run commands in the emacs shell (emacs -nw, then M-x shell) instead of the terminal. It eases searching for and revising commands and navigating and copying-and-pasting
More informationWhen talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used:
Linux Tutorial How to read the examples When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used: $ application file.txt
More informationUNIX System Programming Lecture 3: BASH Programming
UNIX System Programming Outline Filesystems Redirection Shell Programming Reference BLP: Chapter 2 BFAQ: Bash FAQ BMAN: Bash man page BPRI: Bash Programming Introduction BABS: Advanced Bash Scripting Guide
More informationCS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND
CS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND Prof. Michael J. Reale Fall 2014 Finding Files in a Directory Tree Suppose you want to find a file with a certain filename (or with a filename matching
More informationLecture 8. Sequence alignments
Lecture 8 Sequence alignments DATA FORMATS bioawk bioawk is a program that extends awk s powerful processing of tabular data to processing tasks involving common bioinformatics formats like FASTA/FASTQ,
More informationLinux shell programming for Raspberry Pi Users - 2
Linux shell programming for Raspberry Pi Users - 2 Sarwan Singh Assistant Director(S) NIELIT Chandigarh 1 SarwanSingh.com Education is the kindling of a flame, not the filling of a vessel. - Socrates SHELL
More information1. Hello World Bash Shell Script. Last Updated on Wednesday, 13 April :03
1 of 18 21/10/2554 9:39 Bash scripting Tutorial tar -czf myhome_directory.tar.gz /home/linuxcong Last Updated on Wednesday, 13 April 2011 08:03 Article Index 1. Hello World Bash Shell Script 2. Simple
More informationbash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017
bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017 Command Lists A command is a sequence of commands separated by the operators ; & && and ; is used to simply execute commands in
More informationIntroduction Variables Helper commands Control Flow Constructs Basic Plumbing. Bash Scripting. Alessandro Barenghi
Bash Scripting Alessandro Barenghi Dipartimento di Elettronica, Informazione e Bioingegneria Politecnico di Milano alessandro.barenghi - at - polimi.it April 28, 2015 Introduction The bash command shell
More informationCisco IOS Shell. Finding Feature Information. Prerequisites for Cisco IOS.sh. Last Updated: December 14, 2012
Cisco IOS Shell Last Updated: December 14, 2012 The Cisco IOS Shell (IOS.sh) feature provides shell scripting capability to the Cisco IOS command-lineinterface (CLI) environment. Cisco IOS.sh enhances
More informationAssignment clarifications
Assignment clarifications How many errors to print? at most 1 per token. Interpretation of white space in { } treat as a valid extension, involving white space characters. Assignment FAQs have been updated.
More informationB a s h s c r i p t i n g
8 Bash Scripting Any self-respecting hacker must be able to write scripts. For that matter, any selfrespecting Linux administrator must be able to script. Hackers often need to automate commands, sometimes
More informationIntroduction to Linux Basics Part II. Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala
Introduction to Linux Basics Part II 1 Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu 2 Variables in Shell HOW DOES LINUX WORK? Shell Arithmetic I/O and
More informationCSNB113: System Administration - 8 th Topic: Shell Scripts Programming I
CSNB113: System Administration - 8 th Topic: Shell Scripts Programming I The prompt is a prompt is a prompt and a program input It is possible to write and run a program directly on the command prompt!
More informationGenomic Files. University of Massachusetts Medical School. October, 2015
.. Genomic Files University of Massachusetts Medical School October, 2015 2 / 55. A Typical Deep-Sequencing Workflow Samples Fastq Files Fastq Files Sam / Bam Files Various files Deep Sequencing Further
More informationThere are some string operators that can be used in the test statement to perform string comparison.
ACS-294- Unix (Winter Term, 26-7) Part II: Shell Programming Page 3 The test also returns a nonzero exit value if there is no argument: test String Operators There are some string operators that can be
More informationIntroduction to Shell Scripting
Introduction to Shell Scripting Lecture 1. Shell scripts are small programs. They let you automate multi-step processes, and give you the capability to use decision-making logic and repetitive loops. 2.
More informationBash command shell language interpreter
Principles of Programming Languages Bash command shell language interpreter Advanced seminar topic Louis Sugy & Baptiste Thémine Presentation on December 8th, 2017 Table of contents I. General information
More information12.1 UNDERSTANDING UNIX SHELL PROGRAMMING LANGUAGE: AN INTRODUCTION Writing a Simple Script Executing a Script
12 Shell Programming This chapter concentrates on shell programming. It explains the capabilities of the shell as an interpretive high-level language. It describes shell programming constructs and particulars.
More informationAnswers to AWK problems. Shell-Programming. Future: Using loops to automate tasks. Download and Install: Python (Windows only.) R
Today s Class Answers to AWK problems Shell-Programming Using loops to automate tasks Future: Download and Install: Python (Windows only.) R Awk basics From the command line: $ awk '$1>20' filename Command
More informationIntroduction to UNIX command-line II
Introduction to UNIX command-line II Boyce Thompson Institute 2017 Prashant Hosmani Class Content Terminal file system navigation Wildcards, shortcuts and special characters File permissions Compression
More informationINTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT
INTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT GOALS BASIC What is shell scripting? What makes shell scripting useful? What should you know already GOALS PART 1 Script layout/design
More informationCSCI 2132: Software Development. Norbert Zeh. Faculty of Computer Science Dalhousie University. Shell Scripting. Winter 2019
CSCI 2132: Software Development Shell Scripting Norbert Zeh Faculty of Computer Science Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your Shell vs Your File Manager File manager
More informationGenomic Files. University of Massachusetts Medical School. October, 2014
.. Genomic Files University of Massachusetts Medical School October, 2014 2 / 39. A Typical Deep-Sequencing Workflow Samples Fastq Files Fastq Files Sam / Bam Files Various files Deep Sequencing Further
More information5/20/2007. Touring Essential Programs
Touring Essential Programs Employing fundamental utilities. Managing input and output. Using special characters in the command-line. Managing user environment. Surveying elements of a functioning system.
More informationIntroduction to Linux
Introduction to Linux The command-line interface A command-line interface (CLI) is a type of interface, that is, a way to interact with a computer. Window systems, punched cards or a bunch of dials, buttons
More informationITST Searching, Extracting & Archiving Data
ITST 1136 - Searching, Extracting & Archiving Data Name: Step 1 Sign into a Pi UN = pi PW = raspberry Step 2 - Grep - One of the most useful and versatile commands in a Linux terminal environment is the
More informationDigital Humanities. Tutorial Regular Expressions. March 10, 2014
Digital Humanities Tutorial Regular Expressions March 10, 2014 1 Introduction In this tutorial we will look at a powerful technique, called regular expressions, to search for specific patterns in corpora.
More informationCS Unix Tools & Scripting
Cornell University, Spring 2014 1 February 21, 2014 1 Slides evolved from previous versions by Hussam Abu-Libdeh and David Slater Announcement HW 4 is out. Due Friday, February 28, 2014 at 11:59PM. Wrapping
More informationMerge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p.
Preface p. xiii Ideology: Data Skills for Robust and Reproducible Bioinformatics How to Learn Bioinformatics p. 1 Why Bioinformatics? Biology's Growing Data p. 1 Learning Data Skills to Learn Bioinformatics
More informationIntroduction To. Barry Grant
Introduction To Barry Grant bjgrant@umich.edu http://thegrantlab.org Working with Unix How do we actually use Unix? Inspecting text files less - visualize a text file: use arrow keys page down/page up
More informationUnix as a Platform Exercises + Solutions. Course Code: OS 01 UNXPLAT
Unix as a Platform Exercises + Solutions Course Code: OS 01 UNXPLAT Working with Unix Most if not all of these will require some investigation in the man pages. That's the idea, to get them used to looking
More informationBash shell programming Part II Control statements
Bash shell programming Part II Control statements Deniz Savas and Michael Griffiths 2005-2011 Corporate Information and Computing Services The University of Sheffield Email M.Griffiths@sheffield.ac.uk
More informationEssential Skills for Bioinformatics: Unix/Linux
Essential Skills for Bioinformatics: Unix/Linux WORKING WITH COMPRESSED DATA Overview Data compression, the process of condensing data so that it takes up less space (on disk drives, in memory, or across
More informationLast Time. on the website
Last Time on the website Lecture 6 Shell Scripting What is a shell? The user interface to the operating system Functionality: Execute other programs Manage files Manage processes Full programming language
More informationINTRODUCTION TO UNIX
WEEK 1.3 INTRODUCTION TO UNIX Unix is a key operating system in the history of communications. Here, you will gain experience in using Unix both in command line and with a graphical interface. Most importantly,
More informationBasics. I think that the later is better.
Basics Before we take up shell scripting, let s review some of the basic features and syntax of the shell, specifically the major shells in the sh lineage. Command Editing If you like vi, put your shell
More information