Essential Skills for Bioinformatics: Unix/Linux

Size: px
Start display at page:

Download "Essential Skills for Bioinformatics: Unix/Linux"

Transcription

1 Essential Skills for Bioinformatics: Unix/Linux

2 SHELL SCRIPTING

3 Overview Bash, the shell we have used interactively in this course, is a full-fledged scripting language. Unlike Python, Bash is not a general-purpose language. Bash is explicitly designed to make running and interfacing command-line programs as simple as possible. For these reason, Bash often takes the role as the glue language of bioinformatics, as it s used to glue many commands together into a cohesive workflow.

4 Overview Note that Python is a more suitable language for commonly reused or advanced pipelines. Python is a more modern, fully featured scripting language than Bash. Compared to Python, Bash lacks several nice features useful for data-processing scripts: better numeric type support, useful data structures, better string processing, refined option parsing, availability of a large number of libraries, and powerful functions that help with structuring your programs. However, there s more overhead when calling command-line programs from a Python script compared to Bash. Bash is often the best and quickest glue solution.

5 Writing and running bash scripts Most Bash scripts in bioinformatics are simply commands organized into a re-runnable script with some features to check that files exist and ensuring any error causes the script to abort. We will learn the basics of writing and executing Bash scripts, paying particular attention to how create robust Bash scripts.

6 A robust Bash header By convention, Bash scripts have the extension.sh. You can create them in your favorite text editor (e.g. emacs or vi). Anytime you write a Bash script, you should use the following Bash script header, which sets some Bash options that lead to more robust scripts. #!/bin/bash set e set u set o pipefail

7 A robust Bash header #!/bin/bash This is called the shebang, and it indicates the path to the interpreter used to execute this script. set e By default, a shell script containing a command that fails will not cause the entire shell script to exit: the shell script will just continue on to the next line. We always want errors to be loud and noticeable. This option prevents this, by terminating the script if any command exited with a nonzero exit status.

8 A robust Bash header Note that this option ignores nonzero statuses in if conditionals. Also, it ignores all exit statuses in Unix pipes except the last one. set u This option fixes another default behavior of Bash scripts: any command containing a reference to an unset variable name will still run. It prevents this type of error by aborting the script if a variable s value is unset

9 A robust Bash header set o pipefail set e will cause a script to abort if a nonzero exit status is encountered, with some exceptions. One such exception is if a program runs in a Unix pipe exited unsuccessfully. Including set o pipefail will prevent this undesirable behavior: any program that returns a nonzero exit status in the pipe will cause the entire pipe to return a nonzero status. With set e enabled, this will lead the script to abort.

10 Running bash scripts Running Bash scripts can be done one of two ways: 1. bash script.sh 2../script.sh While we can run any script, calling the script as an executable requires that it has executable permissions. We can set these using: chmod u+x script.sh This adds executable permissions for the user who owns the file. Then, the script can be run with./script.sh.

11 Variables Processing pipelines having numerous settings that should be stored in variables. Storing these settings in a variable defined at the top of the file makes adjusting settings and rerunning your pipelines much easier. Rather than having to changes numerous hardcoded values in your scripts, using variables to store settings means you only have to change one value. Bash also reads command-line arguments into variables.

12 Variables Bash s variables don t have data types. It s helpful to think of Bash s variables as strings. We can create a variable and assign it a value with. results_dir= results/ Note that spaces matter when setting Bash variables. Do not use spaces around the equal sign.

13 Variables To access a variable s value, we use a dollar sign in front of the variable s name. Suppose we want to create a directory for a sample s alignment data, called <sample>_aln/, where <sample> is replaced by the sample s name. sample= CNTRL01A mkdir ${sample}_aln/

14 Command-line arguments The variable $0 stores the name of the script, and commandline arguments are assigned to the value $1, $2, $3, etc. Bash assigns the number of command-line arguments to $#.

15 Command-line arguments If you find your script requires numerous or complicated options, it might be easier to use Python instead of Bash. Python s argparse module is much easier to use. Variables created in your Bash script will only be available for the duration of the Bash process running that script.

16 if statement Bash supports the standard if conditional statement. The basic syntax is: if [commands] then [if-statements] else [else-statements] fi

17 if statement A command s exit status provides the true and false. Remember that 0 represents true/success and anything else if false/failure. if [commands] [commands] could be any command, set of commands, pipeline, or test condition. If the exit status of these commands is 0, execution continues to the block after then. Otherwise execution continues to the block after else.

18 if statement [if-statements] is a placeholder for all statements executed if [commands] evaluates to true (0). [else-statements] is a placeholder for all statements executed if [commands] evaluates to false. The else block is optional.

19 if statement Bash is primarily designed to stitch together other commands. This is an advantage Bash has over Python when writing pipelines. Bash allows your scripts to directly work with command-line programs without requiring any overhead to call programs. Although it can be unpleasant to write complicated programs in Bash, writing simple programs is exceedingly easy because Unix tools and Bash harmonize well.

20 if statement Suppose we wanted to run a set of commands only if a file contains a certain string. Because grep returns 0 only if it matches a pattern in a file and 1 otherwise. The redirection is to tidy the output of this script such that grep s output is redirected to /dev/null and not to the script s standard out.

21 test Like other programs, test exits with either 0 or 1. However test s exit status indicates the return value of the test specified through its arguments, rather than exit success or error. test supports numerous standard comparison operators.

22 test String/integer Description -z str String str is null str1 = str2 str1 and str2 are identical str1!= str2 str1 and str2 are different int1 eq int2 Integers int1 and int2 are equal int1 ne int2 int1 and int2 are not equal int1 lt int2 int1 is less than int2 int1 gt int2 int1 is greater than int2 int1 le int2 int1 is less than or equal to int2 int1 ge int2 int1 is greater than or equal to int2

23 test In practice, the most common conditions you ll be checking are to see if files or directories exist and whether you can write to them. test supports numerous file- and directoryrelated test operations.

24 test File/directory expression Description -d dir dir is a directory -f file file is a file -e file file exists -r file file is readable -w file file is writable -x file file is executable

25 test Combining test with if statements is simple: if test f some_file.txt then [ ] fi Bash provides a simpler syntactic alternative: if [ f some_file.txt ] then [ ] fi Note the spaces around and within the brackets: these are required.

26 test When using this syntax, we can chain test expression with a as logical AND, -o as logical OR,! as negation. Our familiar && and operators won t work in test, because these are shell operators. if [ $# ne 1 o! r $1 ] then fi echo usage: script.sh file_in.txt

27 for loop In bioinformatics, most of our data is split across multiple files. At the heart of any processing pipeline is some way to apply the same workflow to each of these files, taking care to keep track of sample names. Looping over files with Bash s for loop is the simplest way to accomplish this. There are three essential parts to creating a pipeline to process a set of files: 1. Selecting which files to apply the commands to 2. Looping over the data and applying the commands 3. Keeping track of the names of any output files created

28 for loop Suppose we have a file called samples.txt that tells you basic information about your raw data: sample name, read pair, and where the file is.

29 for loop Suppose we want to loop over every file, gather quality statistics on each and every file, and save this information to an output file. First, we load our filenames into a Bash array, which we can then loop over. Bash arrays can be created manually using:

30 for loop But creating Bash arrays by hand is tedious and error prone. The beauty of Bash is that we can use a command substitution to construct Bash arrays. We can strip the path and extension from each filename using basename.

31 for loop

32 Learning Unix

33 Sequence Alignments

34 DATA FORMATS

35 Overview Nucleotide (and protein) sequences are stored in two plaintext formats widespread in bioinformatics: FASTA and FASTQ. We will discuss each format and their limitations, and then see some tools for working with data in these formats.

36 FASTA The FASTA format originates from the FASTA alignment suites, created by William R. Pearson and David J. Lipman. The FASTA format is used to store any sort of sequence data not requiring per-base pair quality scores. This includes: reference genome files, protein sequences, coding DNA sequences (CDS), transcript sequences, and so on.

37 FASTA FASTA files are composed of sequence entries, each containing two parts: a description and the sequence data. The description line begins with a greater than symbol (>) and contains the sequence identifier and other optional information The sequence data begins on the next line after the description, and continues until there s another description line.

38 FASTA An example FASTA file: The FASTA format s simplicity and flexibility comes with an unfortunate downside: the FASTA format is a loosely defined ad hoc format.

39 FASTA In general, the following rules should be observed: 1. Sequence lines should not be too long. While a FASTA file that contains the sequence of the entire human chromosome 1 on a single line is a valid FASTA file, most tools that run on such a file would fail. 2. Some tools may accept data containing alphabets beyond those that they know how to deal with. For example, the standard alphabet for nucleotides would contain ATGC. An extended alphabet may also contain 1) N: A, T, G, or C 2) W: A or T 3) Search the web for IUPAC nucleotides to get a list of all such symbols.

40 FASTA 3. The sequence lines should always at the same width with the exception of the last line. Some tools will fail to operate correctly and may not even warn the users if this condition is not satisfied. The following is technically a valid FASTA but it may cause various problems: It should be reformatted to:

41 FASTA 4. Use upper-case letters. Whereas both lower-case and upper-case letters are allowed by the specification, the different capitalization may carry additional meaning and some tools and methods will operate differently when encoutering upper- or lower-case letters. Some communities (e.g. Ensembl) chose to designate the lowercase letters as all repeats and low complexity regions.

42 FASTQ The FASTQ format extends FASTA by including a numeric quality score to each base in the sequence. The FASTQ format is widely used to store high-throughput sequencing data, which is reported with a per-base quality score indicating the confidence of each base call. It is the de facto standard by which all sequencing instruments represent data.

43 FASTQ The FASTQ format looks like: Line1: The description line beginning This contains the record identifier and other information. Line2: Sequence data, which can be on one or many lines. Line3: The line beginning with + indicates the end of the sequence. Line4: Quality data, which can also be on one or many lines, but must be the same length as the sequence. Each numeric base quality is encoded with ASCII characters.

44 FASTQ The FASTQ format is a multi-line format just as the FASTA format is. In the early days of high-throughput sequencing, instruments always produced the entire FASTQ sequence on a single line. The FASTQ format suffers from the unexpected flaw that sign is both a FASTQ record separator and a valid value of the quality string. For that reason it is a little more difficult to design a correct FASTQ parsing program.

Lecture 5. Essential skills for bioinformatics: Unix/Linux

Lecture 5. Essential skills for bioinformatics: Unix/Linux Lecture 5 Essential skills for bioinformatics: Unix/Linux UNIX DATA TOOLS Text processing with awk We have illustrated two ways awk can come in handy: Filtering data using rules that can combine regular

More information

Essentials for Scientific Computing: Bash Shell Scripting Day 3

Essentials for Scientific Computing: Bash Shell Scripting Day 3 Essentials for Scientific Computing: Bash Shell Scripting Day 3 Ershaad Ahamed TUE-CMS, JNCASR May 2012 1 Introduction In the previous sessions, you have been using basic commands in the shell. The bash

More information

Bash scripting basics

Bash scripting basics Bash scripting basics prepared by Anatoliy Antonov for ESSReS community September 2012 1 Outline Definitions Foundations Flow control References and exercises 2 Definitions 3 Definitions Script - [small]

More information

COMP 4/6262: Programming UNIX

COMP 4/6262: Programming UNIX COMP 4/6262: Programming UNIX Lecture 12 shells, shell programming: passing arguments, if, debug March 13, 2006 Outline shells shell programming passing arguments (KW Ch.7) exit status if (KW Ch.8) test

More information

9.2 Linux Essentials Exam Objectives

9.2 Linux Essentials Exam Objectives 9.2 Linux Essentials Exam Objectives This chapter will cover the topics for the following Linux Essentials exam objectives: Topic 3: The Power of the Command Line (weight: 10) 3.3: Turning Commands into

More information

Shell Programming (bash)

Shell Programming (bash) Shell Programming Shell Programming (bash) Commands run from a file in a subshell A great way to automate a repeated sequence of commands. File starts with #!/bin/bash absolute path to the shell program

More information

OPERATING SYSTEMS LAB LAB # 6. I/O Redirection and Shell Programming. Shell Programming( I/O Redirection and if-else Statement)

OPERATING SYSTEMS LAB LAB # 6. I/O Redirection and Shell Programming. Shell Programming( I/O Redirection and if-else Statement) P a g e 1 OPERATING SYSTEMS LAB LAB 6 I/O Redirection and Shell Programming Lab 6 Shell Programming( I/O Redirection and if-else Statement) P a g e 2 Redirection of Standard output/input i.e. Input - Output

More information

Linux Shell Scripting. Linux System Administration COMP2018 Summer 2017

Linux Shell Scripting. Linux System Administration COMP2018 Summer 2017 Linux Shell Scripting Linux System Administration COMP2018 Summer 2017 What is Scripting? Commands can be given to a computer by entering them into a command interpreter program, commonly called a shell

More information

Shell scripting and system variables. HORT Lecture 5 Instructor: Kranthi Varala

Shell scripting and system variables. HORT Lecture 5 Instructor: Kranthi Varala Shell scripting and system variables HORT 59000 Lecture 5 Instructor: Kranthi Varala Text editors Programs built to assist creation and manipulation of text files, typically scripts. nano : easy-to-learn,

More information

CSCI 211 UNIX Lab. Shell Programming. Dr. Jiang Li. Jiang Li, Ph.D. Department of Computer Science

CSCI 211 UNIX Lab. Shell Programming. Dr. Jiang Li. Jiang Li, Ph.D. Department of Computer Science CSCI 211 UNIX Lab Shell Programming Dr. Jiang Li Why Shell Scripting Saves a lot of typing A shell script can run many commands at once A shell script can repeatedly run commands Help avoid mistakes Once

More information

CS Unix Tools & Scripting

CS Unix Tools & Scripting Cornell University, Spring 2014 1 February 24, 2014 1 Slides evolved from previous versions by Hussam Abu-Libdeh and David Slater A note on awk for (item in array) The order in which items are returned

More information

Command Interpreters. command-line (e.g. Unix shell) On Unix/Linux, bash has become defacto standard shell.

Command Interpreters. command-line (e.g. Unix shell) On Unix/Linux, bash has become defacto standard shell. Command Interpreters A command interpreter is a program that executes other programs. Aim: allow users to execute the commands provided on a computer system. Command interpreters come in two flavours:

More information

Shell Start-up and Configuration Files

Shell Start-up and Configuration Files ULI101 Week 10 Lesson Overview Shell Start-up and Configuration Files Shell History Alias Statement Shell Variables Introduction to Shell Scripting Positional Parameters echo and read Commands if and test

More information

bash Execution Control COMP2101 Winter 2019

bash Execution Control COMP2101 Winter 2019 bash Execution Control COMP2101 Winter 2019 Bash Execution Control Scripts commonly can evaluate situations and make simple decisions about actions to take Simple evaluations and actions can be accomplished

More information

Scripting. More Shell Scripts. Adapted from Practical Unix and Programming Hunter College

Scripting. More Shell Scripts. Adapted from Practical Unix and Programming Hunter College Scripting More Shell Scripts Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss Back to shell scripts Now that you've learned a few commands and can edit files,

More information

Contents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...

Contents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version... Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing

More information

A shell can be used in one of two ways:

A shell can be used in one of two ways: Shell Scripting 1 A shell can be used in one of two ways: A command interpreter, used interactively A programming language, to write shell scripts (your own custom commands) 2 If we have a set of commands

More information

Shells & Shell Programming (Part B)

Shells & Shell Programming (Part B) Shells & Shell Programming (Part B) Software Tools EECS2031 Winter 2018 Manos Papagelis Thanks to Karen Reid and Alan J Rosenthal for material in these slides CONTROL STATEMENTS 2 Control Statements Conditional

More information

CSC 2500: Unix Lab Fall 2016

CSC 2500: Unix Lab Fall 2016 CSC 2500: Unix Lab Fall 2016 Control Statements in Shell Scripts: Decision Mohammad Ashiqur Rahman Department of Computer Science College of Engineering Tennessee Tech University Agenda User Input Special

More information

Shells and Shell Programming

Shells and Shell Programming Shells and Shell Programming 1 Shells A shell is a command line interpreter that is the interface between the user and the OS. The shell: analyzes each command determines what actions are to be performed

More information

A Big Step. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers

A Big Step. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers A Big Step Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Copyright 2006 2009 Stewart Weiss What a shell really does Here is the scoop on shells. A shell is a program

More information

Shell programming. Introduction to Operating Systems

Shell programming. Introduction to Operating Systems Shell programming Introduction to Operating Systems Environment variables Predened variables $* all parameters $# number of parameters $? result of last command $$ process identier $i parameter number

More information

COMP 2718: Shell Scripts: Part 1. By: Dr. Andrew Vardy

COMP 2718: Shell Scripts: Part 1. By: Dr. Andrew Vardy COMP 2718: Shell Scripts: Part 1 By: Dr. Andrew Vardy Outline Shell Scripts: Part 1 Hello World Shebang! Example Project Introducing Variables Variable Names Variable Facts Arguments Exit Status Branching:

More information

Shells and Shell Programming

Shells and Shell Programming Shells and Shell Programming Shells A shell is a command line interpreter that is the interface between the user and the OS. The shell: analyzes each command determines what actions are to be performed

More information

Scripting. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers

Scripting. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Scripting Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss What a shell

More information

LING 408/508: Computational Techniques for Linguists. Lecture 5

LING 408/508: Computational Techniques for Linguists. Lecture 5 LING 408/508: Computational Techniques for Linguists Lecture 5 Last Time Installing Ubuntu 18.04 LTS on top of VirtualBox Your Homework 2: did everyone succeed? Ubuntu VirtualBox Host OS: MacOS or Windows

More information

Bash scripting Tutorial. Hello World Bash Shell Script. Super User Programming & Scripting 22 March 2013

Bash scripting Tutorial. Hello World Bash Shell Script. Super User Programming & Scripting 22 March 2013 Bash scripting Tutorial Super User Programming & Scripting 22 March 2013 Hello World Bash Shell Script First you need to find out where is your bash interpreter located. Enter the following into your command

More information

SHELL SCRIPT BASIC. UNIX Programming 2014 Fall by Euiseong Seo

SHELL SCRIPT BASIC. UNIX Programming 2014 Fall by Euiseong Seo SHELL SCRIPT BASIC UNIX Programming 2014 Fall by Euiseong Seo Shell Script Interactive shell sequentially executes a series of commands Some tasks are repetitive and automatable They are what programs

More information

Lecture 3. Essential skills for bioinformatics: Unix/Linux

Lecture 3. Essential skills for bioinformatics: Unix/Linux Lecture 3 Essential skills for bioinformatics: Unix/Linux RETRIEVING DATA Overview Whether downloading large sequencing datasets or accessing a web application hundreds of times to download specific files,

More information

example: name1=jan name2=mike export name1 In this example, name1 is an environmental variable while name2 is a local variable.

example: name1=jan name2=mike export name1 In this example, name1 is an environmental variable while name2 is a local variable. Bourne Shell Programming Variables - creating and assigning variables Bourne shell use the set and unset to create and assign values to variables or typing the variable name, an equal sign and the value

More information

Control Structures. CIS 118 Intro to LINUX

Control Structures. CIS 118 Intro to LINUX Control Structures CIS 118 Intro to LINUX Basic Control Structures TEST The test utility, has many formats for evaluating expressions. For example, when given three arguments, will return the value true

More information

UNIX shell scripting

UNIX shell scripting UNIX shell scripting EECS 2031 Summer 2014 Przemyslaw Pawluk June 17, 2014 What we will discuss today Introduction Control Structures User Input Homework Table of Contents Introduction Control Structures

More information

Computer Systems and Architecture

Computer Systems and Architecture Computer Systems and Architecture Stephen Pauwels UNIX Scripting Academic Year 2018-2019 Outline Basics Conditionals Loops Advanced Exercises Shell Scripts Grouping commands into a single file Reusability

More information

Shell Scripting. Todd Kelley CST8207 Todd Kelley 1

Shell Scripting. Todd Kelley CST8207 Todd Kelley 1 Shell Scripting Todd Kelley kelleyt@algonquincollege.com CST8207 Todd Kelley 1 If we have a set of commands that we want to run on a regular basis, we could write a script A script acts as a Linux command,

More information

SHELL SCRIPT BASIC. UNIX Programming 2015 Fall by Euiseong Seo

SHELL SCRIPT BASIC. UNIX Programming 2015 Fall by Euiseong Seo SHELL SCRIPT BASIC UNIX Programming 2015 Fall by Euiseong Seo Shell Script! Interactive shell sequentially executes a series of commands! Some tasks are repetitive and automatable! They are what programs

More information

Linux Bash Shell Scripting

Linux Bash Shell Scripting University of Chicago Initiative in Biomedical Informatics Computation Institute Linux Bash Shell Scripting Present by: Mohammad Reza Gerami gerami@ipm.ir Day 2 Outline Support Review of Day 1 exercise

More information

Unix Shell scripting. Dr Alun Moon 7th October Introduction. Notation. Spaces

Unix Shell scripting. Dr Alun Moon 7th October Introduction. Notation. Spaces Unix Shell scripting Dr Alun Moon 7th October 2017 Introduction Shell scripts in Unix are a very powerfull tool, they form much of the standard system as installed. What are these good for? So many file

More information

Computer Systems and Architecture

Computer Systems and Architecture Computer Systems and Architecture UNIX Scripting Bart Meyers University of Antwerp August 29, 2012 Outline Basics Conditionals Loops Advanced Exercises Shell scripts Grouping commands into a single file

More information

Linux shell & shell scripting - II

Linux shell & shell scripting - II IBS 574 - Computational Biology & Bioinformatics Spring 2018, Tuesday (02/01), 2:00-4:00PM Linux shell & shell scripting - II Ashok R. Dinasarapu Ph.D Scientist, Bioinformatics Dept. of Human Genetics,

More information

Programming Languages and Uses in Bioinformatics

Programming Languages and Uses in Bioinformatics Programming in Perl Programming Languages and Uses in Bioinformatics Perl, Python Pros: reformatting data files reading, writing and parsing files building web pages and database access building work flow

More information

bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017

bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017 bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017 Command Lists A command is a sequence of commands separated by the operators ; & && and ; is used to simply execute commands in

More information

Grep and Shell Programming

Grep and Shell Programming Grep and Shell Programming Comp-206 : Introduction to Software Systems Lecture 7 Alexandre Denault Computer Science McGill University Fall 2006 Teacher's Assistants Michael Hawker Monday, 14h30 to 16h30

More information

Shell Script Example. Here is a hello world shell script: $ ls -l -rwxr-xr-x 1 horner 48 Feb 19 11:50 hello* $ cat hello #!/bin/sh

Shell Script Example. Here is a hello world shell script: $ ls -l -rwxr-xr-x 1 horner 48 Feb 19 11:50 hello* $ cat hello #!/bin/sh Shell Programming Shells A shell can be used in one of two ways: A command interpreter, used interactively A programming language, to write shell scripts (your own custom commands) Shell Scripts A shell

More information

http://xkcd.com/208/ 1. Review of pipes 2. Regular expressions 3. sed 4. Editing Files 5. Shell loops 6. Shell scripts cat seqs.fa >0! TGCAGGTATATCTATTAGCAGGTTTAATTTTGCCTGCACTTGGTTGGGTACATTATTTTAAGTGTATTTGACAAG!

More information

Assignment 3, Due October 4

Assignment 3, Due October 4 Assignment 3, Due October 4 1 Summary This assignment gives you practice with writing shell scripts. Shell scripting is also known as bash programming. Your shell is bash, and when you write a shell script

More information

Vi & Shell Scripting

Vi & Shell Scripting Vi & Shell Scripting Comp-206 : Introduction to Week 3 Joseph Vybihal Computer Science McGill University Announcements Sina Meraji's office hours Trottier 3rd floor open area Tuesday 1:30 2:30 PM Thursday

More information

sottotitolo A.A. 2016/17 Federico Reghenzani, Alessandro Barenghi

sottotitolo A.A. 2016/17 Federico Reghenzani, Alessandro Barenghi Titolo presentazione Piattaforme Software per la Rete sottotitolo BASH Scripting Milano, XX mese 20XX A.A. 2016/17, Alessandro Barenghi Outline 1) Introduction to BASH 2) Helper commands 3) Control Flow

More information

Title:[ Variables Comparison Operators If Else Statements ]

Title:[ Variables Comparison Operators If Else Statements ] [Color Codes] Environmental Variables: PATH What is path? PATH=$PATH:/MyFolder/YourStuff?Scripts ENV HOME PWD SHELL PS1 EDITOR Showing default text editor #!/bin/bash a=375 hello=$a #No space permitted

More information

Shell script. Shell Scripts. A shell script contains a sequence of commands in a text file. Shell is an command language interpreter.

Shell script. Shell Scripts. A shell script contains a sequence of commands in a text file. Shell is an command language interpreter. Shell Scripts A shell script contains a sequence of commands in a text file. Shell is an command language interpreter. Shell executes commands read from a file. Shell is a powerful programming available

More information

Програмиранев UNIX среда

Програмиранев UNIX среда Програмиранев UNIX среда Използванена команден шел и създаванена скриптове: tcsh, bash, awk, python Shell programming As well as using the shell to run commands you can use its built-in programming language

More information

Bourne Shell Reference

Bourne Shell Reference > Linux Reviews > Beginners: Learn Linux > Bourne Shell Reference Bourne Shell Reference found at Br. David Carlson, O.S.B. pages, cis.stvincent.edu/carlsond/cs330/unix/bshellref - Converted to txt2tags

More information

Advanced Unix Programming Module 03 Raju Alluri spurthi.com

Advanced Unix Programming Module 03 Raju Alluri spurthi.com Advanced Unix Programming Module 03 Raju Alluri askraju @ spurthi.com Advanced Unix Programming: Module 3 Shells & Shell Programming Environment Variables Writing Simple Shell Programs (shell scripts)

More information

Windshield. Language Reference Manual. Columbia University COMS W4115 Programming Languages and Translators Spring Prof. Stephen A.

Windshield. Language Reference Manual. Columbia University COMS W4115 Programming Languages and Translators Spring Prof. Stephen A. Windshield Language Reference Manual Columbia University COMS W4115 Programming Languages and Translators Spring 2007 Prof. Stephen A. Edwards Team members Wei-Yun Ma wm2174 wm2174@columbia.edu Tony Wang

More information

Scripting. More Shell Scripts Loops. Adapted from Practical Unix and Programming Hunter College

Scripting. More Shell Scripts Loops. Adapted from Practical Unix and Programming Hunter College Scripting More Shell Scripts Loops Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss Loops: for The for script version 1: 1 #!/bin/bash 2 for i in 1 2 3 3 do

More information

EECS 470 Lab 5. Linux Shell Scripting. Friday, 1 st February, 2018

EECS 470 Lab 5. Linux Shell Scripting. Friday, 1 st February, 2018 EECS 470 Lab 5 Linux Shell Scripting Department of Electrical Engineering and Computer Science College of Engineering University of Michigan Friday, 1 st February, 2018 (University of Michigan) Lab 5:

More information

Chapter 4. Unix Tutorial. Unix Shell

Chapter 4. Unix Tutorial. Unix Shell Chapter 4 Unix Tutorial Users and applications interact with hardware through an operating system (OS). Unix is a very basic operating system in that it has just the essentials. Many operating systems,

More information

http://xkcd.com/208/ 1. Review of pipes 2. Regular expressions 3. sed 4. awk 5. Editing Files 6. Shell loops 7. Shell scripts cat seqs.fa >0! TGCAGGTATATCTATTAGCAGGTTTAATTTTGCCTGCACTTGGTTGGGTACATTATTTTAAGTGTATTTGACAAG!

More information

COMS 6100 Class Notes 3

COMS 6100 Class Notes 3 COMS 6100 Class Notes 3 Daniel Solus September 1, 2016 1 General Remarks The class was split into two main sections. We finished our introduction to Linux commands by reviewing Linux commands I and II

More information

STP, Unix, SAC tutorial, Ge167 Winter 2014

STP, Unix, SAC tutorial, Ge167 Winter 2014 STP, Unix, SAC tutorial, Ge167 Winter 2014 Asaf Inbal 1 Downloading waveforms In this tutorial we ll learn how to download waveforms using a tool called STP (Seismic Transfer Program) and manipulate them

More information

Simple Shell Scripting for Scientists

Simple Shell Scripting for Scientists Simple Shell Scripting for Scientists Day Three Julian King Bruce Beckles University of Cambridge Computing Service 1 Introduction Who:! Julian King, Unix Support, UCS! Bruce Beckles, e-science Specialist,

More information

Introduction: What is Unix?

Introduction: What is Unix? Introduction Introduction: What is Unix? An operating system Developed at AT&T Bell Labs in the 1960 s Command Line Interpreter GUIs (Window systems) are now available Introduction: Unix vs. Linux Unix

More information

Useful Unix Commands Cheat Sheet

Useful Unix Commands Cheat Sheet Useful Unix Commands Cheat Sheet The Chinese University of Hong Kong SIGSC Training (Fall 2016) FILE AND DIRECTORY pwd Return path to current directory. ls List directories and files here. ls dir List

More information

Bashed One Too Many Times. Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009

Bashed One Too Many Times. Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009 Bashed One Too Many Times Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009 What is a Shell? The shell interprets commands and executes them It provides you with an environment

More information

Basic Linux (Bash) Commands

Basic Linux (Bash) Commands Basic Linux (Bash) Commands Hint: Run commands in the emacs shell (emacs -nw, then M-x shell) instead of the terminal. It eases searching for and revising commands and navigating and copying-and-pasting

More information

When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used:

When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used: Linux Tutorial How to read the examples When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used: $ application file.txt

More information

UNIX System Programming Lecture 3: BASH Programming

UNIX System Programming Lecture 3: BASH Programming UNIX System Programming Outline Filesystems Redirection Shell Programming Reference BLP: Chapter 2 BFAQ: Bash FAQ BMAN: Bash man page BPRI: Bash Programming Introduction BABS: Advanced Bash Scripting Guide

More information

CS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND

CS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND CS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND Prof. Michael J. Reale Fall 2014 Finding Files in a Directory Tree Suppose you want to find a file with a certain filename (or with a filename matching

More information

Lecture 8. Sequence alignments

Lecture 8. Sequence alignments Lecture 8 Sequence alignments DATA FORMATS bioawk bioawk is a program that extends awk s powerful processing of tabular data to processing tasks involving common bioinformatics formats like FASTA/FASTQ,

More information

Linux shell programming for Raspberry Pi Users - 2

Linux shell programming for Raspberry Pi Users - 2 Linux shell programming for Raspberry Pi Users - 2 Sarwan Singh Assistant Director(S) NIELIT Chandigarh 1 SarwanSingh.com Education is the kindling of a flame, not the filling of a vessel. - Socrates SHELL

More information

1. Hello World Bash Shell Script. Last Updated on Wednesday, 13 April :03

1. Hello World Bash Shell Script. Last Updated on Wednesday, 13 April :03 1 of 18 21/10/2554 9:39 Bash scripting Tutorial tar -czf myhome_directory.tar.gz /home/linuxcong Last Updated on Wednesday, 13 April 2011 08:03 Article Index 1. Hello World Bash Shell Script 2. Simple

More information

bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017

bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017 bash Tests and Looping Administrative Shell Scripting COMP2101 Fall 2017 Command Lists A command is a sequence of commands separated by the operators ; & && and ; is used to simply execute commands in

More information

Introduction Variables Helper commands Control Flow Constructs Basic Plumbing. Bash Scripting. Alessandro Barenghi

Introduction Variables Helper commands Control Flow Constructs Basic Plumbing. Bash Scripting. Alessandro Barenghi Bash Scripting Alessandro Barenghi Dipartimento di Elettronica, Informazione e Bioingegneria Politecnico di Milano alessandro.barenghi - at - polimi.it April 28, 2015 Introduction The bash command shell

More information

Cisco IOS Shell. Finding Feature Information. Prerequisites for Cisco IOS.sh. Last Updated: December 14, 2012

Cisco IOS Shell. Finding Feature Information. Prerequisites for Cisco IOS.sh. Last Updated: December 14, 2012 Cisco IOS Shell Last Updated: December 14, 2012 The Cisco IOS Shell (IOS.sh) feature provides shell scripting capability to the Cisco IOS command-lineinterface (CLI) environment. Cisco IOS.sh enhances

More information

Assignment clarifications

Assignment clarifications Assignment clarifications How many errors to print? at most 1 per token. Interpretation of white space in { } treat as a valid extension, involving white space characters. Assignment FAQs have been updated.

More information

B a s h s c r i p t i n g

B a s h s c r i p t i n g 8 Bash Scripting Any self-respecting hacker must be able to write scripts. For that matter, any selfrespecting Linux administrator must be able to script. Hackers often need to automate commands, sometimes

More information

Introduction to Linux Basics Part II. Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala

Introduction to Linux Basics Part II. Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala Introduction to Linux Basics Part II 1 Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu 2 Variables in Shell HOW DOES LINUX WORK? Shell Arithmetic I/O and

More information

CSNB113: System Administration - 8 th Topic: Shell Scripts Programming I

CSNB113: System Administration - 8 th Topic: Shell Scripts Programming I CSNB113: System Administration - 8 th Topic: Shell Scripts Programming I The prompt is a prompt is a prompt and a program input It is possible to write and run a program directly on the command prompt!

More information

Genomic Files. University of Massachusetts Medical School. October, 2015

Genomic Files. University of Massachusetts Medical School. October, 2015 .. Genomic Files University of Massachusetts Medical School October, 2015 2 / 55. A Typical Deep-Sequencing Workflow Samples Fastq Files Fastq Files Sam / Bam Files Various files Deep Sequencing Further

More information

There are some string operators that can be used in the test statement to perform string comparison.

There are some string operators that can be used in the test statement to perform string comparison. ACS-294- Unix (Winter Term, 26-7) Part II: Shell Programming Page 3 The test also returns a nonzero exit value if there is no argument: test String Operators There are some string operators that can be

More information

Introduction to Shell Scripting

Introduction to Shell Scripting Introduction to Shell Scripting Lecture 1. Shell scripts are small programs. They let you automate multi-step processes, and give you the capability to use decision-making logic and repetitive loops. 2.

More information

Bash command shell language interpreter

Bash command shell language interpreter Principles of Programming Languages Bash command shell language interpreter Advanced seminar topic Louis Sugy & Baptiste Thémine Presentation on December 8th, 2017 Table of contents I. General information

More information

12.1 UNDERSTANDING UNIX SHELL PROGRAMMING LANGUAGE: AN INTRODUCTION Writing a Simple Script Executing a Script

12.1 UNDERSTANDING UNIX SHELL PROGRAMMING LANGUAGE: AN INTRODUCTION Writing a Simple Script Executing a Script 12 Shell Programming This chapter concentrates on shell programming. It explains the capabilities of the shell as an interpretive high-level language. It describes shell programming constructs and particulars.

More information

Answers to AWK problems. Shell-Programming. Future: Using loops to automate tasks. Download and Install: Python (Windows only.) R

Answers to AWK problems. Shell-Programming. Future: Using loops to automate tasks. Download and Install: Python (Windows only.) R Today s Class Answers to AWK problems Shell-Programming Using loops to automate tasks Future: Download and Install: Python (Windows only.) R Awk basics From the command line: $ awk '$1>20' filename Command

More information

Introduction to UNIX command-line II

Introduction to UNIX command-line II Introduction to UNIX command-line II Boyce Thompson Institute 2017 Prashant Hosmani Class Content Terminal file system navigation Wildcards, shortcuts and special characters File permissions Compression

More information

INTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT

INTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT INTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT GOALS BASIC What is shell scripting? What makes shell scripting useful? What should you know already GOALS PART 1 Script layout/design

More information

CSCI 2132: Software Development. Norbert Zeh. Faculty of Computer Science Dalhousie University. Shell Scripting. Winter 2019

CSCI 2132: Software Development. Norbert Zeh. Faculty of Computer Science Dalhousie University. Shell Scripting. Winter 2019 CSCI 2132: Software Development Shell Scripting Norbert Zeh Faculty of Computer Science Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your Shell vs Your File Manager File manager

More information

Genomic Files. University of Massachusetts Medical School. October, 2014

Genomic Files. University of Massachusetts Medical School. October, 2014 .. Genomic Files University of Massachusetts Medical School October, 2014 2 / 39. A Typical Deep-Sequencing Workflow Samples Fastq Files Fastq Files Sam / Bam Files Various files Deep Sequencing Further

More information

5/20/2007. Touring Essential Programs

5/20/2007. Touring Essential Programs Touring Essential Programs Employing fundamental utilities. Managing input and output. Using special characters in the command-line. Managing user environment. Surveying elements of a functioning system.

More information

Introduction to Linux

Introduction to Linux Introduction to Linux The command-line interface A command-line interface (CLI) is a type of interface, that is, a way to interact with a computer. Window systems, punched cards or a bunch of dials, buttons

More information

ITST Searching, Extracting & Archiving Data

ITST Searching, Extracting & Archiving Data ITST 1136 - Searching, Extracting & Archiving Data Name: Step 1 Sign into a Pi UN = pi PW = raspberry Step 2 - Grep - One of the most useful and versatile commands in a Linux terminal environment is the

More information

Digital Humanities. Tutorial Regular Expressions. March 10, 2014

Digital Humanities. Tutorial Regular Expressions. March 10, 2014 Digital Humanities Tutorial Regular Expressions March 10, 2014 1 Introduction In this tutorial we will look at a powerful technique, called regular expressions, to search for specific patterns in corpora.

More information

CS Unix Tools & Scripting

CS Unix Tools & Scripting Cornell University, Spring 2014 1 February 21, 2014 1 Slides evolved from previous versions by Hussam Abu-Libdeh and David Slater Announcement HW 4 is out. Due Friday, February 28, 2014 at 11:59PM. Wrapping

More information

Merge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p.

Merge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p. Preface p. xiii Ideology: Data Skills for Robust and Reproducible Bioinformatics How to Learn Bioinformatics p. 1 Why Bioinformatics? Biology's Growing Data p. 1 Learning Data Skills to Learn Bioinformatics

More information

Introduction To. Barry Grant

Introduction To. Barry Grant Introduction To Barry Grant bjgrant@umich.edu http://thegrantlab.org Working with Unix How do we actually use Unix? Inspecting text files less - visualize a text file: use arrow keys page down/page up

More information

Unix as a Platform Exercises + Solutions. Course Code: OS 01 UNXPLAT

Unix as a Platform Exercises + Solutions. Course Code: OS 01 UNXPLAT Unix as a Platform Exercises + Solutions Course Code: OS 01 UNXPLAT Working with Unix Most if not all of these will require some investigation in the man pages. That's the idea, to get them used to looking

More information

Bash shell programming Part II Control statements

Bash shell programming Part II Control statements Bash shell programming Part II Control statements Deniz Savas and Michael Griffiths 2005-2011 Corporate Information and Computing Services The University of Sheffield Email M.Griffiths@sheffield.ac.uk

More information

Essential Skills for Bioinformatics: Unix/Linux

Essential Skills for Bioinformatics: Unix/Linux Essential Skills for Bioinformatics: Unix/Linux WORKING WITH COMPRESSED DATA Overview Data compression, the process of condensing data so that it takes up less space (on disk drives, in memory, or across

More information

Last Time. on the website

Last Time. on the website Last Time on the website Lecture 6 Shell Scripting What is a shell? The user interface to the operating system Functionality: Execute other programs Manage files Manage processes Full programming language

More information

INTRODUCTION TO UNIX

INTRODUCTION TO UNIX WEEK 1.3 INTRODUCTION TO UNIX Unix is a key operating system in the history of communications. Here, you will gain experience in using Unix both in command line and with a graphical interface. Most importantly,

More information

Basics. I think that the later is better.

Basics.  I think that the later is better. Basics Before we take up shell scripting, let s review some of the basic features and syntax of the shell, specifically the major shells in the sh lineage. Command Editing If you like vi, put your shell

More information