Lecture 5. Essential skills for bioinformatics: Unix/Linux
|
|
- Ernest Elliott
- 6 years ago
- Views:
Transcription
1 Lecture 5 Essential skills for bioinformatics: Unix/Linux
2 UNIX DATA TOOLS
3 Text processing with awk We have illustrated two ways awk can come in handy: Filtering data using rules that can combine regular expressions and arithmetic Reformatting the columns of data using arithmetic We will learn more advanced use cases by introducing two special patterns: BEGIN and END. The BEGIN pattern specifies what to do before the first record is read in. It is useful to initialize and set up variables. The END specifies what to do after the last record s processing is complete. It is useful to print data summaries at the end of file processing.
4 Text processing with awk Suppose we want to calculate the mean feature length in Homo_sapiens.GRCh38.87.bed: NR is the current record number, so on the last record NR is set to the total number of records processed.
5 Text processing with awk We can use NR to extract ranges of lines:
6 Text processing with awk awk makes it easy to convert between bioinformatics files like BED and GTF. Our previous solution using grep and cut has an error:
7 Text processing with awk We can generate a three-column BED file from the GTF file as follows. Note that we subtract 1 from the start position to convert to BED format. This is because BED uses zero-indexing while GTF uses 1-indexing.
8 Text processing with awk Using sort uniq c, we counted the number of features belonging to a particular gene. awk also have a very useful data structure known as an associative array which behaves like Python s dictionaries or hashes in other languages.
9 Text processing with awk We can create an associative array by simply assigning a value to a key.
10 Text processing with awk This example illustrates that awk is a programming language: within our action blocks, we can use standard programming statements like if, for, and while. However, when awk programs become complex or start to span multiple lines, you should switch to Python at that point.
11 Stream editing with sed We learned how Unix pipes are fast because they operate on streams of data rather than data written to disk. Additionally, pipes don t require that we load an entire file in memory at once. Instead, we can operate one line at a time. Often we need to make trivial edits to a stream, usually to prepare it for the next step in a Unix pipeline. The stream editor, sed, allows us to do exactly that.
12 Stream editing with sed sed reads data from a file or standard input and can edit a line at a time. Let s look at a very simple example: converting a file containing a single column of chromosomes in the format chrom1 to the format chr1.
13 Stream editing with sed We can edit it without opening the entire file in memory. Our edited output stream is then easy to redirect to a new file. In the previous example, we uses sed s substitute command, by far the most popular use of sed. sed s substitute takes the first occurrence of the pattern between the first two slashes, and replaces it with the string between the second and third slashes. In other words, the syntax of sed s substitute is s/pattern/replacement.
14 Stream editing with sed By default, sed only replaces the first occurrence of a match of each line. We can replace all occurrences of strings that match our pattern by setting the global flag g after the last slash: s/pattern/replacement/g If we need matching to be case-insensitive, we can enable this with the flag I By default, sed s substitutions use POSIX BRE. As with grep, we can use the E option to enable POSOX ERE.
15 Stream editing with sed Most important is the ability to capture chunks of text that match a pattern, and use these chunks in the replacement. Suppose we want to capture the chromosome name, and start and end positions in a string containing a genomic region in the format chr1: , and output this as three columns.
16 Stream editing with sed ^(chr[^:]+): This matches the text that begins at the start of the line (^ enforces this), and then captures everything between ( and ). This pattern used for capturing begins with chr and matches one or more characters that are not :, our delimiter. We match until the first : through a character class defined by everything that s not :, [^:]+
17 Stream editing with sed ([0-9]+): Match and capture more than one number. Finally, our replacement is these three captured groups, interspersed with tabs, \t. Regular expressions are tricky and take time and practice to master.
18 Stream editing with sed Explicitly capturing each component of our genomic region is one way to tackle this, and nicely demonstrates sed s ability to capture patterns. However, there are numerous ways to use sed or other Unix tools to parse strings like this.
19 Stream editing with sed sed 's/[:-]/\t/g : We just replace both delimiters with a tab. Note that we ve enabled the global flag, which is necessary for this approach to work. sed 's/:/\t/' sed 's/-/\t/ : For complex substitutions, it can be much easier to use two or more calls to sed rather than trying to do this with one regular expression. tr ':-' '\t : tr translates all occurrences of its first argument to its second.
20 Stream editing with sed By default, sed prints every lines, making replacements to matching lines. Suppose we want to capture all transcript names from the last column of a GTF file. Some lines of the last column of the GTF file don t contain transcript_id, so sed prints the entire line rather than the captured group.
21 Stream editing with sed One way to solve this would be to use grep transcript_ id before sed. A cleaner way is to disable sed from outputting all lines with n. Then, by appending p after the last slash sed will print all lines it s made a replacement on.
22 Stream editing with sed This example uses an important regular expression idiom: capturing text between delimiters. This is a useful pattern. 1. First, match zero or more of any character (.*) before the string transcript_id 2. Then, match and capture one or more characters that are not a quote ([^ ]+). This is an important idiom. The brackets make up a character class. Character classes specify what characters the expression is allowed to match. Here, we use a caret (^) inside the brackets to match anything except what s inside these brackets. The end result is that we match and capture one or more nonquote characters.
23 Stream editing with sed The following approach ((.*) rather than ([^ ]+)) will not work
24 Stream editing with sed It is also possible to select and print certain ranges of lines with sed. In this case, we are not doing pattern matching, so we don t need slashes. To print lines 20 through 50 of a file, we use sed has features that allow you to make any type of edit to a stream of test, but for complex stream processing tasks it can be easier to write a Python script than a long and complicated sed command.
25 SHELL SCRIPTING
26 Overview Bash, the shell we have used interactively in this course, is a full-fledged scripting language. Unlike Python, Bash is not a general-purpose language. Bash is explicitly designed to make running and interfacing command-line programs as simple as possible. For these reason, Bash often takes the role as the glue language of bioinformatics, as it s used to glue many commands together into a cohesive workflow.
27 Overview Note that Python is a more suitable language for commonly reused or advanced pipelines. Python is a more modern, fully featured scripting language than Bash. Compared to Python, Bash lacks several nice features useful for data-processing scripts: better numeric type support, useful data structures, better string processing, refined option parsing, availability of a large number of libraries, and powerful functions that help with structuring your programs. However, there s more overhead when calling command-line programs from a Python script compared to Bash. Bash is often the best and quickest glue solution.
28 Writing and running bash scripts Most Bash scripts in bioinformatics are simply commands organized into a re-runnable script with some features to check that files exist and ensuring any error causes the script to abort. We will learn the basics of writing and executing Bash scripts, paying particular attention to how create robust Bash scripts.
29 A robust Bash header By convention, Bash scripts have the extension.sh. You can create them in your favorite text editor (e.g. emacs, nano, or vi). Anytime you write a Bash script, you should use the following Bash script header, which sets some Bash options that lead to more robust scripts. #!/bin/bash set e set u set o pipefail
30 A robust Bash header #!/bin/bash This is called the shebang, and it indicates the path to the interpreter used to execute this script. set e By default, a shell script containing a command that fails will not cause the entire shell script to exit: the shell script will just continue on to the next line. We always want errors to be loud and noticeable. This option prevents this, by terminating the script if any command exited with a nonzero exit status.
31 A robust Bash header Note that this option ignores nonzero statuses in if conditionals. Also, it ignores all exit statuses in Unix pipes except the last one. set u This option fixes another default behavior of Bash scripts: any command containing a reference to an unset variable name will still run. It prevents this type of error by aborting the script if a variable s value is unset
32 A robust Bash header set o pipefail set e will cause a script to abort if a nonzero exit status is encountered, with some exceptions. One such exception is if a program runs in a Unix pipe exited unsuccessfully. Including set o pipefail will prevent this undesirable behavior: any program that returns a nonzero exit status in the pipe will cause the entire pipe to return a nonzero status. With set e enabled, this will lead the script to abort.
33 Running bash scripts Running Bash scripts can be done one of two ways: 1. bash script.sh 2../script.sh While we can run any script, calling the script as an executable requires that it has executable permissions. We can set these using: chmod u+x script.sh This adds executable permissions for the user who owns the file. Then, the script can be run with./script.sh.
34 Variables Pipelines have numerous settings that should be stored in variables. Storing these settings in a variable defined at the top of the file makes adjusting settings and rerunning your pipelines much easier. Rather than having to changes numerous hardcoded values in your scripts, using variables to store settings means you only have to change one value. Bash also reads command-line arguments into variables.
35 Variables Bash s variables don t have data types. It s helpful to think of Bash s variables as strings. We can create a variable and assign it a value with. results_dir= results/ Note that spaces matter when setting Bash variables. Do not use spaces around the equal sign.
36 Variables To access a variable s value, we use a dollar sign in front of the variable s name. Suppose we want to create a directory for a sample s alignment data, called <sample>_aln/, where <sample> is replaced by the sample s name. sample= CNTRL01A mkdir ${sample}_aln/
37 Command-line arguments The variable $0 stores the name of the script, and commandline arguments are assigned to the value $1, $2, $3, etc. Bash assigns the number of command-line arguments to $#.
38 Command-line arguments If you find your script requires numerous or complicated options, it might be easier to use Python instead of Bash. Python s argparse module is much easier to use. Variables created in your Bash script will only be available for the duration of the Bash process running that script.
39 if statement Bash supports the standard if conditional statement. The basic syntax is: if [commands] then else fi [if-statements] [else-statements]
40 if statement A command s exit status provides the true and false. Remember that 0 represents true/success and anything else if false/failure. if [commands] [commands] could be any command, set of commands, pipeline, or test condition. If the exit status of these commands is 0, execution continues to the block after then. Otherwise execution continues to the block after else.
41 if statement [if-statements] is a placeholder for all statements executed if [commands] evaluates to true (0). [else-statements] is a placeholder for all statements executed if [commands] evaluates to false. The else block is optional.
42 if statement Bash is primarily designed to stitch together other commands. This is an advantage Bash has over Python when writing pipelines. Bash allows your scripts to directly work with command-line programs without requiring any overhead to call programs. Although it can be unpleasant to write complicated programs in Bash, writing simple programs is exceedingly easy because Unix tools and Bash harmonize well.
43 if statement Suppose we wanted to run a set of commands only if a file contains a certain string. Because grep returns 0 only if it matches a pattern in a file and 1 otherwise. The redirection is to tidy the output of this script such that grep s output is redirected to /dev/null and not to the script s standard out.
44 test Like other programs, test exits with either 0 or 1. However test s exit status indicates the return value of the test specified through its arguments, rather than exit success or error. test supports numerous standard comparison operators.
45 test String/integer Description -z str String str is null str1 = str2 str1 and str2 are identical str1!= str2 str1 and str2 are different int1 eq int2 Integers int1 and int2 are equal int1 ne int2 int1 and int2 are not equal int1 lt int2 int1 is less than int2 int1 gt int2 int1 is greater than int2 int1 le int2 int1 is less than or equal to int2 int1 ge int2 int1 is greater than or equal to int2
46 test In practice, the most common conditions you ll be checking are to see if files or directories exist and whether you can write to them. test supports numerous file- and directoryrelated test operations.
47 test File/directory expression Description -d dir dir is a directory -f file file is a file -e file file exists -r file file is readable -w file file is writable -x file file is executable
48 test Combining test with if statements is simple: if test f some_file.txt then [ ] fi Bash provides a simpler syntactic alternative: if [ f some_file.txt ] then [ ] fi Note the spaces around and within the brackets: these are required.
49 test When using this syntax, we can chain test expression with a as logical AND, -o as logical OR,! as negation. Our familiar && and operators won t work in test, because these are shell operators. if [ $# ne 1 o! r $1 ] then fi echo usage: script.sh file_in.txt
50 for loop In bioinformatics, most of our data is split across multiple files. At the heart of any processing pipeline is some way to apply the same workflow to each of these files, taking care to keep track of sample names. Looping over files with Bash s for loop is the simplest way to accomplish this. There are three essential parts to creating a pipeline to process a set of files: 1. Selecting which files to apply the commands to 2. Looping over the data and applying the commands 3. Keeping track of the names of any output files created
51 for loop Suppose we have a file called samples.txt that tells you basic information about your raw data: sample name, read pair, and where the file is.
52 for loop Suppose we want to loop over every file, gather quality statistics on each and every file, and save this information to an output file. First, we load our filenames into a Bash array, which we can then loop over. Bash arrays can be created manually using:
53 for loop But creating Bash arrays by hand is tedious and error prone. The beauty of Bash is that we can use a command substitution to construct Bash arrays. We can strip the path and extension from each filename using basename.
54 for loop
55 Learning Unix
Essential Skills for Bioinformatics: Unix/Linux
Essential Skills for Bioinformatics: Unix/Linux SHELL SCRIPTING Overview Bash, the shell we have used interactively in this course, is a full-fledged scripting language. Unlike Python, Bash is not a general-purpose
More informationEssentials for Scientific Computing: Bash Shell Scripting Day 3
Essentials for Scientific Computing: Bash Shell Scripting Day 3 Ershaad Ahamed TUE-CMS, JNCASR May 2012 1 Introduction In the previous sessions, you have been using basic commands in the shell. The bash
More informationShell Programming (bash)
Shell Programming Shell Programming (bash) Commands run from a file in a subshell A great way to automate a repeated sequence of commands. File starts with #!/bin/bash absolute path to the shell program
More information9.2 Linux Essentials Exam Objectives
9.2 Linux Essentials Exam Objectives This chapter will cover the topics for the following Linux Essentials exam objectives: Topic 3: The Power of the Command Line (weight: 10) 3.3: Turning Commands into
More informationScripting. More Shell Scripts. Adapted from Practical Unix and Programming Hunter College
Scripting More Shell Scripts Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss Back to shell scripts Now that you've learned a few commands and can edit files,
More informationShells & Shell Programming (Part B)
Shells & Shell Programming (Part B) Software Tools EECS2031 Winter 2018 Manos Papagelis Thanks to Karen Reid and Alan J Rosenthal for material in these slides CONTROL STATEMENTS 2 Control Statements Conditional
More informationShells and Shell Programming
Shells and Shell Programming 1 Shells A shell is a command line interpreter that is the interface between the user and the OS. The shell: analyzes each command determines what actions are to be performed
More informationLecture 3. Essential skills for bioinformatics: Unix/Linux
Lecture 3 Essential skills for bioinformatics: Unix/Linux RETRIEVING DATA Overview Whether downloading large sequencing datasets or accessing a web application hundreds of times to download specific files,
More informationShells and Shell Programming
Shells and Shell Programming Shells A shell is a command line interpreter that is the interface between the user and the OS. The shell: analyzes each command determines what actions are to be performed
More informationCS Unix Tools & Scripting
Cornell University, Spring 2014 1 February 24, 2014 1 Slides evolved from previous versions by Hussam Abu-Libdeh and David Slater A note on awk for (item in array) The order in which items are returned
More informationCOMP 4/6262: Programming UNIX
COMP 4/6262: Programming UNIX Lecture 12 shells, shell programming: passing arguments, if, debug March 13, 2006 Outline shells shell programming passing arguments (KW Ch.7) exit status if (KW Ch.8) test
More informationBash scripting basics
Bash scripting basics prepared by Anatoliy Antonov for ESSReS community September 2012 1 Outline Definitions Foundations Flow control References and exercises 2 Definitions 3 Definitions Script - [small]
More informationShell scripting and system variables. HORT Lecture 5 Instructor: Kranthi Varala
Shell scripting and system variables HORT 59000 Lecture 5 Instructor: Kranthi Varala Text editors Programs built to assist creation and manipulation of text files, typically scripts. nano : easy-to-learn,
More informationA Big Step. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers
A Big Step Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Copyright 2006 2009 Stewart Weiss What a shell really does Here is the scoop on shells. A shell is a program
More informationGrep and Shell Programming
Grep and Shell Programming Comp-206 : Introduction to Software Systems Lecture 7 Alexandre Denault Computer Science McGill University Fall 2006 Teacher's Assistants Michael Hawker Monday, 14h30 to 16h30
More informationScripting. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers
Scripting Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss What a shell
More informationContents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
More informationLinux Shell Scripting. Linux System Administration COMP2018 Summer 2017
Linux Shell Scripting Linux System Administration COMP2018 Summer 2017 What is Scripting? Commands can be given to a computer by entering them into a command interpreter program, commonly called a shell
More informationTable of contents. Our goal. Notes. Notes. Notes. Summer June 29, Our goal is to see how we can use Unix as a tool for developing programs
Summer 2010 Department of Computer Science and Engineering York University Toronto June 29, 2010 1 / 36 Table of contents 1 2 3 4 2 / 36 Our goal Our goal is to see how we can use Unix as a tool for developing
More informationUseful Unix Commands Cheat Sheet
Useful Unix Commands Cheat Sheet The Chinese University of Hong Kong SIGSC Training (Fall 2016) FILE AND DIRECTORY pwd Return path to current directory. ls List directories and files here. ls dir List
More informationPractical Linux examples: Exercises
Practical Linux examples: Exercises 1. Login (ssh) to the machine that you are assigned for this workshop (assigned machines: https://cbsu.tc.cornell.edu/ww/machines.aspx?i=87 ). Prepare working directory,
More informationVi & Shell Scripting
Vi & Shell Scripting Comp-206 : Introduction to Week 3 Joseph Vybihal Computer Science McGill University Announcements Sina Meraji's office hours Trottier 3rd floor open area Tuesday 1:30 2:30 PM Thursday
More information5/8/2012. Exploring Utilities Chapter 5
Exploring Utilities Chapter 5 Examining the contents of files. Working with the cut and paste feature. Formatting output with the column utility. Searching for lines containing a target string with grep.
More informationOPERATING SYSTEMS LAB LAB # 6. I/O Redirection and Shell Programming. Shell Programming( I/O Redirection and if-else Statement)
P a g e 1 OPERATING SYSTEMS LAB LAB 6 I/O Redirection and Shell Programming Lab 6 Shell Programming( I/O Redirection and if-else Statement) P a g e 2 Redirection of Standard output/input i.e. Input - Output
More informationCSCI 211 UNIX Lab. Shell Programming. Dr. Jiang Li. Jiang Li, Ph.D. Department of Computer Science
CSCI 211 UNIX Lab Shell Programming Dr. Jiang Li Why Shell Scripting Saves a lot of typing A shell script can run many commands at once A shell script can repeatedly run commands Help avoid mistakes Once
More informationA shell can be used in one of two ways:
Shell Scripting 1 A shell can be used in one of two ways: A command interpreter, used interactively A programming language, to write shell scripts (your own custom commands) 2 If we have a set of commands
More informationUnix as a Platform Exercises + Solutions. Course Code: OS 01 UNXPLAT
Unix as a Platform Exercises + Solutions Course Code: OS 01 UNXPLAT Working with Unix Most if not all of these will require some investigation in the man pages. That's the idea, to get them used to looking
More informationShell Start-up and Configuration Files
ULI101 Week 10 Lesson Overview Shell Start-up and Configuration Files Shell History Alias Statement Shell Variables Introduction to Shell Scripting Positional Parameters echo and read Commands if and test
More informationEECS 470 Lab 5. Linux Shell Scripting. Friday, 1 st February, 2018
EECS 470 Lab 5 Linux Shell Scripting Department of Electrical Engineering and Computer Science College of Engineering University of Michigan Friday, 1 st February, 2018 (University of Michigan) Lab 5:
More informationCommand Interpreters. command-line (e.g. Unix shell) On Unix/Linux, bash has become defacto standard shell.
Command Interpreters A command interpreter is a program that executes other programs. Aim: allow users to execute the commands provided on a computer system. Command interpreters come in two flavours:
More informationBash scripting Tutorial. Hello World Bash Shell Script. Super User Programming & Scripting 22 March 2013
Bash scripting Tutorial Super User Programming & Scripting 22 March 2013 Hello World Bash Shell Script First you need to find out where is your bash interpreter located. Enter the following into your command
More informationIntroduction Variables Helper commands Control Flow Constructs Basic Plumbing. Bash Scripting. Alessandro Barenghi
Bash Scripting Alessandro Barenghi Dipartimento di Elettronica, Informazione e Bioingegneria Politecnico di Milano alessandro.barenghi - at - polimi.it April 28, 2015 Introduction The bash command shell
More informationsottotitolo A.A. 2016/17 Federico Reghenzani, Alessandro Barenghi
Titolo presentazione Piattaforme Software per la Rete sottotitolo BASH Scripting Milano, XX mese 20XX A.A. 2016/17, Alessandro Barenghi Outline 1) Introduction to BASH 2) Helper commands 3) Control Flow
More informationChapter 4. Unix Tutorial. Unix Shell
Chapter 4 Unix Tutorial Users and applications interact with hardware through an operating system (OS). Unix is a very basic operating system in that it has just the essentials. Many operating systems,
More informationLING 408/508: Computational Techniques for Linguists. Lecture 5
LING 408/508: Computational Techniques for Linguists Lecture 5 Last Time Installing Ubuntu 18.04 LTS on top of VirtualBox Your Homework 2: did everyone succeed? Ubuntu VirtualBox Host OS: MacOS or Windows
More informationBasic Linux (Bash) Commands
Basic Linux (Bash) Commands Hint: Run commands in the emacs shell (emacs -nw, then M-x shell) instead of the terminal. It eases searching for and revising commands and navigating and copying-and-pasting
More informationShell programming. Introduction to Operating Systems
Shell programming Introduction to Operating Systems Environment variables Predened variables $* all parameters $# number of parameters $? result of last command $$ process identier $i parameter number
More informationShell Scripting. Todd Kelley CST8207 Todd Kelley 1
Shell Scripting Todd Kelley kelleyt@algonquincollege.com CST8207 Todd Kelley 1 If we have a set of commands that we want to run on a regular basis, we could write a script A script acts as a Linux command,
More informationModule 8 Pipes, Redirection and REGEX
Module 8 Pipes, Redirection and REGEX Exam Objective 3.2 Searching and Extracting Data from Files Objective Summary Piping and redirection Partial POSIX Command Line and Redirection Command Line Pipes
More informationUNIX II:grep, awk, sed. October 30, 2017
UNIX II:grep, awk, sed October 30, 2017 File searching and manipulation In many cases, you might have a file in which you need to find specific entries (want to find each case of NaN in your datafile for
More informationhttp://xkcd.com/208/ 1. Review of pipes 2. Regular expressions 3. sed 4. awk 5. Editing Files 6. Shell loops 7. Shell scripts cat seqs.fa >0! TGCAGGTATATCTATTAGCAGGTTTAATTTTGCCTGCACTTGGTTGGGTACATTATTTTAAGTGTATTTGACAAG!
More informationUnix as a Platform Exercises. Course Code: OS-01-UNXPLAT
Unix as a Platform Exercises Course Code: OS-01-UNXPLAT Working with Unix 1. Use the on-line manual page to determine the option for cat, which causes nonprintable characters to be displayed. Run the command
More informationBourne Shell Reference
> Linux Reviews > Beginners: Learn Linux > Bourne Shell Reference Bourne Shell Reference found at Br. David Carlson, O.S.B. pages, cis.stvincent.edu/carlsond/cs330/unix/bshellref - Converted to txt2tags
More informationBasics. I think that the later is better.
Basics Before we take up shell scripting, let s review some of the basic features and syntax of the shell, specifically the major shells in the sh lineage. Command Editing If you like vi, put your shell
More informationBASH SHELL SCRIPT 1- Introduction to Shell
BASH SHELL SCRIPT 1- Introduction to Shell What is shell Installation of shell Shell features Bash Keywords Built-in Commands Linux Commands Specialized Navigation and History Commands Shell Aliases Bash
More informationScripting. More Shell Scripts Loops. Adapted from Practical Unix and Programming Hunter College
Scripting More Shell Scripts Loops Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss Loops: for The for script version 1: 1 #!/bin/bash 2 for i in 1 2 3 3 do
More informationShell Programming Overview
Overview Shell programming is a way of taking several command line instructions that you would use in a Unix command prompt and incorporating them into one program. There are many versions of Unix. Some
More informationWhen talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used:
Linux Tutorial How to read the examples When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used: $ application file.txt
More informationCOMS 6100 Class Notes 3
COMS 6100 Class Notes 3 Daniel Solus September 1, 2016 1 General Remarks The class was split into two main sections. We finished our introduction to Linux commands by reviewing Linux commands I and II
More informationIntroduction to Linux
Introduction to Linux The command-line interface A command-line interface (CLI) is a type of interface, that is, a way to interact with a computer. Window systems, punched cards or a bunch of dials, buttons
More informationCOMP 2718: Shell Scripts: Part 1. By: Dr. Andrew Vardy
COMP 2718: Shell Scripts: Part 1 By: Dr. Andrew Vardy Outline Shell Scripts: Part 1 Hello World Shebang! Example Project Introducing Variables Variable Names Variable Facts Arguments Exit Status Branching:
More informationLast Time. on the website
Last Time on the website Lecture 6 Shell Scripting What is a shell? The user interface to the operating system Functionality: Execute other programs Manage files Manage processes Full programming language
More informationUnix Shell scripting. Dr Alun Moon 7th October Introduction. Notation. Spaces
Unix Shell scripting Dr Alun Moon 7th October 2017 Introduction Shell scripts in Unix are a very powerfull tool, they form much of the standard system as installed. What are these good for? So many file
More informationIntroduction: What is Unix?
Introduction Introduction: What is Unix? An operating system Developed at AT&T Bell Labs in the 1960 s Command Line Interpreter GUIs (Window systems) are now available Introduction: Unix vs. Linux Unix
More informationShell script. Shell Scripts. A shell script contains a sequence of commands in a text file. Shell is an command language interpreter.
Shell Scripts A shell script contains a sequence of commands in a text file. Shell is an command language interpreter. Shell executes commands read from a file. Shell is a powerful programming available
More informationINTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT
INTRODUCTION TO SHELL SCRIPTING Dr. Jeffrey Frey University of Delaware, IT GOALS BASIC What is shell scripting? What makes shell scripting useful? What should you know already GOALS PART 1 Script layout/design
More informationComputer Systems and Architecture
Computer Systems and Architecture UNIX Scripting Bart Meyers University of Antwerp August 29, 2012 Outline Basics Conditionals Loops Advanced Exercises Shell scripts Grouping commands into a single file
More informationReview of Fundamentals
Review of Fundamentals 1 The shell vi General shell review 2 http://teaching.idallen.com/cst8207/14f/notes/120_shell_basics.html The shell is a program that is executed for us automatically when we log
More informationAssignment 3, Due October 4
Assignment 3, Due October 4 1 Summary This assignment gives you practice with writing shell scripts. Shell scripting is also known as bash programming. Your shell is bash, and when you write a shell script
More information1. Hello World Bash Shell Script. Last Updated on Wednesday, 13 April :03
1 of 18 21/10/2554 9:39 Bash scripting Tutorial tar -czf myhome_directory.tar.gz /home/linuxcong Last Updated on Wednesday, 13 April 2011 08:03 Article Index 1. Hello World Bash Shell Script 2. Simple
More informationTitle:[ Variables Comparison Operators If Else Statements ]
[Color Codes] Environmental Variables: PATH What is path? PATH=$PATH:/MyFolder/YourStuff?Scripts ENV HOME PWD SHELL PS1 EDITOR Showing default text editor #!/bin/bash a=375 hello=$a #No space permitted
More informationCSCI 2132: Software Development. Norbert Zeh. Faculty of Computer Science Dalhousie University. Shell Scripting. Winter 2019
CSCI 2132: Software Development Shell Scripting Norbert Zeh Faculty of Computer Science Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your Shell vs Your File Manager File manager
More informationexample: name1=jan name2=mike export name1 In this example, name1 is an environmental variable while name2 is a local variable.
Bourne Shell Programming Variables - creating and assigning variables Bourne shell use the set and unset to create and assign values to variables or typing the variable name, an equal sign and the value
More informationSimple Shell Scripting for Scientists
Simple Shell Scripting for Scientists Day Three Julian King Bruce Beckles University of Cambridge Computing Service 1 Introduction Who:! Julian King, Unix Support, UCS! Bruce Beckles, e-science Specialist,
More informationCisco IOS Shell. Finding Feature Information. Prerequisites for Cisco IOS.sh. Last Updated: December 14, 2012
Cisco IOS Shell Last Updated: December 14, 2012 The Cisco IOS Shell (IOS.sh) feature provides shell scripting capability to the Cisco IOS command-lineinterface (CLI) environment. Cisco IOS.sh enhances
More informationhttp://xkcd.com/208/ 1. Review of pipes 2. Regular expressions 3. sed 4. Editing Files 5. Shell loops 6. Shell scripts cat seqs.fa >0! TGCAGGTATATCTATTAGCAGGTTTAATTTTGCCTGCACTTGGTTGGGTACATTATTTTAAGTGTATTTGACAAG!
More informationIntroduction to Shell Scripting
Introduction to Shell Scripting Lecture 1. Shell scripts are small programs. They let you automate multi-step processes, and give you the capability to use decision-making logic and repetitive loops. 2.
More informationAnswers to AWK problems. Shell-Programming. Future: Using loops to automate tasks. Download and Install: Python (Windows only.) R
Today s Class Answers to AWK problems Shell-Programming Using loops to automate tasks Future: Download and Install: Python (Windows only.) R Awk basics From the command line: $ awk '$1>20' filename Command
More informationbash Execution Control COMP2101 Winter 2019
bash Execution Control COMP2101 Winter 2019 Bash Execution Control Scripts commonly can evaluate situations and make simple decisions about actions to take Simple evaluations and actions can be accomplished
More informationUsing bash. Administrative Shell Scripting COMP2101 Fall 2017
Using bash Administrative Shell Scripting COMP2101 Fall 2017 Bash Background Bash was written to replace the Bourne shell The Bourne shell (sh) was not a good candidate for rewrite, so bash was a completely
More informationShell. SSE2034: System Software Experiment 3, Fall 2018, Jinkyu Jeong
Shell Prof. Jinkyu Jeong (Jinkyu@skku.edu) TA -- Minwoo Ahn (minwoo.ahn@csl.skku.edu) TA -- Donghyun Kim (donghyun.kim@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu
More informationSTP, Unix, SAC tutorial, Ge167 Winter 2014
STP, Unix, SAC tutorial, Ge167 Winter 2014 Asaf Inbal 1 Downloading waveforms In this tutorial we ll learn how to download waveforms using a tool called STP (Seismic Transfer Program) and manipulate them
More informationComputer Systems and Architecture
Computer Systems and Architecture Stephen Pauwels UNIX Scripting Academic Year 2018-2019 Outline Basics Conditionals Loops Advanced Exercises Shell Scripts Grouping commands into a single file Reusability
More informationBash command shell language interpreter
Principles of Programming Languages Bash command shell language interpreter Advanced seminar topic Louis Sugy & Baptiste Thémine Presentation on December 8th, 2017 Table of contents I. General information
More informationCSE 374: Programming Concepts and Tools. Eric Mullen Spring 2017 Lecture 4: More Shell Scripts
CSE 374: Programming Concepts and Tools Eric Mullen Spring 2017 Lecture 4: More Shell Scripts Homework 1 Already out, due Thursday night at midnight Asks you to run some shell commands Remember to use
More informationScripting Languages Course 1. Diana Trandabăț
Scripting Languages Course 1 Diana Trandabăț Master in Computational Linguistics - 1 st year 2017-2018 Today s lecture Introduction to scripting languages What is a script? What is a scripting language
More informationUnix/Linux Primer. Taras V. Pogorelov and Mike Hallock School of Chemical Sciences, University of Illinois
Unix/Linux Primer Taras V. Pogorelov and Mike Hallock School of Chemical Sciences, University of Illinois August 25, 2017 This primer is designed to introduce basic UNIX/Linux concepts and commands. No
More informationCSE 374 Programming Concepts & Tools. Brandon Myers Winter 2015 Lecture 4 Shell Variables, More Shell Scripts (Thanks to Hal Perkins)
CSE 374 Programming Concepts & Tools Brandon Myers Winter 2015 Lecture 4 Shell Variables, More Shell Scripts (Thanks to Hal Perkins) test / if Recall from last lecture: test (not built-in) takes arguments
More informationCS 25200: Systems Programming. Lecture 10: Shell Scripting in Bash
CS 25200: Systems Programming Lecture 10: Shell Scripting in Bash Dr. Jef Turkstra 2018 Dr. Jeffrey A. Turkstra 1 Lecture 10 Getting started with Bash Data types Reading and writing Control loops Decision
More informationSHELL SCRIPT BASIC. UNIX Programming 2014 Fall by Euiseong Seo
SHELL SCRIPT BASIC UNIX Programming 2014 Fall by Euiseong Seo Shell Script Interactive shell sequentially executes a series of commands Some tasks are repetitive and automatable They are what programs
More informationIntroduction to UNIX. Logging in. Basic System Architecture 10/7/10. most systems have graphical login on Linux machines
Introduction to UNIX Logging in Basic system architecture Getting help Intro to shell (tcsh) Basic UNIX File Maintenance Intro to emacs I/O Redirection Shell scripts Logging in most systems have graphical
More informationEssential Skills for Bioinformatics: Unix/Linux
Essential Skills for Bioinformatics: Unix/Linux WORKING WITH COMPRESSED DATA Overview Data compression, the process of condensing data so that it takes up less space (on disk drives, in memory, or across
More informationCSC 2500: Unix Lab Fall 2016
CSC 2500: Unix Lab Fall 2016 Control Statements in Shell Scripts: Decision Mohammad Ashiqur Rahman Department of Computer Science College of Engineering Tennessee Tech University Agenda User Input Special
More informationOpen up a terminal, make sure you are in your home directory, and run the command.
More Linux Commands 0.1 wc The Linux command for acquiring size statistics on a file is wc. This command can provide information from line count, to bytes in a file. Open up a terminal, make sure you are
More informationMerge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p.
Preface p. xiii Ideology: Data Skills for Robust and Reproducible Bioinformatics How to Learn Bioinformatics p. 1 Why Bioinformatics? Biology's Growing Data p. 1 Learning Data Skills to Learn Bioinformatics
More informationIntroduction to UNIX command-line II
Introduction to UNIX command-line II Boyce Thompson Institute 2017 Prashant Hosmani Class Content Terminal file system navigation Wildcards, shortcuts and special characters File permissions Compression
More informationA Brief Introduction to the Linux Shell for Data Science
A Brief Introduction to the Linux Shell for Data Science Aris Anagnostopoulos 1 Introduction Here we will see a brief introduction of the Linux command line or shell as it is called. Linux is a Unix-like
More informationThere are some string operators that can be used in the test statement to perform string comparison.
ACS-294- Unix (Winter Term, 26-7) Part II: Shell Programming Page 3 The test also returns a nonzero exit value if there is no argument: test String Operators There are some string operators that can be
More informationAssignment clarifications
Assignment clarifications How many errors to print? at most 1 per token. Interpretation of white space in { } treat as a valid extension, involving white space characters. Assignment FAQs have been updated.
More informationC Shell Tutorial. Section 1
C Shell Tutorial Goals: Section 1 Learn how to write a simple shell script and how to run it. Learn how to use local and global variables. About CSH The Barkley Unix C shell was originally written with
More informationSHELL SCRIPT BASIC. UNIX Programming 2015 Fall by Euiseong Seo
SHELL SCRIPT BASIC UNIX Programming 2015 Fall by Euiseong Seo Shell Script! Interactive shell sequentially executes a series of commands! Some tasks are repetitive and automatable! They are what programs
More informationCSCI 2132: Software Development
CSCI 2132: Software Development Lab 4/5: Shell Scripting Synopsis In this lab, you will: Learn to work with command-line arguments in shell scripts Learn to capture command output in variables Learn to
More informationReview of Fundamentals. Todd Kelley CST8207 Todd Kelley 1
Review of Fundamentals Todd Kelley kelleyt@algonquincollege.com CST8207 Todd Kelley 1 GPL the shell SSH (secure shell) the Course Linux Server RTFM vi general shell review 2 These notes are available on
More informationLab #2 Physics 91SI Spring 2013
Lab #2 Physics 91SI Spring 2013 Objective: Some more experience with advanced UNIX concepts, such as redirecting and piping. You will also explore the usefulness of Mercurial version control and how to
More informationEssentials for Scientific Computing: Stream editing with sed and awk
Essentials for Scientific Computing: Stream editing with sed and awk Ershaad Ahamed TUE-CMS, JNCASR May 2012 1 Stream Editing sed and awk are stream processing commands. What this means is that they are
More informationUNIX shell scripting
UNIX shell scripting EECS 2031 Summer 2014 Przemyslaw Pawluk June 17, 2014 What we will discuss today Introduction Control Structures User Input Homework Table of Contents Introduction Control Structures
More informationTopic 4: Grep, Find & Sed
Topic 4: Grep, Find & Sed grep: a tool for searching for strings within files find: a tool for examining a directory tree sed: a tool for "batch editing" Associated topic: regular expressions 1 Motivation
More informationUnleashing the Shell Hands-On UNIX System Administration DeCal Week 6 28 February 2011
Unleashing the Shell Hands-On UNIX System Administration DeCal Week 6 28 February 2011 Last time Compiling software and the three-step procedure (./configure && make && make install). Dependency hell and
More informationПрограмиранев UNIX среда
Програмиранев UNIX среда Използванена команден шел и създаванена скриптове: tcsh, bash, awk, python Shell programming As well as using the shell to run commands you can use its built-in programming language
More informationWhat is Bash Shell Scripting?
What is Bash Shell Scripting? A shell script is a script written for the shell, or command line interpreter, of an operating system. The shell is often considered a simple domain-specic programming language.
More information