Essentials for Scientific Computing: Stream editing with sed and awk
|
|
- Joleen Gloria Copeland
- 6 years ago
- Views:
Transcription
1 Essentials for Scientific Computing: Stream editing with sed and awk Ershaad Ahamed TUE-CMS, JNCASR May Stream Editing sed and awk are stream processing commands. What this means is that they are programs that can accept input text, transform the text data and write it to the output. Thus, these programs can be part of a shell pipeline much in the same way as uniq, nl and sort, which you have seen earlier, and which also accept input, perform some transformation and write the result to output. The difference lies in the fact that, while commands like uniq and sort perform a predefined transformation of the input, sed and awk are programmable. They have their own languages that can be used to specify rules and transformations that must be performed on the input. This makes them powerful and flexible tools that can perform complex transformations and can be used as part of a shell pipeline. 2 Regexes and Metacharacters As we progress through the sections below, we will be using patterns, where certain characters have special meanings. Although some of the characters might be familiar from our earlier discussion on glob expressions, their meanings are not the same and should not be confused with glob expression syntax. These special characters are referred to as metacharacters, and they are used to build patterns called Regular Expressions or Regex for short. While glob expressions are used to create patterns that match pathnames, regular expressions are much more extensive and can be used to match and manipulate textual data in general. The most commonly used regular expression metacharacters are *,., +, ^, $, and parentheses () among others. You might see that for many of the metacharacters, we precede them by a \, this is called escaping and we do it inform the interpreter that the character should be interpreted as a special symbol and not literally. 1
2 3 sed 3.1 The s Command One of the most common uses of sed is to replace one string with another. Consider the following text file. Teh war of the worlds, teh day of teh year This is the third line We want to replace all occurrences of the typo teh with the. To do that we use the following sed command. cat text.txt sed -e s/teh/the/ nl In the command line above, cat reads the file text.txt containing our text and writes it to stdout. Since we are using the pipe to connect it to sed, the data written to stdout is redirected to the stdin of the sed command. The -e option to sed tells the sed command that the argument following the -e should be interpreted as sed commands. In this example the sed commands or script is s/teh/the/. Here s is the sed substitution command. The pattern between the first set of /s is replaced with the string between the second set of /s. Here the pattern to replace is the literal string teh. As a convenience, we also pipe the output of sed through nl so that we get line numbers. The sed command operates by reading in each line of the input, applying the commands specified (here, the s command) and then printing out the modified line. This is done for each line of the input, until the input file ends. The output of this example will be. 1 Teh war of the worlds, the day of teh year We have a few observations to make here. 1. The word Teh on the line 1 was not substituted. This is because Teh (with an uppercase T ) will not match the pattern teh that we specified for the s command. 2. Only the first occurrence of teh on line 1 and line 2 was replaced. This is the default behaviour for the s command 3. The teh present in the word Statehouse on line 2 is also substituted with the Let us try to fix the problem in item 1. The s command of sed accepts certain flags after the final /. These flags modify the functioning of the s command. One of these flags is i which makes the pattern matching case insensitive. cat text.txt sed -e s/teh/the/i nl The output is now. 2
3 1 the war of the worlds, teh day of teh year The Teh has been replaced, but since the replacement string is the (with a lowercase t ) we have an incorrect case for the replacement. There are a few ways in which we can work around this. One way is to capture the match. For instance, in the example above, our sed command can match Teh, teh, TEH or any other combination of upper and lower case since we have specified a case insensitive match. When sed finds a match, we can store the actual string matched since it can be any of the variants above. We do this by enclosing the part of the pattern we are interested in capturing in capturing parentheses \( and \). Our pattern will now look like. \(t\)eh This means that if the t in our pattern matches a t in the actual input, t is captured. Else, if a T is matched, T is captured. Now what we need to do is to place the captured t or T in our replacement string. We can refer to text that was captured using capturing parentheses inside the replacement string by using \1, \2, etc.,which refers to the first, second, etc. capturing parenthesis. In our example above, \1 will contain either t or T after a match. So our new command will look like. cat text.txt sed -e s/\(t\)eh/\1he/i nl Output is now. 1 The war of the worlds, teh day of teh year Moving on to observation 2. This default behaviour of the s command can be modified by passing the g flag, which tells sed to replace all occurrences of the match on each line. Making our script. cat text.txt sed -e s/\(t\)eh/\1he/ig nl Output is. 1 The war of the worlds, the day of the year 2 Statheouse has the in it Moving on to item 3. We need to tell sed that it should not replace teh if it is a substring, that is, it is part of a larger word. We do this by placing the word boundary pattern \b on either side of the word we would like to match (here teh). \b represents a word boundary, that is, a non-word character followed by a word character, or vice-versa. Word characters are alphabets, digits and the underscore character. Now are script is. cat text.txt sed -e s/\b\(t\)eh\b/\1he/ig nl Output being. 1 The war of the worlds, the day of the year 2 Statehouse has the in it Which looks good. 3
4 4 Some Examples 4.1 Repeated words Here s an example of a text file having repeated words. The war of the the worlds, the day of the year This this is the third third line Lets start by writing a pattern to match any complete word. You can use a pattern like below. \b\w\+\b Remember that \b is for a word boundary. \w is a pattern that matches any word character (alphabets, digits and underscore). The \+ pattern means to match one or more repetitions of the previous pattern, the previous pattern here being \w. That is followed by a closing \b. The complete expression therefore matches a word. Now we need to build on this pattern so that it can match the same word repeated again (with a space separating them). Remember that when we need to refer to a previous match, we need to first capture it and then we can use backreferences, which are \1, \2, etc. \(\b\w\+\b\) \1 Notice the space between the word-match pattern and the backreference. Using the pattern in a sed script, we have. cat text_repeat.txt sed -e s/\(\b\w\+\b\) \1/\1/g The pattern matches a repeated word, but the capturing parentheses captures the first of the repeated words. Therefore in the replacement string we use the backreference \1. Output is. The war of the worlds, the day of the year This this is the third line Notice that, in the last line, the repeated word was not matched because of the difference in case. A quick fix for this will be to use the i flag. cat text_repeat.txt sed -e s/\(\b\w\+\b\) \1/\1/gi That fixes it. 4.2 Removing Empty Lines Consider a file with the text below. C C C
5 C B N B N B N C B N B N B N B N We need to remove the empty lines from the file. It may seem easy to do quickly in an editor, but what if the file had lines. You saw the s command for sed in the previous examples. Now, we will use the d command. Before that, a word on addresses in sed. We can precede a sed command with an address. This address can restrict the commands that follow to be executed only for those lines that satisfy that address. The simplest possible address is a line number. Consider this version of our earlier script for fixing the teh typo. cat text.txt sed -e 2s/\(t\)eh/\1he/ig The only difference being the 2 preceding the s command. This tells sed to execute the s command only for the second line in the input. Thus our output will be. Teh war of the worlds, teh day of teh year Statheouse has the in it This is the third line Suppose, we wanted all lines except the second to be processed. below would do what is expected. The script cat text.txt sed -e 2!s/\(t\)eh/\1he/ig Addresses can be of the form N,M which means the range from line N to line M, inclusive. 5
Essentials for Scientific Computing: Bash Shell Scripting Day 3
Essentials for Scientific Computing: Bash Shell Scripting Day 3 Ershaad Ahamed TUE-CMS, JNCASR May 2012 1 Introduction In the previous sessions, you have been using basic commands in the shell. The bash
More informationTable of contents. Our goal. Notes. Notes. Notes. Summer June 29, Our goal is to see how we can use Unix as a tool for developing programs
Summer 2010 Department of Computer Science and Engineering York University Toronto June 29, 2010 1 / 36 Table of contents 1 2 3 4 2 / 36 Our goal Our goal is to see how we can use Unix as a tool for developing
More informationComputer Systems and Architecture
Computer Systems and Architecture Stephen Pauwels Regular Expressions Academic Year 2018-2019 Outline What is a Regular Expression? Tools Anchors, Character sets and Modifiers Advanced Regular Expressions
More informationComputer Systems and Architecture
Computer Systems and Architecture Regular Expressions Bart Meyers University of Antwerp August 29, 2012 Outline What? Tools Anchors, character sets and modifiers Advanced Regular expressions Exercises
More informationITST Searching, Extracting & Archiving Data
ITST 1136 - Searching, Extracting & Archiving Data Name: Step 1 Sign into a Pi UN = pi PW = raspberry Step 2 - Grep - One of the most useful and versatile commands in a Linux terminal environment is the
More informationpsed [-an] script [file...] psed [-an] [-e script] [-f script-file] [file...]
NAME SYNOPSIS DESCRIPTION OPTIONS psed - a stream editor psed [-an] script [file...] psed [-an] [-e script] [-f script-file] [file...] s2p [-an] [-e script] [-f script-file] A stream editor reads the input
More informationRegular Expressions. Regular expressions are a powerful search-and-replace technique that is widely used in other environments (such as Unix and Perl)
Regular Expressions Regular expressions are a powerful search-and-replace technique that is widely used in other environments (such as Unix and Perl) JavaScript started supporting regular expressions in
More informationLecture 18 Regular Expressions
Lecture 18 Regular Expressions In this lecture Background Text processing languages Pattern searches with grep Formal Languages and regular expressions Finite State Machines Regular Expression Grammer
More informationBasic Linux (Bash) Commands
Basic Linux (Bash) Commands Hint: Run commands in the emacs shell (emacs -nw, then M-x shell) instead of the terminal. It eases searching for and revising commands and navigating and copying-and-pasting
More informationMore Scripting and Regular Expressions. Todd Kelley CST8207 Todd Kelley 1
More Scripting and Regular Expressions Todd Kelley kelleyt@algonquincollege.com CST8207 Todd Kelley 1 Regular Expression Summary Regular Expression Examples Shell Scripting 2 Do not confuse filename globbing
More informationRegex, Sed, Awk. Arindam Fadikar. December 12, 2017
Regex, Sed, Awk Arindam Fadikar December 12, 2017 Why Regex Lots of text data. twitter data (social network data) government records web scrapping many more... Regex Regular Expressions or regex or regexp
More informationLecture 3 Tonight we dine in shell. Hands-On Unix System Administration DeCal
Lecture 3 Tonight we dine in shell Hands-On Unix System Administration DeCal 2012-09-17 Review $1, $2,...; $@, $*, $#, $0, $? environment variables env, export $HOME, $PATH $PS1=n\[\e[0;31m\]\u\[\e[m\]@\[\e[1;34m\]\w
More informationBasics. I think that the later is better.
Basics Before we take up shell scripting, let s review some of the basic features and syntax of the shell, specifically the major shells in the sh lineage. Command Editing If you like vi, put your shell
More informationWildcards and Regular Expressions
CSCI 2132: Software Development Wildcards and Regular Expressions Norbert Zeh Faculty of Computer Science Dalhousie University Winter 2019 Searching Problem: Find all files whose names match a certain
More informationUNIX / LINUX - REGULAR EXPRESSIONS WITH SED
UNIX / LINUX - REGULAR EXPRESSIONS WITH SED http://www.tutorialspoint.com/unix/unix-regular-expressions.htm Copyright tutorialspoint.com Advertisements In this chapter, we will discuss in detail about
More informationBASH SHELL SCRIPT 1- Introduction to Shell
BASH SHELL SCRIPT 1- Introduction to Shell What is shell Installation of shell Shell features Bash Keywords Built-in Commands Linux Commands Specialized Navigation and History Commands Shell Aliases Bash
More informationTopic 4: Grep, Find & Sed
Topic 4: Grep, Find & Sed grep: a tool for searching for strings within files find: a tool for examining a directory tree sed: a tool for "batch editing" Associated topic: regular expressions 1 Motivation
More informationRegular Expressions. Michael Wrzaczek Dept of Biosciences, Plant Biology Viikki Plant Science Centre (ViPS) University of Helsinki, Finland
Regular Expressions Michael Wrzaczek Dept of Biosciences, Plant Biology Viikki Plant Science Centre (ViPS) University of Helsinki, Finland November 11 th, 2015 Regular expressions provide a flexible way
More informationGetting to grips with Unix and the Linux family
Getting to grips with Unix and the Linux family David Chiappini, Giulio Pasqualetti, Tommaso Redaelli Torino, International Conference of Physics Students August 10, 2017 According to the booklet At this
More information5/8/2012. Exploring Utilities Chapter 5
Exploring Utilities Chapter 5 Examining the contents of files. Working with the cut and paste feature. Formatting output with the column utility. Searching for lines containing a target string with grep.
More informationPattern Matching. An Introduction to File Globs and Regular Expressions. Adapted from Practical Unix and Programming Hunter College
Pattern Matching An Introduction to File Globs and Regular Expressions Adapted from Practical Unix and Programming Hunter College Copyright 2006 2009 Stewart Weiss The danger that lies ahead Much to your
More informationPattern Matching. An Introduction to File Globs and Regular Expressions
Pattern Matching An Introduction to File Globs and Regular Expressions Copyright 2006 2009 Stewart Weiss The danger that lies ahead Much to your disadvantage, there are two different forms of patterns
More informationRegular Expressions Explained
Found at: http://publish.ez.no/article/articleprint/11/ Regular Expressions Explained Author: Jan Borsodi Publishing date: 30.10.2000 18:02 This article will give you an introduction to the world of regular
More informationUnleashing the Shell Hands-On UNIX System Administration DeCal Week 6 28 February 2011
Unleashing the Shell Hands-On UNIX System Administration DeCal Week 6 28 February 2011 Last time Compiling software and the three-step procedure (./configure && make && make install). Dependency hell and
More informationSTREAM EDITOR - REGULAR EXPRESSIONS
STREAM EDITOR - REGULAR EXPRESSIONS http://www.tutorialspoint.com/sed/sed_regular_expressions.htm Copyright tutorialspoint.com It is the regular expressions that make SED powerful and efficient. A number
More informationCisco IOS Shell. Finding Feature Information. Prerequisites for Cisco IOS.sh. Last Updated: December 14, 2012
Cisco IOS Shell Last Updated: December 14, 2012 The Cisco IOS Shell (IOS.sh) feature provides shell scripting capability to the Cisco IOS command-lineinterface (CLI) environment. Cisco IOS.sh enhances
More informationBashed One Too Many Times. Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009
Bashed One Too Many Times Features of the Bash Shell St. Louis Unix Users Group Jeff Muse, Jan 14, 2009 What is a Shell? The shell interprets commands and executes them It provides you with an environment
More informationIntroduction to UNIX Part II
T H E U N I V E R S I T Y of T E X A S H E A L T H S C I E N C E C E N T E R A T H O U S T O N S C H O O L of H E A L T H I N F O R M A T I O N S C I E N C E S Introduction to UNIX Part II For students
More informationRegular Expressions. Regular Expression Syntax in Python. Achtung!
1 Regular Expressions Lab Objective: Cleaning and formatting data are fundamental problems in data science. Regular expressions are an important tool for working with text carefully and eciently, and are
More informationLecture 5. Essential skills for bioinformatics: Unix/Linux
Lecture 5 Essential skills for bioinformatics: Unix/Linux UNIX DATA TOOLS Text processing with awk We have illustrated two ways awk can come in handy: Filtering data using rules that can combine regular
More informationModule 8 Pipes, Redirection and REGEX
Module 8 Pipes, Redirection and REGEX Exam Objective 3.2 Searching and Extracting Data from Files Objective Summary Piping and redirection Partial POSIX Command Line and Redirection Command Line Pipes
More informationMotivation (Scenarios) Topic 4: Grep, Find & Sed. Displaying File Names. grep
Topic 4: Grep, Find & Sed grep: a tool for searching for strings within files find: a tool for examining a directory tree sed: a tool for "batch editing" Associated topic: regular expressions Motivation
More informationIntroduction p. 1 Who Should Read This Book? p. 1 What You Need to Know Before Reading This Book p. 2 How This Book Is Organized p.
Introduction p. 1 Who Should Read This Book? p. 1 What You Need to Know Before Reading This Book p. 2 How This Book Is Organized p. 2 Conventions Used in This Book p. 2 Introduction to UNIX p. 5 An Overview
More informationCS Unix Tools. Fall 2010 Lecture 5. Hussam Abu-Libdeh based on slides by David Slater. September 17, 2010
Fall 2010 Lecture 5 Hussam Abu-Libdeh based on slides by David Slater September 17, 2010 Reasons to use Unix Reason #42 to use Unix: Wizardry Mastery of Unix makes you a wizard need proof? here is the
More informationPerl Regular Expressions. Perl Patterns. Character Class Shortcuts. Examples of Perl Patterns
Perl Regular Expressions Unlike most programming languages, Perl has builtin support for matching strings using regular expressions called patterns, which are similar to the regular expressions used in
More informationRegular Expressions. Regular expressions match input within a line Regular expressions are very different than shell meta-characters.
ULI101 Week 09 Week Overview Regular expressions basics Literal matching.wildcard Delimiters Character classes * repetition symbol Grouping Anchoring Search Search and replace in vi Regular Expressions
More informationRegular Expressions 1
Regular Expressions 1 Basic Regular Expression Examples Extended Regular Expressions Extended Regular Expression Examples 2 phone number 3 digits, dash, 4 digits [[:digit:]][[:digit:]][[:digit:]]-[[:digit:]][[:digit:]][[:digit:]][[:digit:]]
More informationSTATS Data Analysis using Python. Lecture 15: Advanced Command Line
STATS 700-002 Data Analysis using Python Lecture 15: Advanced Command Line Why UNIX/Linux? As a data scientist, you will spend most of your time dealing with data Data sets never arrive ready to analyze
More informationEssentials for Scientific Computing: Source Code, Compilation and Libraries Day 8
Essentials for Scientific Computing: Source Code, Compilation and Libraries Day 8 Ershaad Ahamed TUE-CMS, JNCASR May 2012 1 Introduction In the first session we discussed instructions that the CPU processes
More informationCSCI 2132 Software Development. Lecture 7: Wildcards and Regular Expressions
CSCI 2132 Software Development Lecture 7: Wildcards and Regular Expressions Instructor: Vlado Keselj Faculty of Computer Science Dalhousie University 20-Sep-2017 (7) CSCI 2132 1 Previous Lecture Pipes
More informationShell scripting and system variables. HORT Lecture 5 Instructor: Kranthi Varala
Shell scripting and system variables HORT 59000 Lecture 5 Instructor: Kranthi Varala Text editors Programs built to assist creation and manipulation of text files, typically scripts. nano : easy-to-learn,
More informationBash Script. CIRC Summer School 2015 Baowei Liu
Bash Script CIRC Summer School 2015 Baowei Liu Filename Expansion / Globbing Expanding filenames containing special characters Wild cards *?, not include... Square brackets [set]: - Special characters:!
More informationCSE 390a Lecture 7. Regular expressions, egrep, and sed
CSE 390a Lecture 7 Regular expressions, egrep, and sed slides created by Marty Stepp, modified by Jessica Miller and Ruth Anderson http://www.cs.washington.edu/390a/ 1 2 Lecture summary regular expression
More informationIntroduction to Perl. c Sanjiv K. Bhatia. Department of Mathematics & Computer Science University of Missouri St. Louis St.
Introduction to Perl c Sanjiv K. Bhatia Department of Mathematics & Computer Science University of Missouri St. Louis St. Louis, MO 63121 Contents 1 Introduction 1 2 Getting started 1 3 Writing Perl scripts
More informationCommon File System Commands
Common File System Commands ls! List names of all files in current directory ls filenames! List only the named files ls -t! List in time order, most recent first ls -l! Long listing, more information.
More informationIntroduction to UNIX. Introduction. Processes. ps command. The File System. Directory Structure. UNIX is an operating system (OS).
Introduction Introduction to UNIX CSE 2031 Fall 2012 UNIX is an operating system (OS). Our goals: Learn how to use UNIX OS. Use UNIX tools for developing programs/ software, specifically shell programming.
More informationIntroduction to UNIX. CSE 2031 Fall November 5, 2012
Introduction to UNIX CSE 2031 Fall 2012 November 5, 2012 Introduction UNIX is an operating system (OS). Our goals: Learn how to use UNIX OS. Use UNIX tools for developing programs/ software, specifically
More informationCSCI 2132: Software Development
CSCI 2132: Software Development Lab 4/5: Shell Scripting Synopsis In this lab, you will: Learn to work with command-line arguments in shell scripts Learn to capture command output in variables Learn to
More informationDigital Humanities. Tutorial Regular Expressions. March 10, 2014
Digital Humanities Tutorial Regular Expressions March 10, 2014 1 Introduction In this tutorial we will look at a powerful technique, called regular expressions, to search for specific patterns in corpora.
More informationAdvanced Handle Definition
Tutorial for Windows and Macintosh Advanced Handle Definition 2017 Gene Codes Corporation Gene Codes Corporation 525 Avis Drive, Ann Arbor, MI 48108 USA 1.800.497.4939 (USA) +1.734.769.7249 (elsewhere)
More informationCS 246 Winter Tutorial 2
CS 246 Winter 2016 - Tutorial 2 Detailed Version January 14, 2016 1 Summary git Stuff File Properties Regular Expressions Output Redirection Piping Commands Basic Scripting Persistent Data Password-less
More informationUnderstanding Regular Expressions, Special Characters, and Patterns
APPENDIXA Understanding Regular Expressions, Special Characters, and Patterns This appendix describes the regular expressions, special or wildcard characters, and patterns that can be used with filters
More information22-Sep CSCI 2132 Software Development Lecture 8: Shells, Processes, and Job Control. Faculty of Computer Science, Dalhousie University
Lecture 8 p.1 Faculty of Computer Science, Dalhousie University CSCI 2132 Software Development Lecture 8: Shells, Processes, and Job Control 22-Sep-2017 Location: Goldberg CS 127 Time: 14:35 15:25 Instructor:
More informationC Shell Tutorial. Section 1
C Shell Tutorial Goals: Section 1 Learn how to write a simple shell script and how to run it. Learn how to use local and global variables. About CSH The Barkley Unix C shell was originally written with
More informationIB047. Unix Text Tools. Pavel Rychlý Mar 3.
Unix Text Tools pary@fi.muni.cz 2014 Mar 3 Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple tools, each tool doing only one operation Pipe (pipeline):
More informationQUESTION BANK ON UNIX & SHELL PROGRAMMING-502 (CORE PAPER-2)
BANK ON & SHELL PROGRAMMING-502 (CORE PAPER-2) TOPIC 1: VI-EDITOR MARKS YEAR 1. Explain set command of vi editor 2 2011oct 2. Explain the modes of vi editor. 7 2013mar/ 2013 oct 3. Explain vi editor 5
More informationReview of Fundamentals
Review of Fundamentals 1 The shell vi General shell review 2 http://teaching.idallen.com/cst8207/14f/notes/120_shell_basics.html The shell is a program that is executed for us automatically when we log
More informationRegular Expressions. Todd Kelley CST8207 Todd Kelley 1
Regular Expressions Todd Kelley kelleyt@algonquincollege.com CST8207 Todd Kelley 1 POSIX character classes Some Regular Expression gotchas Regular Expression Resources Assignment 3 on Regular Expressions
More informationINTRODUCTION TO SHELL SCRIPTING ITPART 2
INTRODUCTION TO SHELL SCRIPTING ITPART 2 Dr. Jeffrey Frey University of Delaware, version 2 GOALS PART 2 Shell plumbing review Standard files Redirection Pipes GOALS PART 2 Command substitution backticks
More informationLecture Outline. COMP-421 Compiler Design. What is Lex? Lex Specification. ! Lexical Analyzer Lex. ! Lex Examples. Presented by Dr Ioanna Dionysiou
Lecture Outline COMP-421 Compiler Design! Lexical Analyzer Lex! Lex Examples Presented by Dr Ioanna Dionysiou Figures and part of the lecture notes taken from A compact guide to lex&yacc, epaperpress.com
More informationRegular Expressions. with a brief intro to FSM Systems Skills in C and Unix
Regular Expressions with a brief intro to FSM 15-123 Systems Skills in C and Unix Case for regular expressions Many web applications require pattern matching look for tag for links Token search
More informationIntroduction to Regular Expressions Version 1.3. Tom Sgouros
Introduction to Regular Expressions Version 1.3 Tom Sgouros June 29, 2001 2 Contents 1 Beginning Regular Expresions 5 1.1 The Simple Version........................ 6 1.2 Difficult Characters........................
More informationls /data/atrnaseq/ egrep "(fastq fasta fq fa)\.gz" ls /data/atrnaseq/ egrep "(cn ts)[1-3]ln[^3a-za-z]\."
Command line tools - bash, awk and sed We can only explore a small fraction of the capabilities of the bash shell and command-line utilities in Linux during this course. An entire course could be taught
More informationUnix/Linux Primer. Taras V. Pogorelov and Mike Hallock School of Chemical Sciences, University of Illinois
Unix/Linux Primer Taras V. Pogorelov and Mike Hallock School of Chemical Sciences, University of Illinois August 25, 2017 This primer is designed to introduce basic UNIX/Linux concepts and commands. No
More informationMastering Modern Linux by Paul S. Wang Appendix: Pattern Processing with awk
Mastering Modern Linux by Paul S. Wang Appendix: Pattern Processing with awk The awk program is a powerful yet simple filter. It processes its input one line at a time, applying user-specified awk pattern
More informationFundamentals of Programming Session 4
Fundamentals of Programming Session 4 Instructor: Reza Entezari-Maleki Email: entezari@ce.sharif.edu 1 Fall 2011 These slides are created using Deitel s slides, ( 1992-2010 by Pearson Education, Inc).
More informationSystems Programming/ C and UNIX
Systems Programming/ C and UNIX December 7-10, 2017 1/17 December 7-10, 2017 1 / 17 Outline 1 2 Using find 2/17 December 7-10, 2017 2 / 17 String Pattern Matching Tools Regular Expressions Simple Examples
More informationAnswers to AWK problems. Shell-Programming. Future: Using loops to automate tasks. Download and Install: Python (Windows only.) R
Today s Class Answers to AWK problems Shell-Programming Using loops to automate tasks Future: Download and Install: Python (Windows only.) R Awk basics From the command line: $ awk '$1>20' filename Command
More informationIntroduction Variables Helper commands Control Flow Constructs Basic Plumbing. Bash Scripting. Alessandro Barenghi
Bash Scripting Alessandro Barenghi Dipartimento di Elettronica, Informazione e Bioingegneria Politecnico di Milano alessandro.barenghi - at - polimi.it April 28, 2015 Introduction The bash command shell
More informationUnix as a Platform Exercises + Solutions. Course Code: OS 01 UNXPLAT
Unix as a Platform Exercises + Solutions Course Code: OS 01 UNXPLAT Working with Unix Most if not all of these will require some investigation in the man pages. That's the idea, to get them used to looking
More information9.2 Linux Essentials Exam Objectives
9.2 Linux Essentials Exam Objectives This chapter will cover the topics for the following Linux Essentials exam objectives: Topic 3: The Power of the Command Line (weight: 10) 3.3: Turning Commands into
More informationLanguage Basics. /* The NUMBER GAME - User tries to guess a number between 1 and 10 */ /* Generate a random number between 1 and 10 */
Overview Language Basics This chapter describes the basic elements of Rexx. It discusses the simple components that make up the language. These include script structure, elements of the language, operators,
More informationThe e switch allows Perl to execute Perl statements at the command line instead of from a script.
CH02.DOC Page 5 Friday, January 26, 2001 2:35 PM 2 Perl Scripts 2.1 Perl at the Command Line Although most of your work with Perl will be done in scripts, Perl can also be executed at the command line
More information5/20/2007. Touring Essential Programs
Touring Essential Programs Employing fundamental utilities. Managing input and output. Using special characters in the command-line. Managing user environment. Surveying elements of a functioning system.
More informationDr. Sarah Abraham University of Texas at Austin Computer Science Department. Regular Expressions. Elements of Graphics CS324e Spring 2017
Dr. Sarah Abraham University of Texas at Austin Computer Science Department Regular Expressions Elements of Graphics CS324e Spring 2017 What are Regular Expressions? Describe a set of strings based on
More informationCSE 303 Lecture 7. Regular expressions, egrep, and sed. read Linux Pocket Guide pp , 73-74, 81
CSE 303 Lecture 7 Regular expressions, egrep, and sed read Linux Pocket Guide pp. 66-67, 73-74, 81 slides created by Marty Stepp http://www.cs.washington.edu/303/ 1 discuss reading #2 Lecture summary regular
More informationServer-side Web Development (I3302) Semester: 1 Academic Year: 2017/2018 Credits: 4 (50 hours) Dr Antoun Yaacoub
Lebanese University Faculty of Science Computer Science BS Degree Server-side Web Development (I3302) Semester: 1 Academic Year: 2017/2018 Credits: 4 (50 hours) Dr Antoun Yaacoub 2 Regular expressions
More informationPart III. Shell Config. Tobias Neckel: Scripting with Bash and Python Compact Max-Planck, February 16-26,
Part III Shell Config Compact Course @ Max-Planck, February 16-26, 2015 33 Special Directories. current directory.. parent directory ~ own home directory ~user home directory of user ~- previous directory
More informationAC109/AT109 UNIX & SHELL PROGRAMMING DEC 2014
Q.2 a. Explain the principal components: Kernel and Shell, of the UNIX operating system. Refer Page No. 22 from Textbook b. Explain absolute and relative pathnames with the help of examples. Refer Page
More informationCSE 374: Programming Concepts and Tools. Eric Mullen Spring 2017 Lecture 4: More Shell Scripts
CSE 374: Programming Concepts and Tools Eric Mullen Spring 2017 Lecture 4: More Shell Scripts Homework 1 Already out, due Thursday night at midnight Asks you to run some shell commands Remember to use
More informationIntroduction To. Barry Grant
Introduction To Barry Grant bjgrant@umich.edu http://thegrantlab.org Working with Unix How do we actually use Unix? Inspecting text files less - visualize a text file: use arrow keys page down/page up
More informationCSE 374 Programming Concepts & Tools. Laura Campbell (thanks to Hal Perkins) Winter 2014 Lecture 6 sed, command-line tools wrapup
CSE 374 Programming Concepts & Tools Laura Campbell (thanks to Hal Perkins) Winter 2014 Lecture 6 sed, command-line tools wrapup Where we are Learned how to use the shell to run, combine, and write programs
More informationExpr Language Reference
Expr Language Reference Expr language defines expressions, which are evaluated in the context of an item in some structure. This article describes the syntax of the language and the rules that govern the
More informationCST Lab #5. Student Name: Student Number: Lab section:
CST8177 - Lab #5 Student Name: Student Number: Lab section: Working with Regular Expressions (aka regex or RE) In-Lab Demo - List all the non-user accounts in /etc/passwd that use /sbin as their home directory.
More informationSection 5.5: Text Menu Input from Character Strings
Chapter 5. Text User Interface TGrid user interface also consists of a textual command line reference. The text user interface (TUI) is written in a dialect of Lisp called Scheme. Users familiar with Scheme
More informationUnix as a Platform Exercises. Course Code: OS-01-UNXPLAT
Unix as a Platform Exercises Course Code: OS-01-UNXPLAT Working with Unix 1. Use the on-line manual page to determine the option for cat, which causes nonprintable characters to be displayed. Run the command
More informationUser Commands sed ( 1 )
NAME sed stream editor SYNOPSIS /usr/bin/sed [-n] script [file...] /usr/bin/sed [-n] [-e script]... [-f script_file]... [file...] /usr/xpg4/bin/sed [-n] script [file...] /usr/xpg4/bin/sed [-n] [-e script]...
More informationPractical Linux examples: Exercises
Practical Linux examples: Exercises 1. Login (ssh) to the machine that you are assigned for this workshop (assigned machines: https://cbsu.tc.cornell.edu/ww/machines.aspx?i=87 ). Prepare working directory,
More information1. Introduction. 2. Scalar Data
1. Introduction What Does Perl Stand For? Why Did Larry Create Perl? Why Didn t Larry Just Use Some Other Language? Is Perl Easy or Hard? How Did Perl Get to Be So Popular? What s Happening with Perl Now?
More informationPractical 02. Bash & shell scripting
Practical 02 Bash & shell scripting 1 imac lab login: maclab password: 10khem 1.use the Finder to visually browse the file system (single click opens) 2.find the /Applications folder 3.open the Utilities
More informationTitle:[ Variables Comparison Operators If Else Statements ]
[Color Codes] Environmental Variables: PATH What is path? PATH=$PATH:/MyFolder/YourStuff?Scripts ENV HOME PWD SHELL PS1 EDITOR Showing default text editor #!/bin/bash a=375 hello=$a #No space permitted
More informationCOPYRIGHTED MATERIAL. Getting Started with Windows PowerShell. Installing Windows PowerShell
Getting Started with Windows PowerShell If you are like me, then when you begin to look seriously at an interesting piece of software, you like to get your hands dirty and play with it from the beginning.
More informationFILTERS USING REGULAR EXPRESSIONS grep and sed
FILTERS USING REGULAR EXPRESSIONS grep and sed We often need to search a file for a pattern, either to see the lines containing (or not containing) it or to have it replaced with something else. This chapter
More informationAdvanced training. Linux components Command shell. LiLux a.s.b.l.
Advanced training Linux components Command shell LiLux a.s.b.l. alexw@linux.lu Kernel Interface between devices and hardware Monolithic kernel Micro kernel Supports dynamics loading of modules Support
More informationA shell can be used in one of two ways:
Shell Scripting 1 A shell can be used in one of two ways: A command interpreter, used interactively A programming language, to write shell scripts (your own custom commands) 2 If we have a set of commands
More informationCS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND
CS 307: UNIX PROGRAMMING ENVIRONMENT FIND COMMAND Prof. Michael J. Reale Fall 2014 Finding Files in a Directory Tree Suppose you want to find a file with a certain filename (or with a filename matching
More informationA Brief Introduction to the Linux Shell for Data Science
A Brief Introduction to the Linux Shell for Data Science Aris Anagnostopoulos 1 Introduction Here we will see a brief introduction of the Linux command line or shell as it is called. Linux is a Unix-like
More informationChapter 2 Working with Data Types and Operators
JavaScript, Fourth Edition 2-1 Chapter 2 Working with Data Types and Operators At a Glance Instructor s Manual Table of Contents Overview Objectives Teaching Tips Quick Quizzes Class Discussion Topics
More informationShell Programming Systems Skills in C and Unix
Shell Programming 15-123 Systems Skills in C and Unix The Shell A command line interpreter that provides the interface to Unix OS. What Shell are we on? echo $SHELL Most unix systems have Bourne shell
More informationCS 301. Lecture 05 Applications of Regular Languages. Stephen Checkoway. January 31, 2018
CS 301 Lecture 05 Applications of Regular Languages Stephen Checkoway January 31, 2018 1 / 17 Characterizing regular languages The following four statements about the language A are equivalent The language
More information