CSC 2500: Unix Lab Fall 2016 IO Redirection Mohammad Ashiqur Rahman Department of Computer Science College of Engineering Tennessee Tech University
Agenda Standard IO IO Redirection Pipe Various File Processing Unix Commands Lab 3: Learning IO Redirection M. Ashiq Rahman, Tennessee Tech University 2
What is IO Redirection? With the IO redirection one can: Redirect the input and output of commands to and from files Connect multiple commands together into powerful command pipelines M. Ashiq Rahman, Tennessee Tech University 3
Standard IO Programs typically produce: Useful results or data Status or error messages Everything is a file, so the output is sent to one of two files Standard Output (results) (1) Standard Error (errors or status) (2) Input comes from one file Standard Input (0) M. Ashiq Rahman, Tennessee Tech University 4
File and Pipe Redirection The > redirects stdout to a file The left hand side is the command/program that writes to stdout The file with the provided name may exist If not, it is created If it does exist, it is truncated Examples: ls -l > list.txt files.txt will contain a long listing of the current directory wc -l file.txt > num_lines.dat num_lines.dat will have a single number, i.e. the number of lines in file.txt M. Ashiq Rahman, Tennessee Tech University 5
File and Pipe Redirection (2) The >> also redirects stdout to a file, but The file with the provided name may exist If not, it is created If it does exist, it is appended Examples: $ wc backup.tcl > counts.dat $ cat counts.dat 68 240 1600 backup.tcl $ wc install.sh >> counts.dat $ cat counts.dat 68 240 1600 backup.tcl 66 227 1608 install.sh M. Ashiq Rahman, Tennessee Tech University 6
Redirecting Standard Input The < redirects from stdin The file must exist and can be opened for reading The program reads the data from the file just as if it were typed in at the keyboard Example: $ cat names.txt Joe Mary Ken Steve Annie $ sort < names.txt > sorted_names.txt $ cat sorted_names.txt Annie Joe Ken Mary Steve M. Ashiq Rahman, Tennessee Tech University 7
Redirecting Standard Error One can redirect stderr using 2> For example, to save your compiler output: $ g++ -Wall -o sortit sortit.cpp 2> error.txt $ tail error.txt sortit.cpp:32:16: warning: range-based for loop is a C++11 extension [-Wc++11-extensions] for (int a : s) { ^ sortit.cpp:38:35: error: expected expression std::sort(s.begin(), s.end(), [](int a, int b) { ^ sortit.cpp:41:16: warning: range-based for loop is a C++11 extension [-Wc++11-extensions] for (int a : s) { ^ 6 warnings and 1 error generated. M. Ashiq Rahman, Tennessee Tech University 8
Redirecting stdout and stderr to One File There are cases in which one may wish to capture all of the output of a command to a single file. Redirect both standard output and standard error at the same time $ ls -l /bin/usr &> ls-output.txt M. Ashiq Rahman, Tennessee Tech University 9
Pipes The UNIX system allows stdout of a command to be connected to stdin of another command. Using the pipe character The above command is written like so: ps -ef grep login less What does it do? Lists all processes that have the word login in their command line one page at a time M. Ashiq Rahman, Tennessee Tech University 10
The Difference Between > and The redirection operator connects a command with a file While the pipeline operator connects the output of one command with the input of a second command The following is not possible: command1 > command2 M. Ashiq Rahman, Tennessee Tech University 11
Filters Pipelines are often used to perform complex operations on data Pipelines used in this way are called filters Example: uniq -- report or filter out repeated lines in a file wc -- word, line, character, and byte count $ ls /bin /usr/bin sort uniq wc -l What does it will do? M. Ashiq Rahman, Tennessee Tech University 12
Lab 03 Objective: Executing Unix commands with options and arguments Getting rid of output going to stderr Pipes The use of XQuery Lab Prerequisites: Your system should have the following commands installed: (1) 7z, (2) xqilla, (3) sort, (4) wget, and (5) split M. Ashiq Rahman, Tennessee Tech University 13
Lab 03: Problem Description You will be provided a file that I downloaded from the Internet. The file contains a database, structured as XML, of proteins. Unfortunately, I do not want the file in XML format because it is too hard to read. Furthermore, I am only interested in two fields from the file. Those fields are the id field and the description field. The file is sorted by id, but I want the file sorted by description. I do not want to have to deal with the data as one big because loading this file into text editors is problematic. So, I want the data divided into smaller, more manageable, files. M. Ashiq Rahman, Tennessee Tech University 14
Lab 03: Homework Write a shell script called that does all of the above. All the works need to be accomplished in two lines. The entire shell script is going to be 3 lines long. The first line will be the obligatory #!/bin/bash The following 2 lines will download the compressed XML file and generate the manageable files. Your script file should have the following name: Protein_Info_<First Name><Last Name>.sh. Submission Deadline: Friday, September 16, 2016 Submission Site: ilearn (a Dropbox folder named Homework 03 ) Submission Content: Submit the script file. M. Ashiq Rahman, Tennessee Tech University 15
Lab Practice Download the XML file from the link below and show the downloaded file: http://users.csc.tntech.edu/~marahman/courses/csc_2500/largefile.txt Split the downloaded file into multiple smaller files (using default, the number of lines, and the number of bytes options). Merge (using cat) of two small files (split based on bytes) and print the number of lines/words in the merged result using pipe and wc. To install xqilla and p7zip (7z) sudo apt-get update sudo apt-get install xqilla p7zip p7zip-full M. Ashiq Rahman, Tennessee Tech University 16
THANKS Acknowledgement: - Dr. Mike Rogers - Linux man pages M. Ashiq Rahman, Tennessee Tech University 17