Linux Command Line and Shell Scripting Bible

Size: px

Start display at page:

Download "Linux Command Line and Shell Scripting Bible"

Ralf Welch
5 years ago
Views:

1 Linux Command Line and Shell Scripting Bible Richard Blum Wiley Publishing, Inc.

3 Linux Command Line and Shell Scripting Bible

5 Linux Command Line and Shell Scripting Bible Richard Blum Wiley Publishing, Inc.

6 Linux Command Line and Shell Scripting Bible Published by Wiley Publishing, Inc Crosspoint Boulevard Indianapolis, IN Copyright 2008 by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: Manufactured in the United States of America No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) , fax (978) Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., Crosspoint Blvd., Indianapolis, IN 46256, (317) , fax (317) , or online at Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Website is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Website may provide or recommendations it may make. Further, readers should be aware that Internet Websites listed in this work may have changed or disappeared between when this work was written and when it is read. For general information on our other products and services or to obtain technical support, please contact our Customer Care Department within the U.S. at (800) , outside the U.S. at (317) or fax (317) Library of Congress Cataloging-in-Publication Data is available from the publisher. Trademarks: Wiley, the Wiley logo, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. Linux is a registered trademark of Linus Torvald. All other trademarks are the property of their respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may notbeavailableinelectronicbooks.

7 To all the people who ve helped form my education. Parents, relatives, teachers, coworkers, and even anonymous posters on the Internet. Always be prepared to accept education from wherever you find it. Always continue to learn new things. For the LORD gives wisdom, and from his mouth come knowledge and understanding. Proverbs 2:6 (NIV)

8 About the Author Richard Blum has worked in the IT industry for over 19 years as both a systems and network administrator. During this time he has administered Unix, Linux, Novell, and Microsoft servers, as well as helped design and maintain a 3500-user network utilizing Cisco switches and routers. He has used Linux servers and shell scripts to perform automated network monitoring, and has written shell scripts in just about every Unix shell environment. Rich has a bachelor of science degree in Electrical Engineering, and a master of science degree in Management, specializing in management information systems, from Purdue University. He is the author of several Linux books, including sendmail for Linux, Running qmail, Postfix, Open Source Security, Network Performance Open Source Toolkit, andprofessional Assembly Language Programming. He s also a coauthor of Professional Linux Programming and Linux For Dummies, 8th Edition. When he s not being a computer nerd, Rich plays bass guitar for his church worship band and enjoys spending time with his wife, Barbara, and their two daughters, Katie Jane and Jessica.

9 Credits Acquisitions Editor Jenny Watson Senior Development Editor Tom Dinse Technical Editor John Kennedy Production Editor Angela Smith Copy Editor Foxxe Editorial Services Editorial Manager Mary Beth Wakefield Production Manager Tim Tate Vice President and Executive Group Publisher Richard Swadley Vice President and Executive Publisher Joseph B. Wikert Project Coordinator, Cover Lynsey Stanford Proofreader Word One New York Indexer Melanie Belkin

11 Acknowledgments xxv Introduction xxvii Part I The Linux Command Line Chapter 1: Starting with Linux Shells...3 Chapter 2: Getting to the Shell...25 Chapter 3: Basic bash Shell Commands...59 Chapter 4: More bash Shell Commands...91 Chapter 5: Using Linux Environment Variables Chapter 6: Understanding Linux File Permissions Chapter 7: Working with Editors Part II Shell Scripting Basics Chapter 8: Basic Script Building Chapter 9: Using Structured Commands Chapter 10: More Structured Commands Chapter 11: Handling User Input Chapter 12: Presenting Data Chapter 13: Script Control Part III Advanced Shell Scripting Chapter 14: Creating Functions Chapter 15: Adding Color to Scripts Chapter 16: Introducing sed and gawk Chapter 17: Regular Expressions Chapter 18: Advanced sed Chapter 19: Advanced gawk Part IV Alternative Linux Shells Chapter 20: The ash Shell Chapter 21: The tcsh Shell Chapter 22: The Korn Shell Chapter 23: The zsh Shell ix

12 Contents at a Glance Part V Advanced Topics Chapter 24: Using a Database Chapter 25: Using the Web Chapter 26: Using Chapter 27: Shell Scripts for Administrators Appendix A: Quick Guide to bash Commands Appendix B: Quick Guide to sed and gawk Appendix C: Comparing Shells Index x

Acknowledgments............................ xxv Introduction............................... xxvii Part I The Linux Command Line Chapter 1: Starting with Linux Shells.................... 3 What Is Linux?

13 Acknowledgments xxv Introduction xxvii Part I The Linux Command Line Chapter 1: Starting with Linux Shells What Is Linux?...3 Looking into the Linux kernel...4 The GNU utilities...12 The Linux desktop environment...14 Linux Distributions...20 Core Linux distributions...21 Specialized Linux distributions...22 The Linux LiveCD...23 Summary...24 Chapter 2: Getting to the Shell Terminal Emulation...25 Graphics capabilities...27 The keyboard...30 The terminfo Database...31 The Linux Console...35 The xterm Terminal...36 Command line parameters...37 The xterm main menu...38 The VT options menu...41 The VT fonts menu...43 The Konsole Terminal...45 Command line parameters...45 Sessions...45 The menu bar...48 The GNOME Terminal...52 The command line parameters...52 Tabs...53 The menu bar...54 Summary...58 xi

14 Contents Chapter 3: Basic bash Shell Commands Starting the Shell...59 The Shell Prompt...60 The bash Manual...63 Filesystem Navigation...64 The Linux filesystem...64 Traversing directories...66 File and Directory Listing...69 Basic listing...69 Modifying the information presented...71 The complete parameter list...72 Filtering listing output...74 File Handling...75 Creating files...75 Copying files...76 Linking files...79 Renaming files...80 Deleting files...81 Directory Handling...82 Creating directories...82 Deleting directories Viewing File Contents...83 Viewing file statistics...84 Viewing the file type...84 Viewing the whole file...85 Viewing parts of a file...89 Summary...90 Chapter 4: More bash Shell Commands Monitoring Programs...91 Peeking at the processes...91 Real-time process monitoring...98 Stopping processes Monitoring Disk Space Mounting media Using the df command Using the du command Working with Data Files Sorting data Searching for data Compressing data Archiving data Summary xii

15 Contents Chapter 5: Using Linux Environment Variables What Are Environment Variables? Global environment variables Local environment variables Setting Environment Variables Setting local environment variables Setting global environment variables Removing Environment Variables Default Shell Environment Variables Setting the PATH Environment Variable Locating System Environment Variables Login shell Interactive shell Non-interactive shell Variable Arrays Using Command Aliases Summary Chapter 6: Understanding Linux File Permissions Linux Security The /etc/passwd file The /etc/shadow file Adding a new user Removing a user Modifying a user Using Linux Groups The /etc/group file Creating new groups Modifying groups Decoding File Permissions Using file permission symbols Default file permissions Changing Security Settings Changing permissions Changing ownership Sharing Files Summary Chapter 7: Working with Editors The vim Editor The basics of vim Editing data Copy and paste Search and substitute xiii

16 Contents The emacs Editor Using emacs on the console Using emacs in X Windows The KDE Family of Editors The KWrite editor The Kate editor The GNOME Editor Starting gedit Basic gedit features Setting preferences Summary Part II Shell Scripting Basics Chapter 8: Basic Script Building Using Multiple Commands Creating a Script File Displaying Messages Using Variables Environment variables User variables The backtick Redirecting Input and Output Output redirection Input redirection Pipes Performing Math The expr command Using brackets A floating-point solution Exiting the Script Checking the exit status The exit command Summary Chapter 9: Using Structured Commands Working with the if-then Statement The if-then-else Statement Nesting ifs The test Command Numeric comparisons String comparisons File comparisons Compound Condition Testing xiv

17 Contents Advanced if-then Features Using double parentheses Using double brackets The case Command Summary Chapter 10: More Structured Commands The for Command Reading values in a list Reading complex values in a list Reading a list from a variable Reading values from a command Changing the field separator Reading a directory using wildcards The C-Style for Command The C language for command Using multiple variables The while Command Basic while format Using multiple test commands The until Command Nesting Loops Looping on File Data Controlling the Loop The break command The continue Command Processing the Output of a Loop Summary Chapter 11: Handling User Input Command Line Parameters Reading parameters Reading the program name Testing parameters Special Parameter Variables Counting parameters Grabbing all the data Being Shifty Working With Options Finding your options Using the getopt command The more advanced getopts Standardizing Options Getting User Input Basic reading xv

18 Contents Timing out Silent reading Reading from a file Summary Chapter 12: Presenting Data Understanding Input and Output Standard file descriptors Redirecting errors Redirecting Output in Scripts Temporary redirections Permanent redirections Redirecting Input in Scripts Creating Your Own Redirection Creating output file descriptors Redirecting file descriptors Creating input file descriptors Creating a read/write file descriptor Closing file descriptors Listing Open File Descriptors Suppressing Command Output Using Temporary Files Creating a local temporary file Creating a temporary file in /tmp Creating a temporary directory Logging Messages Summary Chapter 13: Script Control Handling Signals Linux signals revisited Generating signals Trapping signals Trapping a script exit Removing a trap Running Scripts in Background Mode Running in the background Running multiple background jobs Exiting the terminal Running Scripts without a Console Job Control Viewing jobs Restarting stopped jobs Being Nice The nice command xvi

19 Contents The renice command Running Like Clockwork Scheduling a job using the at command Using the batch command Scheduling regular scripts Start At the Beginning Starting your scripts at boot Starting with a new shell Summary Part III Advanced Shell Scripting Chapter 14: Creating Functions Basic Script Functions Creating a function Using functions Returning a Value The default exit status Using the return command Using function output Using Variables in Functions Passing parameters to a function Handling variables in a function Array Variables and Functions Passing arrays to functions Returning arrays from functions Function Recursion Creating a Library Using Functions on the Command Line Creating functions on the command line Defining functions in the.bashrc file Summary Chapter 15: Adding Color to Scripts Creating Text Menus Create the menu layout Create the menu functions Add the menu logic Putting it all together Using the select command Adding Color The ANSI escape codes Displaying ANSI escape codes Using colors in scripts xvii

20 Contents Doing Windows The dialog package The dialog options Using the dialog command in a script Getting Graphic The KDE environment The GNOME environment Summary Chapter 16: Introducing sed and gawk Text Manipulation The sed editor The gawk program The sed Editor Basics More substitution options Using addresses Deleting lines Inserting and appending text Changing lines The transform command Printing revisited Using files with sed Summary Chapter 17: Regular Expressions What Are Regular Expressions? A definition Types of regular expressions Defining BRE Patterns Plain text Special characters Anchor characters The dot character Character classes Negating character classes Using ranges Special character classes The asterisk Extended Regular Expressions The question mark The plus sign Using braces The pipe symbol Grouping expressions Regular Expressions in Action xviii

21 Contents Counting directory files Validating a phone number Parsing an address Summary Chapter 18: Advanced sed Multiline Commands The next commands The multiline delete command The multiline print command The Hold Space Negating a Command Changing the Flow Branching Testing Pattern Replacement The ampersand Replacing individual words Using sed in Scripts Using wrappers Redirecting sed output Creating sed Utilities Double spacing lines Double spacing files that may have blanks Numbering lines in a file Printing last lines Deleting lines Removing HTML tags Summary Chapter 19: Advanced gawk Using Variables Built-in variables User-defined variables Working with Arrays Defining array variables Iterating through array variables Deleting array variables Using Patterns Regular expressions The matching operator Mathematical expressions Structured Commands The if statement The while statement xix

22 Contents The do-while statement The for statement Formatted Printing Built-in Functions Mathematical functions String functions Time functions User-Defined Functions Defining a function Using your functions Creating a function library Summary Part IV Alternative Linux Shells Chapter 20: The ash Shell What Is the ash Shell? The Original ash Shell The Original ash command line parameters The original ash built-in commands The ash shell files The dash Shell The dash command line parameters The dash environment variables The dash built-in commands Scripting in dash Creating ash and dash scripts Things that won t work Summary Chapter 21: The tcsh Shell What Is the tcsh Shell? The tcsh Shell Components The tcsh command line parameters The tcsh files The tcsh login files Shell startup files The logout files The tcsh environment variables Shell variables Environment variables Setting variables in tcsh Using the set command Using the setenv command xx

23 Contents The tcsh built-in commands Scripting in tcsh Working with variables Array variables Handling mathematical operations Structured commands The if statements The foreach statement The while statement The switch command Summary Chapter 22: The Korn Shell The Korn Shell History The Parts of the ksh93 Shell Command line parameters Default files Environment variables Built-in commands Scripting in the ksh93 Shell Mathematical operations Structured commands Command redirection Discipline functions Summary Chapter 23: The zsh Shell History of the zsh Shell Parts of the zsh Shell Shell options The zsh shell files Environment variables Built-in commands Scripting with zsh Mathematical operations Structured commands Functions Summary Part V Advanced Topics Chapter 24: Using a Database The MySQL Database Installing MySQL Completing the MySQL configuration xxi

24 Contents The MySQL client interface Creating MySQL database objects The PostgreSQL Database Installing PostgreSQL The PostgreSQL command interface Creating PostgreSQL database objects Working with Tables Creating a table Inserting and deleting data Querying data Using the Database in Your Scripts Connecting to the databases Sending commands to the server Formatting data Summary Chapter 25: Using the Web The Lynx Program Installing Lynx The lynx command line The Lynx configuration file The Lynx environment variables Capturing data from Lynx The curl Program Installing curl The curl command line Exploring with curl Networking with zsh The TCP module The client/server paradigm Client/server programming with zsh Summary Chapter 26: Using The Basics of Linux in Linux The Mail Transfer Agent The Mail Delivery Agent The Mail User Agent Setting Up Your Server sendmail Postfix Sending a Message with Mailx The Mutt Program Installing Mutt xxii

25 Contents The Mutt command line Using Mutt Summary Chapter 27: Shell Scripts for Administrators Monitoring System Statistics Monitoring disk free space Catching disk hogs Watching CPU and memory usage Performing Backups Archiving data files Storing backups off-site Summary Appendix A: Quick Guide to bash Commands Built-In Commands Bash Commands Environment Variables Appendix B: Quick Guide to sed and gawk The sed Editor Starting the sed editor sed commands The gawk program The gawk command format Using gawk The gawk variables The gawk program features Appendix C: Comparing Shells Variables Environment variables User-defined variables Array variables Structured Commands The if-then, while, and until statements The for statement Mathematical Operations Index xxiii

27 First, all glory and praise go to God, who through His Son makes all things possible, and gives us the gift of eternal life. Many thanks go to the great team of people at John Wiley & Sons for their outstanding work on this project. Thanks to Jenny Watson, the acquisitions editor, for offering me the opportunity to work on this book. Also thanks to Tom Dinse, the development editor, for keeping things on track and making this book more presentable. The technical editor, John Kennedy, did an amazing job of double-checking all the work in this book, plus making suggestions to improve the content. Thanks, John, for your hard work and diligence. I would also like to thank Carole McClendon at Waterside Productions, Inc. for arranging this opportunity for me, and for helping out in my writing career. Finally, I would like to thank my parents, Mike and Joyce Blum, for their dedication and support while raising me, and my wife, Barbara, and daughters, Katie Jane and Jessica, for their love, patience, and understanding, especially while I was writing this book. xxv

29 Welcome to Linux Command Line and Shell Scripting Bible. Like all books in the Bible series, you can expect to find both hands-on tutorials and real-world practical application information, as well as reference and background information that provides a context for what you are learning. This book is a fairly comprehensive resource on the Linux command line and shell commands. By the time you have completed Linux Command Line and Shell Scripting Bible, you will be well prepared to write your own shell scripts that can automate practically any task on your Linux system. Who Should Read This Book If you re a system administrator in a Linux environment, you ll benefit greatly by knowing how to write shell scripts. The book doesn t walk through setting up a Linux system, but once you have it running, you ll want to start automating some of the routine administrative tasks. That s where shell scripting comes in, and that s where this book will help you out. This book will demonstrate how to automate any administrative task using shell scripts, from monitoring system statistics and data files to generating reports for your boss. If you re a home Linux enthusiast, you ll also benefit from Linux Command Line and Shell Scripting Bible. Nowadays it s easy to get lost in the graphical world of prebuilt widgets. Most desktop Linux distributions try their best to hide the Linux system from the typical user. However, there are times when you have to know what s going on under the hood. This book shows you how to access the Linux command line prompt, and what to do once you get there. Often performing simple tasks, such as file management, can be done more quickly from the command line than from a fancy graphical interface. There s a wealth of commands you can use from the command line, and this book shows you just how to use them. How This Book Is Organized This book is organized in a way that leads you through the basics of the Linux command line all the way to creating your own shell scripts. The book is divided into five parts, each one building on the previous parts. Part I assumes that you either have a Linux system running or are looking into getting a Linux system. Chapter 1, Starting with Linux Shells, describes the parts of a total Linux system and xxvii

30 Introduction shows how the shell fits in. After learning the basics of the Linux system, this section continues with: Using a terminal emulation package to access the shell (Chapter 2) Introducing the basic shell commands (Chapter 3) Using more advanced shell commands to peek at system information (Chapter 4) Working with shell variables to manipulate data (Chapter 5) Understanding the Linux filesystem and security (Chapter 6) Knowing how to use the Linux editors to start writing shell scripts (Chapter 7) In Part II, you ll start writing shell scripts: Learn how to create and run shell scripts (Chapter 8) Alter the program flow in a shell script (Chapter 9) Iterate through code sections (Chapter 10) Handle data from the user in your scripts (Chapter 11) See different methods for storing and displaying data from your script (Chapter 12) Control how and when your shell scripts run on the system (Chapter 13) Part III dives into more advanced areas of shell script programming: Create your own functions to use in all your scripts (Chapter 14) See different methods for interacting with your script users (Chapter 15) Use advanced Linux commands to filter and parse data files (Chapter 16) Use regular expressions to define data (Chapter 17) Learn advanced methods of manipulating data in your scripts (Chapter 18) See how to generate reports from raw data (Chapter 19) In Part IV, you ll get to see how to write shell scripts using some of the alternative shells available in the Linux environment: Write scripts for the ash or dash shells (Chapter 20) See how writing scripts in the tcsh shell is different (Chapter 21) Work with floating-point numbers in the ksh93 shell (Chapter 22) Use advanced network and math features in the zsh shell (Chapter 23) xxviii

Linux Command Line and Shell Scripting Bible

Linux Command Line and Shell Scripting Bible Richard Blum Wiley Publishing, Inc. Linux Command Line and Shell Scripting Bible Linux Command Line and Shell Scripting Bible Richard Blum Wiley Publishing,