Bash command shell language interpreter

Transcription

1 Principles of Programming Languages Bash command shell language interpreter Advanced seminar topic Louis Sugy & Baptiste Thémine Presentation on December 8th, 2017

2 Table of contents I. General information about bash... 2 The Bourne-again shell... 2 Basic syntax and use... 2 II. A command-line language interpreter... 3 Line by line evaluation... 3 Lexical analysis and parsing... 3 Word expansion... 3 Execution... 4 III. Technical details... 5 Variables, memory management... 5 Pipes... 5 References... 6 P a g e 1 6

3 I. General information about bash Before giving any details about the interpreter, let s explain what bash is and why it is different from most of the programming languages. This step is important to understand some aspects of the interpreter itself. The Bourne-again shell The name bash is used both for the Unix shell and the command language which can be used in this shell. The name is an acronym for Bourne-again shell, to highlight its inheritance from the Bourne shell: bash is a free software version of Stephen Bourne s work, developed by Brian Fox and released in It is available on Unix-like systems (Linux, MacOS, etc.), and now also on Windows 10 (there were already some emulators long before the official port). Though it is not the only Unix shell, it is the most popular one, and comes by default on most Linux distributions. Basic syntax and use Most of the Unix command-line languages share the same syntax for the most common operations. In fact, most of them are supersets of the Bourne shell: they can run any script designed for the Bourne shell, and add some extra features. Some of the improvements brought by bash are the following: indexed arrays of unlimited size, integer arithmetic, functions and aliases. Bash is designed to interact with the system: run programs, navigate in the filesystem, use the network. The syntax is oriented towards that purpose: variables don t have types, they contain text which can be used as a shortcut, or interpreted as integers with some operators. Bash has no support for floating point operations (it s not a general-purpose programming language!). An example illustrating this: COMP=g++; VERSION=-std=c++11; CFLAGS=-Wall $(COMP) $(VERSION) $(CFLAGS) -c prog.o The language has loops and tests. Here are some extracts from the manual: while test; do commands; done for name [ [in [words ] ] ; ] do commands; done if test; then commands; [elif test; then commands;] [else commands;] fi The language also features arrays. In fact, every variable is an array, often of size 1. tab=(tampere university of technology) Functions can be defined, their arguments are named $1, $2, etc. function e { echo $1 } An important feature is the pipe which directs the output of an operation to the input of another: ls -l grep "\.txt$" P a g e 2 6

4 II. A command-line language interpreter Line by line evaluation There is a huge difference between command-line languages interpreters and scripting languages interpreters: a command-line language interpreter doesn t know all the code when it starts executing it. Whereas in Python there is a difference when interacting with the console and running a script (a script is analyzed completely and converted into a lower-level code before being executed), there is not such a thing with bash: the code is always executed line by line. It means that if there is a syntax error at some line, this will only be detected if the execution reaches that exact line. Lexical analysis and parsing First, a list of tokens is generated from the line of text. The identification of the tokens is based on the context. For example, a reserved keyword can be an identifier, it is valid in bash (in contrary to almost all the other programming languages). Though this can create problems in some cases when using shortcuts, because the expansion of variables is processed later. The aliases are managed at this moment (an alias is a hard shortcut for a word, for example alias compile= g++ -std=c++11 -Wall ). The tokens belong to three categories: operators, reserved words and other words. The operators include all special characters which have a meaning for bash, e.g the piping operator. Everything that is not an operator or a reserved word is a word, even numbers. Then, the parser transforms this token list into a C structure representing a command. For some complex lines including flow control, this command can include other commands, on so on. A basic command is a word list, which will be updated throughout the pipeline (cf next paragraph): typedef struct word_desc { char *word; int flags } WORD_DESC; typedef struct word_list { struct word_list *next; WORD_DESC *word; } WORD_LIST; Word expansion We can imagine the analysis of a word as a pipeline: the data representing each word goes through a fixed number of operations, which give their result to the next one. In order to allow a parallel execution of this pipeline for the different words, the context of the word is given through flags, and information that a step would need to give to the next one (for example this is an assignment, you re dealing with a variable ). Let s have an overview of the different phases: Brace expansion: in bash, braces allow to simplify the code by emulating some feature of regular expressions. For example *.{png,jpg} is a shortcut for *.png *.jpg. This phase replaces the expression with braces by the value it represents. P a g e 3 6

5 Tilde expansion: many directories have a shortcut beginning by a tilde. For example, the home directory of the user is ~/, but there is also a directory ~mail, ~sys, ~bin, etc. This allows bash scripts to reference these common directories without having to know their location, which can vary from one computer to another. Variable and parameter expansion: this phase replaces variables by their values (which, as we have seen before, is text, there is no typing). The syntax ${var:-word} will give the value of var if the variable is set, and word otherwise. Command process and arithmetic substitution: this phase evaluates expressions and applies operators when possible. For example, 1+2 would be replaced by 3. Word splitting: because of all the previous substitutions, some words may contain more than 1 word. This phase separates words according to the characters contained in the shell variable IFS. The quotes protect the content from being split. Filename generation: as this is a command-line language, it is very likely that some words are referring to files. This step analyzes the words and tries to match them with a file (for example parsing a relative path). Execution In the case of simple commands, bash simply extracts the name of the command, the parameters, and asks the system to call the program with these parameters. Some commands are also directed to the shell itself, for example variable affectation. In that case the shell updates its state (we ll talk about memory management in the next part). Another good example of internal command is cd: it can t be implemented as an external command because it s not printing or returning a value but modifying the state of the shell itself (the current folder). Commands are searched in builtins at the first place, then the PATH is used to search an executable in a list of directories. If no executable is found, the interpreter checks if a function with that name is defined. Launching external commands is done by executing a fork which replicates the shell environment, and then deals with some detail like files opened and data that the new process doesn t need, before finally launching the program. This takes some time; therefore, it is recommended to limit the number of external calls (one call processing a big amount of data is better than many calls in a loop). It gets a bit more complicated when dealing with flow control (loops and tests). The interpreter constructs a structure with the different commands and conditions. Each flow control tool is implemented by a specific C function which takes care of running the different commands, doing verifications, putting the right values in the right variables (for example updating the iteration variable in a for loop). Some details about the implementation of pipes will be given in next part. P a g e 4 6

6 III. Technical details Variables, memory management Bash has a very different way to approach variables scopes than most of the other programming languages. Whereas the other programming languages consider every variable as local to a function of block, except for global variables which are declared at the root of the program, bash considers all variables to be global by default. Functions can be made local with the keyword local, and they are then accessible by all the functions called by the function which own them. When a function uses a variable name, it always refers to the last function having a local variable of that name in the call stack. The implementation of the memory in the bash interpreter is quite simple then: the shell holds a linked list of hashtables. Hashtables allow an access to a value knowing the symbol. The linked lists contain the new variables of each scope. Finding the value associated to a symbol consists in finding the last appearance of the symbol in one of the tables of the linked list. This system is quite efficient because it avoids having to copy values when calling functions like in many languages (and it is still possible to copy values in local variables if needed), but it is criticized for the bad encapsulation, and if people don t declare variables as local in functions, it will stay in memory for the shell s lifetime. When executing a script from a file, the variables used in the script are not accessible in the shell. This is one of the few differences between running a script and running the same lines in the shell (a script is in fact run in a shell of its own). Pipes As we have seen before, it is possible to have pipes which redirect the output of command to the input of another one, and so on. Each operation in the pipeline is executed in a different process, but all the operations of a pipeline are in the same job for the shell (sadly we will not discuss about jobs because it is complicated and would require more time and pages, but you can find more information about these in the links given in the References section). These processes communicate together through their standard input and ouput. They are indeed chained together, and the error output are directed to the shell. This image from Wikipedia illustrates that principle. P a g e 5 6

7 References Most of the information we have found about bash interpreter comes from the work of Chet Ramey, who has been the primary developer of the bash interpreter for 20 years. These books are full of information about bash and its interpreter, so it is the best (and, to be honest, the only) material to begin with if you want to learn more about this topic. The Architecture of Open Source Applications, The Bourne-Again Shell, Chet Ramey Bash Reference Manual, by Chet Ramey. P a g e 6 6