Linux and C Programming Language. Department of Computer Science and Engineering Yonghong Yan

Transcription

1 Linux and C Programming Language Department of Computer Science and Engineering Yonghong Yan yanyh@cse.sc.edu 1

2 C Syntax and Hello World What do the < > mean? #include inserts another file..h files are called header files. They contain declarations/definitions needed to interface to libraries and code in other.c files. A comment, ignored by the compiler #include <stdio.h> /* The simplest C Program */ int main(int argc, char **argv) The main() function is always where your program starts running. { } printf( Hello World\n ); return 0; Blocks of code ( lexical scopes ) are marked by { } Return 0 from this function 2

3 Compilation Process in C Compilation process: gcc hello.c o hello Constructing an executable image for an application FOUR stages Command: gcc <options> <source_file.c> Compiler Tool gcc (GNU Compiler) man gcc (on Linux m/c) icc (Intel C compiler)

4 4 Stages of Compilation Process Preprocessing gcc -E hello.c -o hello.i hello.c à hello.i Compilation (after preprocessing) gcc -S hello.i -o hello.s Assembling (after compilation) gcc -c hello.s -o hello.o Linking object files gcc hello.o -o hello Output à Executable (a.out) Run à./hello (Loader)

5 4 Stages of Compilation Process 1. Preprocessing (Those with # ) Expansion of Header files (#include ) Substitute macros and inline functions (#define ) 2. Compilation Generates assembly language Verification of functions usage using prototypes Header files: Prototypes declaration 3. Assembling Generates re-locatable object file (contains m/c instructions) nm app.o T main U puts nm or objdump tool used to view object files

6 4 Stages of Compilation Process (contd..) 4. Linking Generates executable file (nm tool used to view exe file) Binds appropriate libraries Static Linking Dynamic Linking (default) Loading and Execution (of an executable file) Evaluate size of code and data segment Allocates address space in the user mode and transfers them into memory Load dependent libraries needed by program and links them Invokes Process Manager à Program registration

7 Compiling a C Program gcc <options> program_name.c Options: Four stages into one Wall: Shows all warnings -o output_file_name: By default a.out executable file is created when we compile our program with gcc. Instead, we can specify the output file name using "-o" option. -g: Include debugging information in the binary. man gcc

8 Linking Multiple files to make executable file Two programs, prog1.c and prog2.c for one single task To make single executable file using following instructions First, compile these two files with option "-c" gcc -c prog1.c gcc -c prog2.c -c: Tells gcc to compile and assemble the code, but not link. We get two files as output, prog1.o and prog2.o Then, we can link these object files into single executable file using below instruction. gcc -o prog prog1.o prog2.o Now, the output is prog executable file. We can run our program using./prog

9 Linking with other libraries Normally, compiler will read/link libraries from /usr/lib directory to our program during compilation process. Library are precompiled object files To link our programs with libraries like pthreads and realtime libraries (rt library). gcc <options> program_name.c -lpthread -lrt -lpthread: Link with pthread library à libpthread.so file -lrt: Link with rt library à librt.so file Option here is "-l<library>" Another option "-L<dir>" used to tell gcc compiler search for library file in given <dir> directory.

10 Compilation, Linking, Execution of C/C++ Programs source file 1 object file 1 source file 2 compilation object file 2 library object file 1 linking (relocation + linking) load file source file N object file N library object file M usually performed by a compiler, usually in one uninterrupted sequence

11 More on C Programming 11

12 What is it? Things with # #include <stdio.h> Processes the C source files BEFORE handing it to compiler. `Pre`-process gcc E cpp

13 Why care? File Inclusion Basic Macros Parameterized Macros Conditional Compilation Style and (mis)use of Macros

14 File Inclusion Recall : #include <filename> #include <foo.h> System directories #include foo.h Current directories gcc -I

15 Macros We can #define constants #define HALF_PI #define BAD_PI HALF_PI+HALF_PI Solution? ((((((((((The parentheses!)))))))))) #define PI ((HALF_PI)+(HALF_PI))

16 Parameterized Macros Macros with param : #define BAAAD_DOUBLE(x) (x*2) #define BAD_DOUBLE(x) ((x)+(x)) #define OKAY_DOUBLE(x) ((x)*2) WHAT TO TAKE HOME: Don t use 2+ parameters. No assignments within macro

17 Parameterized Macros cont d Want to wrap malloc() calls. Can we? YES! #define NEW_T(t,sz) ((t*) malloc((sz)*sizeof(t))

18 Appropriate use of Macros Do Use parentheses Handle header file inclusion Capitalize macro names Don t Use more than 2 parameters. Replace function calls with macros for performance Make the code harder to read #define LOOP(s) while(1) {s};

19 #if #ifdef #ifndef #elif #else #endif Conditional Inclusion

20 Conditional Compilation Debugging #define DEBUG_MODE #ifdef DEBUG_MODE /* Debug code */ #endif OS/Platform #ifdef _WIN32 #include <windows.h> #elif /* */

21 File Header File header #ifndef FOO_H #define FOO_H /* the whole file * #endif Why??? Global variables Cycles of includes Inefficient

22 Basics printf( format string,vars); Format string? This year is %d\n Your score is %d\n Conversion by % %d : int %f : float, double %c : char %s : char *, string %e : float, double in scientific form

23 Examples %d conversion Padding ( width ) %12d %012d Precision %12.4d %12.6d Left/right justification %12d %-12d Similar in %f, %e, conversions

24 Example %c, %s conversion %c has only precision 1 Precision in %s means string length starting from the beginning Padding and justification are same

25 Lexical Scoping Every Variable is Defined within some scope. A Variable cannot be referenced by name (a.k.a. Symbol) from outside of that scope. Lexical scopes are defined with curly braces { }. The scope of Function Arguments is the complete body of that function. The scope of Variables defined inside a function starts at the definition and ends at the closing brace of the containing block The scope of Variables defined outside a function starts at the definition and ends at the end of the file. Called Global Vars. void p(char x) { /* p,x */ char y; /* p,x,y */ char z; /* p,x,y,z */ } /* p */ char z; /* p,z */ void q(char a) { char b; /* p,z,q,a,b */ } { } char c; /* p,z,q,a,b,c */ char d; /* p,z,q,a,b,d (not c) */ /* p,z,q */ char b? legal? 25

26 Comparison and Mathematical Operators == equal to < less than <= less than or equal > greater than >= greater than or equal!= not equal && logical and logical or! logical not Beware division: 17/5=3, 17%5=2 5 / 10 = 0 whereas 5 / 10.0 = 0.5 Division by 0 will cause a FPE(Floatpoint exception) + plus - minus * mult / divide % modulo & bitwise and bitwise or ^ bitwise xor ~ bitwise not << shift left >> shift right Don t confuse & and &&.. 1 & 2 = 0 whereas 1 && 2 = <true> The rules of precedence are clearly defined but often difficult to remember or non-intuitive. When in doubt, add parentheses to make it explicit. 26

27 Assignment Operators x = y x++ ++x x-- --x assign y to x post-increment x pre-increment x post-decrement x pre-decrement x x += y assign (x+y) to x x -= y assign (x-y) to x x *= y assign (x*y) to x x /= y assign (x/y) to x x %= y assign (x%y) to x Note the difference between ++x and x++ (high vs low priority (precedence)): int x=5; int y; y = ++x; /* x == 6, y == 6 */ int x=5; int y; y = x++; /* x == 6, y == 5 */ Don t confuse = and ==! int x=5; if (x==6) /* false */ { /*... */ } /* x is still 5 */ int x=5; if (x=6) /* always true */ { /* x is now 6 */ } /*... */ 27

28 A Quick Digression About the Compiler #include <stdio.h> /* The simplest C Program */ int main(int argc, char **argv) { printf( Hello World\n ); return 0; } Preprocess Compilation occurs in two steps: Preprocessing and Compiling extension typedef unsigned long long int dev_t; extension typedef unsigned int extension typedef unsigned int extension typedef unsigned long int ino_t; my_program uid_t; gid_t; extension typedef unsigned long long int ino64_t; extension typedef unsigned int nlink_t; extension typedef long int extension typedef long long int off64_t; off_t; extern void flockfile (FILE * stream) ; extern int ftrylockfile (FILE * stream) ; extern void funlockfile (FILE * stream) ; int main(int argc, char **argv) { } printf( Hello World\n ); return 0; Compile In Preprocessing, source code is expanded into a larger form that is simpler for the compiler to understand. Any line that starts with # is a line that is interpreted by the Preprocessor. Include files are pasted in (#include) Macros are expanded (#define) Comments are stripped out ( /* */, // ) Continued lines are joined ( \ ) The compiler then converts the resulting text (called translation unit) into binary code the CPU can execute. 28

29 C Memory Pointers To discuss memory pointers, we need to talk a bit about the concept of memory We ll conclude by touching on a couple of other C elements: Arrays, typedef, and structs 29

30 The memory Memory: similar to a big table of numbered slots where bytes of data are stored. The number of a slot is its Address. One byte Value can be stored in each slot. Some data values span more than one slot, like the character string Hello\n A Type provides a logical meaning to a span of memory. Some simple types are: char char [10] int float int64_t a single character (1 slot) an array of 10 characters signed 4 byte integer 4 byte floating point signed 8 byte integer Addr Value H (72) 5 e (101) 6 l (108) 7 l (108) 8 o (111) 9 \n (10) 10 \0 (0)

31 What is a Variable? symbol table? A Variable names a place in memory where you store a Value of a certain Type. You first Declare a variable by giving it a name and specifying its type and optionally an initial value declare vs. define char x; char y= e ; Type is single character (char) 31 Initial value Variable x declared but undefined Name What names are legal? extern? static? const? The compiler puts x and y somewhere in memory. Symbol Addr Value x 4 Some garbage y 5 e (101)

32 Multi-byte Variables Different types require different amounts of memory. Most architectures store data on word boundaries, or even multiples of the size of a primitive data type (int, char) char x; char y= e ; int z = 0x ; 0x means the constant is written in hex padding An int requires 4 bytes Symbol Addr Value x 4 Some garbage y 5 e (101) 6 7 z

33 Memory, a more detailed view A sequential list of words, starting from 0. On 32bit architectures (e.g. Win32): each word is 4 bytes. Local variables are stored in the stack Dynamically allocated memory is set aside on the heap (more on this later ) For multiple-byte variables, the address is that of the smallest byte (little endian) Stack Heap word 0 word 1 word 2 33

34 Example #include <iostream> int main() { char c[10]; int d[10]; int* darr; darr = (int *)(malloc(10*sizeof(int))); size_t sizec = sizeof(c); size_t sized = sizeof(d); size_t sizedarr = sizeof(darr); What is the value of: - sizec - sized - sizedarr } free(darr); return 0; NOTE: sizeof is a compile-time operator that returns the size, in multiples of the size of char, of the variable or parenthesized type-specifier that it precedes. 34

35 Can a C function modify its arguments? What if we wanted to implement a function pow_assign() that modified its argument, so that these are equivalent: float p = 2.0; /* p is 2.0 here */ p = pow(p, 5); /* p is 32.0 here */? float p = 2.0; /* p is 2.0 here */ pow_assign(p, 5); /* Is p is 32.0 here? */ Native function, to use you need #include <math.h> Would this work? void pow_assign(float x, uint exp) { float result=1.0; int i; for (i=0; (i < exp); i++) { result = result * x; } x = result; } 35

36 In C you can t change the value of any variable passed as an argument in a function call Pass by value void pow_assign(float x, uint exp) { float result=1.0; int i; for (i=0; (i < exp); i++) { result = result * x; } x = result; } // a code snippet that uses above // function { float p=2.0; pow_assign(p, 5); // the value of p is 2 here } In C, all arguments are passed by value Keep in mind: pass by value requires the variable to be copied. That copy is then passed to the function. Sometime generating a copy can be expensive But, what if the argument is the address of a variable? 36

37 C Pointers What is a pointer? A variable that contains the memory address of another variable or of a function In general, it is safe to assume that on 32 bit architectures pointers occupy one word Pointers to int, char, float, void, etc. ( int*, char*, *float, void* ), they all occupy 4 bytes (one word). Pointers: *very* many bugs in C programs are traced back to mishandling of pointers 37

38 The need for pointers Pointers (cont.) Needed when you want to modify a variable (its value) inside a function The pointer is passed to that function as an argument Passing large objects to functions without the overhead of copying them first Accessing memory allocated on the heap Referring to functions, i.e. function pointers 38

39 Pointer Validity A Valid pointer is one that points to memory that your program controls. Using invalid pointers will cause non-deterministic behavior Very often the code will crash with a SEGV, that is, Segment Violation, or Segmentation Fault. There are two general causes for these errors: Coding errors that end up setting the pointer to a strange number Use of a pointer that was at one time valid, but later became invalid Good practice: Initialize pointers to 0 (or NULL). NULL is never a valid pointer value, but it is known to be invalid and means no pointer set. char * get_pointer() { char x=0; return &x; } Will ptr be valid or invalid? { } char * ptr = get_pointer(); *ptr = 12; /* valid? */ 39

40 Answer: No, it s invalid A pointer to a variable allocated on the stack becomes invalid when that variable goes out of scope and the stack frame is popped. The pointer will point to an area of the memory that may later get reused and rewritten. char * get_pointer() { char x=0; return &x; } int main() { char * ptr = get_pointer(); *ptr = 12; /* valid? */ other_function(); return 0; } But now, ptr points to a location that s no longer in use, and will be reused the next time a function is called! Here is what I get in DevStudio when compiling: main.cpp(6) : warning C4172: returning address of local variable or temporary 40

41 Example: What gets printed out? int main() { int d; char c; } short s; int* p; int arr[2]; printf( %p, %p, %p, %p, %p\n,&d, &c, &s, &p, arr ); return 0; s arr p d c NOTE: Here &d = 920 (in practice a 4- byte hex number such as 0x22FC3A08)

42 Example: Usage of Pointers & Pointer Arithmetic int main() { int d; char c; short s; int* p; int arr[2]; p = &d; *p = 10; c = (char)1; p = arr; *(p+1) = 5; p[0] = d; +3 s +2 arr[0] arr[1] = 920 p d +1 = c = } *( (char*)p + 1 ) = c; return 0; Q: What are the values stored in arr? [assume little endian architecture] 42

43 Example [Cntd.] p = &d; *p = 10; c = (char)1; p = arr; *(p+1) = 5; // int* p; p[0] = d; +3 s +2 arr[0] arr[1] p d +1 = 904 = 10 = 10 = 5 +0 c = *( (char*)p + 1 ) = c; Question: arr[0] =?

44 Use of pointers, another example Pass pointer parameters into function void swap(int *px, int *py) { int temp; temp = *px; *px = *py; *py = temp; } int a = 5; int b = 6; swap(&a, &b); l What will happen here? int * a; int * b; swap(a, b); 44

45 Dynamic Memory Allocation (on the Heap) Allows the program to determine how much memory it needs at run time and to allocate exactly the right amount of storage. It is your responsibility to clean up after you (free the dynamic memory you allocated) The region of memory where dynamic allocation and deallocation of memory can take place is called the heap. 45

46 Recall Discussion on Dynamic Memory Allocation Recall that variables are allocated statically by having declared with a given size. This allocates them in the stack. Allocating memory at run-time requires dynamic allocation. This allocates them on the heap. sizeof() reports the size of a type in bytes int * alloc_ints(size_t requested_count) { int * big_array; big_array = (int *)calloc(requested_count, sizeof(int)); if (big_array == NULL) { printf( can t allocate %d ints: %m\n, requested_count); return NULL; } } 46 /* big_array[0] through big_array[requested_count-1] are * valid and zeroed. */ return big_array; calloc() allocates memory for N elements of size k Returns NULL if can t alloc It s OK to return this pointer. It will remain valid until it is freed with free(). However, it s a bad practice to return it (if you need is somewhere else, declare and define it there )

47 Caveats with Dynamic Memory Dynamic memory is useful. But it has several caveats: Whereas the stack is automatically reclaimed, dynamic allocations must be tracked and free() d when they are no longer needed. With every allocation, be sure to plan how that memory will get freed. Losing track of memory causes memory leak. Whereas the compiler enforces that reclaimed stack space can no longer be reached, it is easy to accidentally keep a pointer to dynamic memory that was freed. Whenever you free memory you must be certain that you will not try to use it again. Because dynamic memory always uses pointers, there is generally no way for the compiler to statically verify usage of dynamic memory. This means that errors that are detectable with static allocation are not with dynamic 47

48 Data Structures A data structure is a collection of one or more variables, possibly of different types. An example of student record struct StudRecord { char name[50]; int id; int age; int major; }; 48

49 Data Structures (cont.) A data structure is also a data type struct StudRecord my_record; struct StudRecord * pointer; pointer = & my_record; Accessing a field inside a data structure my_record.id = 10; // or pointer->id = 10; 49

50 Data Structures (cont.) Allocating a data structure instance This is a new type now struct StudRecord* pstudentrecord; pstudentrecord = (StudRecord*)malloc(sizeof(struct StudRecord)); pstudentrecord ->id = 10; IMPORTANT: Never calculate the size of a data structure yourself. Rely on the sizeof() function Example: Because of memory padding, the size of struct StudRecord is 64 (instead of 62 as one might estimate) 50

51 The typedef Construct struct StudRecord { char name[50]; int id; int age; int major; }; Using typedef to improve readability typedef struct StudRecord RECORD_t; int main() { RECORD_t my_record; strcpy_s(my_record.name, Joe Doe ); my_record.age = 20; my_record.id = 6114; 51 } RECORD_t* p = &my_record; p->major = 643; return 0;

52 Arrays Arrays in C are composed of a particular type, laid out in memory in a repeating pattern. Array elements are accessed by stepping forward in memory from the base of the array by a multiple of the element size. /* define an array of 10 chars */ char x[5] = { t, e, s, t, \0 }; /* access element 0, change its value */ x[0] = T ; /* pointer arithmetic to get elt 3 */ char elt3 = *(x+3); /* x[3] */ Brackets specify the count of elements. Initial values optionally set in braces. Arrays in C are 0-indexed (here, 0 4) x[3] == *(x+3) == t (notice, it s not s!) /* x[0] evaluates to the first element; * x evaluates to the address of the * first element, or &(x[0]) */ /* 0-indexed for loop idiom */ #define COUNT 10 char y[count]; int i; for (i=0; i<count; i++) { /* process y[i] */ printf( %c\n, y[i]); } For loop that iterates from 0 to COUNT-1. Symbol Addr Value char x [0] 100 t char x [1] 101 e char x [2] 102 s char x [3] 103 t char x [4] 104 \0 Q: What s the difference between char x[5] and a declaration like char *x? 52

53 How to Parse and Define C Types At this point we have seen a few basic types, arrays, pointer types, and structures. So far we ve glossed over how types are named. int x; /* int; */ typedef int T; int *x; /* pointer to int; */ typedef int *T; int x[10]; /* array of ints; */ typedef int T[10]; int *x[10]; /* array of pointers to int; */ typedef int *T[10]; int (*x)[10]; /* pointer to array of ints; */ typedef int (*T)[10]; typedef defines a new type C type names are parsed by starting at the type name and working outwards according to the rules of precedence: int *x[10]; int (*x)[10]; x is an array of pointers to int x is a pointer to an array of int Arrays are the primary source of confusion. When in doubt, use extra parens to clarify the expression. REMEMBER THIS: (), which stands for function, and [], which stands for array, have higher precedence than *, which stands for pointer 53

54 Function Types Another less obvious construct is the pointer to function type. For example, qsort: (a sort function in the standard library) void qsort(void *base, size_t nmemb, size_t size, int (*compar)(const void *, const void *)); /* function matching this type: */ int cmp_function(const void *x, const void *y); /* typedef defining this type: */ typedef int (*cmp_type) (const void *, const void *); The last argument is a comparison function const means the function is not allowed to modify memory via this pointer. /* rewrite qsort prototype using our typedef */ void qsort(void *base, size_t nmemb, size_t size, cmp_type compar); size_t is an unsigned int void * is a pointer to memory of unknown type. 54

55 Row Major and Column Major REAL * A

56 References Linux/Unix Introduction VI Editor C Programming Tutorial Compiler, Assembler, Linker and Loader: A Brief Story 56

57 Backup and More 57

58 Vector/Matrix and Array in C C has row-major storage for multiple dimensional array A[2,2] is followed by A[2,3] 3-dimensional array B[3][100][100] char A[4][4] Memory address of A[2][3] = A[0][0] + offset = A + sizeof (char) * (2 * # columns + 3) = * (2 * 4 + 3) = 11 58

59 Store Array in Memory in Row Major 3x4 matrix C array for matrix: int A[3][4] C storage type Address of element A[1][2]? Address of element A[1][2]? = A + sizeof (int) * (1 * 4 + 2) = A + 4 * 6 = A

60 Store Array in Memory in Column Major 3x4 matrix

61 For a Memory Region to Store Data for an Array in Either Row or Col Major 3 X X

62 Compiler A programming language is an artificial language that can be used to control the behavior of a machine, particularly a computer. A compiler is a computer program (or set of programs) that translates text written in a computer language (the source language) into another computer language (the target language). The original sequence is usually called the source code and the output called object code. Commonly the output has a form suitable for processing by other programs (e.g., a linker), but it may be a human-readable text file. 62

63 Debug and Performance Analysis Debugging is a methodical process of finding and reducing the number of bugs, or defects, in a computer program or a piece of electronic hardware thus making it behave as expected. In software engineering, performance analysis (a field of dynamic program analysis) is the investigation of a program's behavior using information gathered as the program runs, as opposed to static code analysis. The usual goal of performance analysis is to determine which parts of a program to optimize for speed or memory usage. A profiler is a performance analysis tool that measures the behavior of a program as it runs, particularly the frequency and duration of function calls. The output is a stream of recorded events (a trace) or a statistical summary of the events observed (a profile). 63

64 Optimization In computing, optimization is the process of modifying a system to make some aspect of it work more efficiently or use less resources. For instance, a computer program may be optimized so that it executes more rapidly, or is capable of operating within a reduced amount of memory storage, or draws less battery power in a portable computer. The system may be a single computer program, a collection of computers or even an entire network such as the Internet. 64 (

65 Object module structure Header section Machine code section (a.k.a. text section) Initialized data section Symbol table section Relocation information section

66 A sample C program: #include <stdio.h> int a[10]={0,1,2,3,4,5,6,7,8,9}; int b[10]; void main() { int i; static int k = 3; } for(i = 0; i < 10; i++) { printf("%d\n",a[i]); b[i] = k*a[i]; }

67 Object module of the sample C program: Offset Contents Comment Header section number of bytes of Machine code section 4 44 number of bytes of initialized data section 8 40 number of bytes of Uninitialized data section (array b[]) (not part of this object module) number of bytes of Symbol table section number of bytes of Relocation information section Machine code section (124 bytes) 20 X code for the top of the for loop (36 bytes) 56 X code for call to printf() (22 bytes) 68 X code for the assignment statement (10 bytes) 88 X code for the bottom of the for loop (4 bytes) 92 X code for exiting main() (52 bytes) Initialized data section (44 bytes) beginning of array a[] : end of array a[] (40 bytes) variable k (4 bytes) Symbol table section (60 bytes) 188 X array a[] : offset 0 in Initialized data section (12 bytes) 200 X variable k : offset 40 in Initialized data section (10 bytes) 210 X array b[] : offset 0 in Uninitialized data section (12 bytes) 222 X main : offset 0 in Machine code section (12 bytes) 234 X printf : external, used at offset 56 of Machine code section (14 bytes) Relocation information section (44 bytes) 248 X relocation information

68 Creation of load module Object Module A Load Module Header Section Machine Code Section Initialized data Section Symbol table Section Header Section Machine Code Section Header Section Machine Code Section Initialized data Section Symbol table Section Object Module B Initialized data Section Symbol table Section

69 Loading and memory mapping Header Section Machine Code Section Initialized data Section Code initialized Static data Code Static data (logical) address space of program 2 Symbol table Section uninitialized Code load module (logical) address space of program 1 Dynamic data Unused logical address space Stack Dynamic data Unused logical address space Stack Static data Dynamic data Unused Logical address space Stack loading memory mapping OPERATING SYSTEM PHYSICAL MEMORY memory mapping (logical) address space of program 3

70 From source program to placement in memory during execution physical memory code for printf() source program int a[10]={0,1,2,3,4,5,6,7,8,9}; int b[10]; void main() { int i; static int k = 3; for(i = 0; i < 10; i++) { printf("%d\n",a[i]); b[i] = k*a[i]; }/*endfor*/ }/*end main*/ code for top of for loop code for call to printf() code for b[i] = k*a[i] array a[] array b[] variable k

71 Dynamic memory allocation Code Code initialized initialized Static data Static data uninitialized uninitialized Dynamic data Dynamic data Unused logical address space increment of dynamic data Unused logical address space Stack Stack (logical) address space of the program OPERATING SYSTEM (logical) address space of the program OPERATING SYSTEM PHYSICAL MEMORY Before dynamic memory allocation PHYSICAL MEMORY After dynamic memory allocation

72 Overview of memory management Stack-allocated memory When a function is called, memory is allocated for all of its parameters and local variables. Each active function call has memory on the stack (with the current function call on top) When a function call terminates, the memory is deallocated ( freed up ) g() Ex: main() calls f(), f() calls g() g() recursively calls g() g() f() main() 72

73 Overview of memory management Heap-allocated memory This is used for persistent data, that must survive beyond the lifetime of a function call global variables dynamically allocated memory C statements can create new heap data (similar to new in Java/C++) Heap memory is allocated in a more complex way than stack memory Like stack-allocated memory, the underlying system determines where to get more memory the programmer doesn t have to search for free memory space! 73

74 Note: void * denotes a generic pointer type Allocating new heap memory void *malloc(size_t size); Allocate a block of size bytes, return a pointer to the block (NULL if unable to allocate block) void *calloc(size_t num_elements, size_t element_size); Allocate a block of num_elements * element_size bytes, initialize every byte to zero, return pointer to the block (NULL if unable to allocate block) 74

75 Allocating new heap memory void *realloc(void *ptr, size_t new_size); Given a previously allocated block starting at ptr, change the block size to new_size, return pointer to resized block If block size is increased, contents of old block may be copied to a completely different region In this case, the pointer returned will be different from the ptr argument, and ptr will no longer point to a valid memory region If ptr is NULL, realloc is identical to malloc Note: may need to cast return value of malloc/calloc/realloc: char *p = (char *) malloc(buffer_size); 75

76 Deallocating heap memory void free(void *pointer); Given a pointer to previously allocated memory, put the region back in the heap of unallocated memory Note: easy to forget to free memory when no longer needed... especially if you re used to a language with garbage collection like Java This is the source of the notorious memory leak problem Difficult to trace the program will run fine for some time, until suddenly there is no more memory! 76

77 Memory errors Using memory that you have not initialized Using memory that you do not own Using more memory than you have allocated Using faulty heap memory management 77

78 Using memory that you have not initialized Uninitialized memory read Uninitialized memory copy not necessarily critical unless a memory read follows void foo(int *pi) { int j; *pi = j; /* UMC: j is uninitialized, copied into *pi */ } void bar() { int i=10; foo(&i); printf("i = %d\n", i); /* UMR: Using i, which is now junk value */ } 78

79 Using memory that you don t own Null pointer read/write Zero page read/write typedef struct node { struct node* next; int val; } Node; What if head is NULL? int findlastnodevalue(node* head) { while (head->next!= NULL) { /* Expect NPR */ head = head->next; } return head->val; /* Expect ZPR */ } 79

80 Using memory that you don t own Invalid pointer read/write Pointer to memory that hasn t been allocated to program void genipr() { int *ipr = (int *) malloc(4 * sizeof(int)); int i, j; i = *(ipr ); j = *(ipr ); /* Expect IPR */ free(ipr); } void genipw() { int *ipw = (int *) malloc(5 * sizeof(int)); *(ipw ) = 0; *(ipw ) = 0; /* Expect IPW */ free(ipw); } 80

81 Using memory that you don t own Common error in 64-bit applications: ints are 4 bytes but pointers are 8 bytes If prototype of malloc() not provided, return value will be cast to a 4-byte int /*Forgot to #include <malloc.h>, <stdlib.h> in a 64-bit application*/ void illegalpointer() { } int *pi = (int*) malloc(4 * sizeof(int)); pi[0] = 10; /* Expect IPW */ Four bytes will be lopped off this value resulting in an invalid pointer value printf("array value = %d\n", pi[0]); /* Expect IPR */ 81

82 Using memory that you don t own Free memory read/write Access of memory that has been freed earlier int* init_array(int *ptr, int new_size) { ptr = (int*) realloc(ptr, new_size*sizeof(int)); memset(ptr, 0, new_size*sizeof(int)); return ptr; } Remember: realloc may move entire block int* fill_fibonacci(int *fib, int size) { int i; /* oops, forgot: fib = */ init_array(fib, size); /* fib[0] = 0; */ fib[1] = 1; for (i=2; i<size; i++) } fib[i] = fib[i-1] + fib[i-2]; return fib; What if array is moved to new location? 82

83 Using memory that you don t own Beyond stack read/write char *append(const char* s1, const char *s2) { const int MAXSIZE = 128; char result[128]; result is a local array name int i=0, j=0; stack memory allocated for (j=0; i<maxsize-1 && j<strlen(s1); i++,j++) { result[i] = s1[j]; } for (j=0; i<maxsize-1 && j<strlen(s2); i++,j++) { result[i] = s2[j]; } result[++i] = '\0'; } return result; Function returns pointer to stack memory won t be valid after function returns 83

84 Using memory that you haven t allocated Array bound read/write void genabrandabw() { const char *name = Safety Critical"; char *str = (char*) malloc(10); strncpy(str, name, 10); str[11] = '\0'; /* Expect ABW */ printf("%s\n", str); /* Expect ABR */ } 84

85 Memory leak Faulty heap management int *pi; void foo() { pi = (int*) malloc(8*sizeof(int)); /* Allocate memory for pi */ /* Oops, leaked the old memory pointed to by pi */ free(pi); /* foo() is done with pi, so free it */ } void main() { pi = (int*) malloc(4*sizeof(int)); /* Expect MLK: foo leaks it */ foo(); } 85

86 Faulty heap management Potential memory leak no pointer to the beginning of a block not necessarily critical block beginning may still be reachable via pointer arithmetic int *plk = NULL; void genplk() { plk = (int *) malloc(2 * sizeof(int)); /* Expect PLK as pointer variable is incremented past beginning of block */ plk++; } 86

87 Faulty heap management Freeing non-heap memory Freeing unallocated memory void genfnh() { int fnh = 0; free(&fnh); /* Expect FNH: freeing stack memory */ } void genfum() { int *fum = (int *) malloc(4 * sizeof(int)); free(fum+1); /* Expect FUM: fum+1 points to middle of a block */ free(fum); free(fum); /* Expect FUM: freeing already freed memory */ } 87

88 The C Language Prof. Stephen A. Edwards

89 The C Language Currently, the most commonly-used language for embedded systems High-level assembly Very portable: compilers exist for virtually every processor Easy-to-understand compilation Produces efficient code Fairly concise

90 C History Developed between 1969 and 1973 along with Unix Due mostly to Dennis Ritchie Designed for systems programming Operating systems Utility programs Compilers Filters Evolved from B, which evolved from BCPL

91 BCPL Designed by Martin Richards (Cambridge) in 1967 Typeless Everything an n-bit integer (a machine word) Pointers (addresses) and integers identical Memory is an undifferentiated array of words Natural model for word-addressed machines Local variables depend on frame-pointer-relative addressing: dynamically-sized automatic objects not permitted Strings awkward Routines expand and pack bytes to/from word arrays

92 C History Original machine (DEC PDP-11) was very small 24K bytes of memory, 12K used for operating system Written when computers were big, capital equipment Group would get one, develop new language, OS

93 C History Many language features designed to reduce memory Forward declarations required for everything Designed to work in one pass: must know everything No function nesting PDP-11 was byte-addressed Now standard Meant BCPL s word-based model was insufficient

94 #include <stdio.h> Hello World in C void main() { printf( Hello, world!\n ); } Preprocessor used to share information among source files - Clumsy + Cheaply implemented + Very flexible

95 #include <stdio.h> Hello World in C void main() { printf( Hello, world!\n ); } Program mostly a collection of functions main function special: the entry point void qualifier indicates function does not return anything I/O performed by a library function: not included in the language

96 Euclid s algorithm in C int gcd(int m, int n) { int r; while ( (r = m % n)!= 0) { } m = n; n = r; } return n; New Style function declaration lists number and type of arguments Originally only listed return type. Generated code did not care how many arguments were actually passed. Arguments are call-by-value

97 Euclid s algorithm in C int gcd(int m, int n) { int r; while ( (r = m % n)!= 0) { } m = n; n = r; } return n; Frame pointer n m ret. addr. r Excess arguments simply ignored Stack pointer Automatic variable Storage allocated on stack when function entered, released when it returns. All parameters, automatic variables accessed w.r.t. frame pointer. Extra storage needed while evaluating large expressions also placed on the stack

98 Euclid s algorithm in C int gcd(int m, int n) { int r; while ( (r = m % n)!= 0) { } m = n; n = r; } return n; Expression: C s basic type of statement. Arithmetic and logical Assignment (=) returns a value, so can be used in expressions % is remainder!= is not equal

99 Euclid s algorithm in C int gcd(int m, int n) { int r; while ( (r = m % n)!= 0) { m = n; n = r; } } return n; Each function returns a single value, usually an integer. Returned through a specific register by convention. High-level control-flow statement. Ultimately becomes a conditional branch. Supports structured programming

100 Euclid Compiled on PDP-11.globl _gcd r0-r7.text PC is r7, SP is r6, FP is r5 _gcd: jsr r5,rsave save sp in frame pointer r5 L2:mov 4(r5),r1 r1 = n sxt r0 sign extend div 6(r5),r0 m / n = r0,r1 mov r1,-10(r5) r = m % n jeq L3 mov 6(r5),4(r5) m = n mov -10(r5),6(r5) n = r jbr L2 L3:mov 6(r5),r0 return n in r0 jbr L1 L1:jmp rretrn restore sp ptr, return int gcd(int m, int n) { int r; while ( (r = m % n)!= 0) { m = n; n = r; } return n; }

101 Euclid Compiled on PDP-11.globl _gcd.text _gcd: jsr r5,rsave L2:mov 4(r5),r1 sxt r0 div 6(r5),r0 mov r1,-10(r5) jeq L3 mov 6(r5),4(r5) mov -10(r5),6(r5) jbr L2 L3:mov 6(r5),r0 jbr L1 L1:jmp rretrn Very natural mapping from C into PDP-11 instructions. Complex addressing modes make framepointer-relative accesses easy. Another idiosyncrasy: registers were memory-mapped, so taking address of a variable in a register is straightforward.

102 Pieces of C Types and Variables Definitions of data in memory Expressions Arithmetic, logical, and assignment operators in an infix notation Statements Sequences of conditional, iteration, and branching instructions Functions Groups of statements and variables invoked recursively

103 C Types Basic types: char, int, float, and double Meant to match the processor s native types Natural translation into assembly Fundamentally nonportable Declaration syntax: string of specifiers followed by a declarator Declarator s notation matches that in an expression Access a symbol using its declarator and get the basic type back

104 int i; C Type Examples Integer int *j, k; j: pointer to integer, int k unsigned char *ch; ch: pointer to unsigned char float f[10]; Array of 10 floats char nextchar(int, 2-arg function char*); Array of three arrays of five int a[3][5][10]; int *func1(float); function returning int * int (*func2)(void); pointer to function returning int

105 C Typedef Type declarations recursive, complicated. Name new types with typedef Instead of int (*func2)(void) use typedef int func2t(void); func2t *func2;

106 C Structures A struct is an object with named fields: struct { char *name; int x, y; int h, w; } box; Accessed using dot notation: box.x = 5; box.y = 2;

107 Struct bit-fields Way to aggressively pack data in memory struct { unsigned int baud : 5; unsigned int div2 : 1; unsigned int use_external_clock : 1; } flags; Compiler will pack these fields into words Very implementation dependent: no guarantees of ordering, packing, etc. Usually less efficient Reading a field requires masking and shifting

108 C Unions Can store objects of different types at different times union { int ival; float fval; char *sval; }; Useful for arrays of dissimilar objects Potentially very dangerous Good example of C s philosophy Provide powerful mechanisms that can be abused

109 Alignment of data in structs Most processors require n-byte objects to be in memory at address n*k Side effect of wide memory busses E.g., a 32-bit memory bus Read from address 3 requires two accesses, shifting

110 Alignment of data in structs Compilers add padding to structs to ensure proper alignment, especially for arrays Pad to ensure alignment of largest object (with biggest requirement) struct { char a; int b; char c; } Moral: rearrange to save memory b b b b Pad b b a b c a b c

111 #include <stdlib.h> C Storage Classes int global_static; static int file_static; void foo(int auto_param) { static int func_static; int auto_i, auto_a[10]; double *auto_d = malloc(sizeof(double)*5); } Linker-visible. Allocated at fixed location Visible within file. Allocated at fixed location. Visible within func. Allocated at fixed location.

112 #include <stdlib.h> C Storage Classes int global_static; static int file_static; void foo(int auto_param) { static int func_static; int auto_i, auto_a[10]; double *auto_d = malloc(sizeof(double)*5); } Space allocated on stack by caller. Space allocated on stack by function. Space allocated on heap by library routine.

113 malloc() and free() Library routines for managing the heap int *a; a = (int *) malloc(sizeof(int) * k); a[5] = 3; free(a); Allocate and free arbitrary-sized chunks of memory in any order

114 malloc() and free() More flexible than automatic variables (stacked) More costly in time and space malloc() and free() use complicated non-constant-time algorithms Each block generally consumes two additional words of memory Pointer to next empty block Size of this block Common source of errors Using uninitialized memory Using freed memory Not allocating enough Neglecting to free disused blocks (memory leaks)

115 malloc() and free() Memory usage errors so pervasive, entire successful company (Pure Software) founded to sell tool to track them down Purify tool inserts code that verifies each memory access Reports accesses of uninitialized memory, unallocated memory, etc. Publicly-available Electric Fence tool does something similar

116 Dynamic Storage Allocation What are malloc() and free() actually doing? Pool of memory segments: Free malloc( )

117 Dynamic Storage Allocation Rules: Each segment contiguous in memory (no holes) Segments do not move once allocated malloc() Find memory area large enough for segment Mark that memory is allocated free() Mark the segment as unallocated

118 Three issues: Dynamic Storage Allocation How to maintain information about free memory The algorithm for locating a suitable block The algorithm for freeing an allocated block

119 Simple Dynamic Storage Allocation Three issues: How to maintain information about free memory Linked list The algorithm for locating a suitable block First-fit The algorithm for freeing an allocated block Coalesce adjacent free blocks

120 Simple Dynamic Storage Allocation Next Size Size Next Size Free block Allocated block malloc( ) First large-enough free block selected Free block divided into two Previous next pointer updated Newly-allocated region begins with a size value

121 Simple Dynamic Storage Allocation free(a) Appropriate position in free list identified Newly-freed region added to adjacent free regions

122 Dynamic Storage Allocation Many, many variants Other fit algorithms Segregation of objects by sizes 8-byte objects in one region, 16 in another, etc. More intelligent list structures

123 Memory Pools An alternative: Memory pools Separate management policy for each pool Stack-based pool: can only free whole pool at once Very cheap operation Good for build-once data structures (e.g., compilers) Pool for objects of a single size Useful in object-oriented programs Not part of the C standard library

124 Arrays Array: sequence of identical objects in memory int a[10]; means space for ten integers By itself, a is the address of the first integer *a and a[0] mean the same thing Filippo Brunelleschi, Ospdale degli Innocenti, Firenze, Italy, 1421 The address of a is not stored in memory: the compiler inserts code to compute it when it appears Ritchie calls this interpretation the biggest conceptual jump from BCPL to C

125 Seagram Building, Ludwig Mies van der Rohe,1957 Multidimensional Arrays Array declarations read right-to-left int a[10][3][2]; an array of ten arrays of three arrays of two ints In memory

126 Multidimensional Arrays Passing a multidimensional array as an argument requires all but the first dimension int a[10][3][2]; void examine( a[][3][2] ) { } Address for an access such as a[i][j][k] is a + k + 2*(j + 3*i)

127 Multidimensional Arrays Use arrays of pointers for variable-sized multidimensional arrays You need to allocate space for and initialize the arrays of pointers int ***a; a[3][5][4] expands to *(*(*(a+3)+5)+4) int ***a The value int ** int * int

128 C Expressions Traditional mathematical expressions y = a*x*x + b*x + c; Very rich set of expressions Able to deal with arithmetic and bit manipulation

129 C Expression Classes arithmetic: + * / % comparison: ==!= < <= > >= bitwise logical: & ^ ~ shifting: << >> lazy logical: &&! conditional:? : assignment: = += -= increment/decrement: sequencing:, pointer: * -> & []

130 Bitwise operators and: & or: xor: ^ not: ~ left shift: << right shift: >> Useful for bit-field manipulations #define MASK 0x040 if (a & MASK) { } /* Check bits */ c = MASK; /* Set bits */ c &= ~MASK; /* Clear bits */ d = (a & MASK) >> 4; /* Select field */

131 Lazy Logical Operators Short circuit tests save time if ( a == 3 && b == 4 && c == 5 ) { } equivalent to if (a == 3) { if (b ==4) { if (c == 5) { } } } Evaluation order (left before right) provides safety if ( i <= SIZE && a[i] == 0 ) { }

132 c = a < b? a + 1 : b 1; Conditional Operator Evaluate first expression. If true, evaluate second, otherwise evaluate third. Puts almost statement-like behavior in expressions. BCPL allowed code in an expression: a := 5 + valof{ int i, s = 0; for (i = 0 ; i < 10 ; i++) s += a[i]; return s; }

133 Side-effects in expressions Evaluating an expression often has side-effects a++ increment a afterwards a = 5 a = foo() changes the value of a function foo may have side-effects

134 Pointer Arithmetic From BCPL s view of the world Pointer arithmetic is natural: everything s an integer int *p, *q; *(p+5) equivalent to p[5] If p and q point into same array, p q is number of elements between p and q. Accessing fields of a pointed-to structure has a shorthand: p->field means (*p).field

135 C Statements Expression Conditional if (expr) { } else { } switch (expr) { case c1: case c2: } Iteration while (expr) { } zero or more iterations do while (expr) at least one iteration for ( init ; valid ; next ) { } Jump goto label continue; go to start of loop break; exit loop or switch return expr; return from function

136 The Switch Statement Performs multi-way branches switch (expr) { case 1: break; case 5: case 6: break; default: break; } tmp = expr; if (tmp == 1) goto L1 else if (tmp == 5) goto L5 else if (tmp == 6) goto L6 else goto Default; L1: goto Break; L5:; L6: goto Break; Default: goto Break; Break:

137 Switch Generates Interesting Code Sparse case labels tested sequentially if (e == 1) goto L1; else if (e == 10) goto L2; else if (e == 100) goto L3; Dense cases use a jump table table = { L1, L2, Default, L4, L5 }; if (e >= 1 and e <= 5) goto table[e]; Clever compilers may combine these

138 setjmp/longjmp A way to exit from deeply nested functions A hack now a formal part of the standard library #include <setjmp.h> jmp_buf jmpbuf; Space for a return address and registers (including stack pointer, frame pointer) void top(void) { Stores context, returns 0 switch (setjmp(jmpbuf)) { case 0: child(); break; case 1: /* longjmp called */ break; } } Returns to context, making it appear void deeplynested() { longjmp(jmpbuf, setjmp() returned 1 1); }

139 The Macro Preprocessor Relatively late and awkward addition to the language Symbolic constants #define PI Macros with arguments for emulating inlining #define min(x,y) ((x) < (y)? (x) : (y)) Conditional compilation #ifdef STDC File inclusion for sharing of declarations #include myheaders.h

140 Macro Preprocessor Pitfalls Header file dependencies usually form a directed acyclic graph (DAG) How do you avoid defining things twice? Convention: surround each header (.h) file with a conditional: #ifndef MYHEADER_H #define MYHEADER_H /* Declarations */ #endif

141 Macro Preprocessor Pitfalls Macros with arguments do not have function call semantics Function Call: Each argument evaluated once, in undefined order, before function is called Macro: Each argument evaluated once every time it appears in expansion text

142 Macro Preprocessor pitfalls Example: the min function int min(int a, int b) { if (a < b) return a; else return b; } #define min(a,b) ((a) < (b)? (a) : (b)) Identical for min(5,x) Different when evaluating expression has side-effect: min(a++,b) min function increments a once min macro may increment a twice if a < b

143 Macro Preprocessor Pitfalls Text substitution can expose unexpected groupings #define mult(a,b) a*b mult(5+3,2+4) Expands to * Operator precedence evaluates this as 5 + (3*2) + 4 = 15 not (5+3) * (2+4) = 48 as intended Moral: By convention, enclose each macro argument in parenthesis: #define mult(a,b) (a)*(b)

144 Nondeterminism in C Library routines malloc() returns a nondeterministically-chosen address Address used as a hash key produces nondeterministic results Argument evaluation order myfunc( func1(), func2(), func3() ) func1, func2, and func3 may be called in any order Word sizes int a; a = 1 << 16; /* Might be zero */ a = 1 << 32; /* Might be zero */

145 Nondeterminism in C Uninitialized variables Automatic variables may take values from stack Global variables left to the whims of the OS Reading the wrong value from a union union { int a; float b; } u; u.a = 10; printf( %g, u.b); Pointer dereference *a undefined unless it points within an allocated array and has been initialized Very easy to violate these rules Legal: int a[10]; a[-1] = 3; a[10] = 2; a[11] = 5; int *a, *b; a - b only defined if a and b point into the same array