APPENDIX Summary of Benchmarks

Transcription

1 158 APPENDIX Summary of Benchmarks The experimental results presented throughout this thesis use programs from four benchmark suites: Cyclone benchmarks (available from [Cyc]): programs used to evaluate the Cyclone tool [Jim + 02]. The ones we tested are small but computationally intensive applications that make heavy use of arrays and pointers. aes: Rijndael block-cipher encryption. cacm: Adaptive arithmetic coding for data compression. cfrac: Continued fraction algorithm. grobner: Gröbner bases computation. matxmult: Matrix multiplication. ppm: Arithmetic encoding and decoding. tile: Text document partitioning into tiles. Olden benchmarks (available from [Olden]): programs used to evaluate the Olden C compiler [Car + 95]. These are relatively small programs that each perform a monolithic task, using a variety of dynamically allocated data structures. bh: Barnes-Hut N-body force-computation algorithm; uses a heterogeneous octree. bisort: Forward and backward sort of integers using 2 disjoint bitonic sequences that are merged to obtain the sorted result; uses a binary tree.

2 159 em3d: Electromagnetic wave propagation in a 3D object; uses singly-linked lists. health: Columbian healthcare simulation; uses doubly-linked lists. mst: Minimum spanning tree of a graph; uses an array of singly-linked lists. perimeter: Perimeters of regions in images; uses a quad-tree. power: Power pricing system optimization problem solver; uses an N-way tree and singly-linked lists. treeadd: Recursive sum of values in a balanced B-tree. tsp: Traveling-salesman-problem solver using a partitioning algorithm and a closest point heuristic; uses a balanced binary tree. Spec CPU95 [SPEC]: includes all the C programs from the integer (CINT) suite. compress: An in-memory version of the common UNIX utility. gcc: Based on the GNU C compiler version go: An internationally ranked go-playing program. ijpeg: Image compression/decompression on in-memory images. li: Xlisp interpreter. m88ksim: A chip simulator for the Motorola microprocessor. perl: An interpreter for the Perl language. vortex: An object oriented database. Spec CPU2000 [SPEC]: includes select C programs from both the integer (CINT) and floating point (CFP) suites. ammp (CFP): Computational chemistry. art (CFP): Image recognition / neural networks. bzip2: Compression.

3 160 crafty: Game playing: chess. equake (CFP): Seismic wave propagation simulation. gap: Group theory, interpreter. gzip: Compression. mcf: Combinatorial optimization. mesa (CFP): 3-D graphics library. parser: Word processing. twolf: Place and route simulator. vpr: FPGA circuit placement and routing. Tables A.1 and A.2 list the programs used in our experiments, along with their size (in lines of code), baseline compilation time (wallclock time, in seconds), and baseline execution times (wallclock time, in seconds). Inputs were selected to give reasonable running times for comparison: for the Cyclone and Olden benchmarks, the inputs we used are listed in Table A.1 (either command-line arguments, or input files supplied with the benchmarks). For the SPEC benchmarks, we used two different datasets, which we call the slow and fast datasets. The slow dataset is the ref set for Spec 95 and the train set for Spec 2000, and is used to evaluate the more efficient Memory-Safety Enforcer (MSE) and Sensitive Location Checker (SLC), in Chapters 3 7. The fast dataset is the train set for Spec 95 and the test set for Spec 2000, and is used to evaluate the slower Runtime Type Checker (RTC) in Chapters In Figure A.2, columns (c) and (d) give the baseline execution times for the slow and fast datasets respectively. The programs were compiled with gcc (version 3.3.2) and executed on a 1GHz Pentium III with 512MB RAM, running Linux (RedHat 9). For the MSE and SLC experiments, programs were compiled with -O3 optimizations, while for the RTC experiments, optimizations were disabled (-O0) because they slowed down compilation time considerably, and we felt that the typical usage of the RTC as a debugging tool would be to compile programs without optimization.

4 161 Compile Exec LOC Time (s) Time (s) Input Program (a) (b) (c) (d) Cyclone aes 1, cacm encode test2 cfrac 4, grobner 4, eg03 matxmult 1, ppm 1, decode test1 tile 4, sample2 Olden bisort em3d health mst perimeter power treeadd tsp Table A.1 Benchmark Information

5 162 Compile Exec Time (s) LOC Time (s) slow fast Program (a) (b) (c) (d) Spec 95 (ref) (train) compress 3, gcc 205, go 29, ijpeg 31, li 7, m88ksim 19, perl 26, vortex 67, Spec 2000 (train) (test) ammp 13, art 1, bzip2 4, crafty 20, equake 1, gap 71, gzip 8, mcf 2, mesa 58, parser 11, twolf 20, vpr 17, Table A.2 Benchmark Information