SimBench. A Portable Benchmarking Methodology for Full-System Simulators. Harry Wagstaff Bruno Bodin Tom Spink Björn Franke

Transcription

1 SimBench A Portable Benchmarking Methodology for Full-System Simulators Harry Wagstaff Bruno Bodin Tom Spink Björn Franke Institute for Computing Systems Architecture University of Edinburgh ISPASS

2 Motivation Instruction Set Simulation Evaluating Simulation Tools The SimBench Methodology Overview Implementation Porting SimBench Evaluation 2

3 Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: Motivation Instruction Set Simulation 3

4 Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: Design Space Exploration Gem5 Multi2Sim Motivation Instruction Set Simulation 3

5 Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: Design Space Exploration Software Development QEMU Android Emulator Motivation Instruction Set Simulation 3

6 Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: Design Space Exploration Software Development Backwards Compatibility Apple Rosetta Nintendo NES Classic Motivation Instruction Set Simulation 3

7 Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: Design Space Exploration Software Development Backwards Compatibility Simulators can be broken up into several categories: Functional Only Profiling/Performance Modelling User-Mode Full System Motivation Instruction Set Simulation 3

8 User-Mode Simulation Simulated CPU Flat Memory Execute a single binary Emulate System Calls Simplified Memory System Syscall Emulation Layer Host Console Host Timers Host File System Motivation Instruction Set Simulation 4

9 Full-System Simulation Simulated CPU Boot out of reset System-mode Instructions Simulated MMU Physical Memory Memory Translation Asynchronous Interrupts External Devices Simulated Serial Port Simulated Timers Simulated Storage Device Host Console Host Timers Host File System Motivation Instruction Set Simulation 5

10 Fast Simulation We want to make our simulators go fast! To be clear, we want the simulator itself to be efficient. Motivation Instruction Set Simulation 6

11 Fast Simulation We want to make our simulators go fast! To be clear, we want the simulator itself to be efficient. How? Dynamic Binary Translation Memory-Related Techniques Efficient Interrupt Modelling Control Flow Handling Motivation Instruction Set Simulation 6

12 Fast Simulation We want to make our simulators go fast! To be clear, we want the simulator itself to be efficient. How? Dynamic Binary Translation Memory-Related Techniques Efficient Interrupt Modelling Control Flow Handling How do we evaluate the effectiveness of these techniques in spite of complex interactions? Motivation Instruction Set Simulation 6

13 Typical Benchmarking Approaches Most common approaches fall into two categories: Large, complex macrobenchmark suites Small, targeted but ad-hoc microbenchmarks Both approaches are problematic for several reasons. Motivation Evaluating Simulation Tools 7

14 Macrobenchmarking The most common approach is to use large macrobenchmarks However, this presents several problems: Benchmark runtimes are long Difficult to perform detailed analysis Aggregated runtimes may hide interesting results Speedup sjeng SPEC (overall) mcf rc rc rc2 QEMU Version Motivation Evaluating Simulation Tools 8

15 Ad-Hoc Microbenchmarking Many papers also attempt to use ad-hoc microbenchmarks to assess performance. However, these also have problems: Source code may be unavailable, damaging reproducibility Secondary effects may not be taken into account Motivation Evaluating Simulation Tools 9

16 Sum Two Arrays - Source Code float out[array_size]; float a[array_size]; float b[array_size];... void foo() { for(int i = 0; i < ARRAY_SIZE; ++i) { out[i] = a[i] + b[i]; } } Motivation Evaluating Simulation Tools 10

17 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Motivation Evaluating Simulation Tools 11

18 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Instruction Page fault, Code generation Motivation Evaluating Simulation Tools 11

19 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Cold memory access, Data Page fault Motivation Evaluating Simulation Tools 11

20 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access Hot memory access Motivation Evaluating Simulation Tools 11

21 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access FP Operation FP Operation Motivation Evaluating Simulation Tools 11

22 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access FP Operation Hot memory access Motivation Evaluating Simulation Tools 11

23 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access FP Operation Direct Control Flow Interrupt Direct control flow, Interrupt Motivation Evaluating Simulation Tools 11

24 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access FP Operation Direct Control Flow Interrupt Code generation Motivation Evaluating Simulation Tools 11

25 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access FP Operation Direct Control Flow Interrupt Code generation Motivation Evaluating Simulation Tools 11

26 Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page Fault Code Generation Cold Memory Access Data Page Fault Hot Memory Access FP Operation Direct Control Flow Interrupt Indirect Control Flow Indirect control flow, Interrupt Motivation Evaluating Simulation Tools 11

27 SimBench We present SimBench, which is designed to address some of these problems. The SimBench Methodology Overview 12

28 SimBench We present SimBench, which is designed to address some of these problems. SimBench... Contains a range of targeted microbenchmarks The SimBench Methodology Overview 12

29 SimBench We present SimBench, which is designed to address some of these problems. SimBench... Contains a range of targeted microbenchmarks Is aimed at full-system simulation techniques The SimBench Methodology Overview 12

30 SimBench We present SimBench, which is designed to address some of these problems. SimBench... Contains a range of targeted microbenchmarks Is aimed at full-system simulation techniques Is easily portable to new platforms and architectures The SimBench Methodology Overview 12

31 SimBench We present SimBench, which is designed to address some of these problems. SimBench... Contains a range of targeted microbenchmarks Is aimed at full-system simulation techniques Is easily portable to new platforms and architectures Runs bare-metal on the guest (i.e., without an OS) The SimBench Methodology Overview 12

32 Guest System Benchmarks Platform Library Arch. Library Timing Application Host System The SimBench Methodology Overview 13

33 Guest System Benchmarks Platform Library Arch. Library Timing Application Host System The SimBench Methodology Overview 14

34 Categories SimBench currently features five categories of benchmark: Code Generation Control Flow Exception Handling I/O Memory System A total of 18 benchmarks are in the suite. The SimBench Methodology Overview 15

35 Implementation The benchmarks are implemented entirely in C: No inline assembly in the benchmarks No weird intrinsics or builtins Some extensions used for code alignment The SimBench Methodology Implementation 16

36 Implementation - Defeating Optimisations We want to preserve interesting operations while still using our compiler optimisations: Use of empty volatile assembly statements Volatile memory accesses where necessary Indirect jumps via opaque pointers where necessary Benchmarks split into many objects to defeat inlining for(int i = 0; i < ITERATIONS; ++i) { volatile asm ("" ::: memory); } The SimBench Methodology Implementation 17

37 Implementation - Defeating Optimisations We want to preserve interesting operations while still using our compiler optimisations: Use of empty volatile assembly statements Volatile memory accesses where necessary Indirect jumps via opaque pointers where necessary Benchmarks split into many objects to defeat inlining static volatile uint32_t value = 0;... value = value; value = value; value = value; The SimBench Methodology Implementation 17

38 Implementation - Defeating Optimisations We want to preserve interesting operations while still using our compiler optimisations: Use of empty volatile assembly statements Volatile memory accesses where necessary Indirect jumps via opaque pointers where necessary Benchmarks split into many objects to defeat inlining static benchmark_kernel_t fn_table[];... while(iterations-- > 0) { fn_table[i % FN_TABLE_SIZE](); } The SimBench Methodology Implementation 17

39 Implementation - Defeating Optimisations We want to preserve interesting operations while still using our compiler optimisations: Use of empty volatile assembly statements Volatile memory accesses where necessary Indirect jumps via opaque pointers where necessary Benchmarks split into many objects to defeat inlining inter-page-direct.c: void ipd-fn-10();... ipd-fn-10(); inter-page-direct-10.c: void ipd-fn-9(); void ipd-fn-10() { ipd-fn-9(); } The SimBench Methodology Implementation 17

40 Guest System Benchmarks Platform Library Arch. Library Timing Application Host System The SimBench Methodology Porting SimBench 18

41 Porting To A New Platform To port SimBench to a new platform (where the architecture is already supported), 4 main components are required: A simple UART driver An IRQ Controller driver A description of the memory layout A linker script Lines of Code C ASM 0 ARM x86 The SimBench Methodology Porting SimBench 19

42 Porting To A New Architecture Porting to a new architecture is somewhat more complex. Bring system out of reset Manage page tables Manage interrupt/exception vectors Lines of Code C ASM 0 ARM X86 The SimBench Methodology Porting SimBench 20

43 Guest System Benchmarks Platform Library Arch. Library Timing Application Host System The SimBench Methodology Porting SimBench 21

44 Timing SimBench uses the Host system to collect benchmark timings: We don t trust the timing accuracy of the Guest Guest might report simulated cycles instead of real time Only needs platform-specific UART rather than Timer+IRQ+handler+... Guest Startup Prepare Benchmark Benchmark Kernel Cleanup Benchmark Guest Sends '[' Guest Sends ']' Host Time Benchmark Time The SimBench Methodology Porting SimBench 22

45 Evaluation We ran SimBench on a variety of platforms A range of full system simulators including QEMU A virtualized system, using KVM Natively on real hardware Evaluation 23

46 Categories - ARM Guest Simulators on x86 Host, QEMU-KVM and Hardware on ARM Host Speedup over QEMU-DBT Code Generation Control Flow Exception Handling I/O Memory System 0.01 QEMU-DBT SimIt-ARM Gem5 QEMU-KVM Hardware Evaluation 24

47 Categories Speedup v2.1.3 v2.1.2 v2.1.1 v2.1.0 v2.0.2 v2.0.1 v2.0.0 v1.7.2 v1.7.1 v1.7.0 Control Flow, QEMU-ARM on x86 Same-Page-Indirect Inter-Page-Indirect Same-Page-Direct Inter-Page-Direct v2.5.0-rc2 v2.5.0-rc1 v2.5.0-rc0 v2.4.1 v2.4.0 v v2.3.1 v2.3.0 v2.2.1 v2.2.0 Evaluation QEMU Version 25

48 Categories This represents 13% of executed instructions! Speedup v2.1.3 v2.1.2 v2.1.1 v2.1.0 v2.0.2 v2.0.1 v2.0.0 v1.7.2 v1.7.1 v1.7.0 Control Flow, QEMU-ARM on x86 Same-Page-Indirect Inter-Page-Indirect Same-Page-Direct Inter-Page-Direct v2.5.0-rc2 v2.5.0-rc1 v2.5.0-rc0 v2.4.1 v2.4.0 v v2.3.1 v2.3.0 v2.2.1 v2.2.0 Evaluation QEMU Version 26

49 Summary In summary: Existing benchmarking methods have several shortcomings SimBench is capable of addressing these shortcomings But SimBench does not replace application benchmarks Future Work: Additional categories of benchmark Ports to new architectures & platforms Improve robustness of existing benchmarks Summary 27

50 Thanks for Listening! Any Questions? SimBench is available now at 28

51 Benchmarks Code Generation Small Blocks Large Blocks Control Flow I/O Inter-Page Direct Inter-Page Indirect Intra-Page Direct Intra-Page Indirect Memory Mapped Device Coprocessor Access Exception Handling Data Access Fault Instruction Access Fault Undefined Instruction System Call External Software Interrupt Memory Cold Memory Access Hot Memory Access Nonprivileged Access TLB Eviction TLB Flush Extra Slides Benchmarks 29

52 Evaluation Platforms Machine ODROID-XU3 HP z440 CPU Exynos 5422 Xeon E v3 CPU GHz 2.0 (A15) 1.4 (A7) 3.5 (3.6 Boost) Memory 2GB 16GB Compiler gcc gcc OS Name Ubuntu Fedora 21 OS Kernel Extra Slides Benchmarks 30