USING JAVA HARTSTONE BENCHMARK IN A REAL-TIME SYSTEMS COURSE

Size: px

Start display at page:

Download "USING JAVA HARTSTONE BENCHMARK IN A REAL-TIME SYSTEMS COURSE"

Cecilia Bishop
5 years ago
Views:

1 USING JAVA HARTSTONE BENCHMARK IN A REAL-TIME SYSTEMS COURSE Andy Ju An Wang 1 Abstract This paper shows how a Java benchmark program can be utilized in a real-time systems course to emphasize fundamental concepts and principles in hard real-time systems. Benchmark programs are used to compare the relative speeds of computers and language implementations. The Java Hartstone benchmark was used in our course project to test various Java virtual machines (JVMs) for their suitability of supporting hard real time applications. With the Java Hartstone benchmark, our experiment was carried out using all four benchmark experiments under controlled conditions to provide reliable analysis. The characteristics and expected behavior of the benchmark are described, actual results from the benchmark testing are presented and analyzed, and the lessons learned about using real-time benchmark programs in a real-time systems course are discussed. Index Terms Benchmark testing, Hartstone benchmark, Java virtual machines, real-time systems, real-time performance 1. INTRODUCTION One of the difficulties of teaching a real-time course is to help students distinguishing real-time software from conventional software and understanding issues of timecritical computing in a "real" environment. Textbooks (for instance, [5,6,7]) provide good examples for hard real-time systems like air traffic control systems, process monitor and control systems, missile-guiding systems, etc. None of them, however, can be practiced by students in a classroom setting. Hard real-time systems are those where it is absolutely imperative that responses occur within the specified deadline. Missing a hard deadline can result in catastrophic consequences or loss of system performance. The primary issue in meeting deadlines in a real-time system is that multiple tasks can compete for shared resources such as processors, memory, and I/O devices. Higher processor speed or higher communication bandwidth is important to meet system timing requirements, but speed alone is not enough. A well-designed scheduler is also important to ensure that a hard real-time system meets its performance requirements. Our experience shows that understanding basic concepts of task scheduling helps students to capture major characteristics of real-time systems. Instead of asking students to design and implement their own scheduling algorithms, we ask our students to learn a benchmark program for real-time systems, and use the benchmark program to compare different Java virtual machines for their suitability in hard real-time applications. Benchmark programs are used to compare the relative speeds of computers and language implementations. One of the most important benchmarks for real-time systems is the Whetstone benchmark [1]. It is a synthetic benchmark, which means that rather than performing any useful function, it is constructed based on the instruction frequencies of a representative set of programs. The Hartstone benchmark originally began as a specification at the Software Engineering Institute at Carnegie Mellon University in 1989, which specified five different series of experiments that would analyze an Ada runtime system for its suitability in a hard real-time environment. The name Hartstone was derived from Hard Real Time and the fact that the workloads are based on the well-known Whetstone benchmark. After about one year, an Ada implementation for the first series of experiments described in the specification was released, as well as results for a number of Ada runtime systems. Since then, Hartstone has been an influential benchmark in evaluating Ada systems [1, 2, 3, 4]. Java Hartstone is a Java implementation based on the Ada version Hartstone to test the suitability of Java virtual machines for hard real-time applications. It consists of four experiments, creating five separate threads, each of which will run at a specific frequency, and execute a fixed amount of work each time it is awakened. The work it executes is an altered Whetstone benchmark, designed to execute 1 Whetstone instructions, and it executes that workload a given number of times. The baseline test involves five tasks given the following frequencies and workloads in Table 1: Task TABLE I FIVE TASKS IN THE BASELINE TEST Frequency (Hz) Workload (Work Units) Work Units/Second 1 Andy Ju An Wang, School of Computing and Software Engineering, Southern Polytechnic State University, 11 S Marietta Parkway, Marietta, GA 36 jwang@spsu.edu S2F-25

2 When running each of the four experiments, a series of tests start, each running 1 seconds, until a failure occurs that represents one missed deadline or a number or percentage of missed deadlines. The raw speed of the system is calculated by running the workload in a single thread. The system s utilization is calculated by dividing the number of work units performed in the last test by the raw speed. This percentage indicates how much of the computational power of the system can be used and still be able to meet its deadlines. For each successive test, the experiments alter the parameters in the following way: Experiment 1: The highest frequency task (#5) has its frequency increased by the frequency of the next highest (#4). This experiment tests if the system can switch between threads quickly and if it can handle fine-grained scheduling. Experiment 2: The frequencies of all the tasks are scaled by 1% (so the second test is 11%, third is 12%, 13%, and so on). This tests the performance of the system on an increasing workload, but with task frequencies increasing evenly. Experiment 3: The workload each task performs at each activation is increased by 1. This experiment increases the workload gradually, but has no effect on switching overhead because task frequencies are not changed. Experiment 4: A new task is created with frequency and workload equal to that of the middle task (#3), so the second test has 6 tasks running, the third has 7, and so on. This determines the number of tasks a system can handle and still meet real-time deadlines. The author used Java Hartstone benchmark successfully in our CS6283 Real-Time System course. For the purpose of this course, four common Java virtual machines were selected as the target systems as demonstrated in Table II: TABLE II FOUR JAVA VIRTUAL MACHINES UNDER TEST Name Version Vendor Justification java Sun Microsystems, Inc. Original Java reference implementation, widely used java 1.3 IBM Strong supporter for Java, big player in computer industry pvm NewMonics, Inc. jrockit Appeal Virtual Machines AB With real-time capabilities for embedded systems Highly scalable, combining OO optimization techniques Our selection of JVMs for testing was largely based on their appealing features claimed by the vendors. For instance, the PERC JVM (pvm) can be tuned to prefer runtime performance, startup time or memory footprint. It is available for a variety of operating systems and processors. JRockit by Appeal is a highly scalable, adaptive, optimized JVM with full JDK1.3 support. It combines object-oriented optimization techniques with a highly scalable runtime architecture. As a result, JRockit allows mo re threads, faster I/O and better memory management than any other JVMs running on the same hardware. It is interesting to test their capability for supporting hard real-time applications with a standard benchmark program. The next section introduces the methods and procedures of running Java Hartstone benchmark. Section 3 provides some of our experiment results. Section 4 discusses the interpretations of the results. Conclusions and discussions are summarized in Section METHODS AND PROCEDURES The PERC3.1.1 product comes with a source version of the Java Hartstone benchmark. We used this version to build and run the Hartstone tests. For the convenience of writing, we describe only here the methods and procedures on a Windows platform. Similar methods can be applied on other computing platforms to run Java Hartstone benchmark. The first step is to start up a command window to compile the Java Hartstone source files using an appropriate compiler. The experiment presented here used the IBM compiler with the following command: jikes benchmarks/hartstone/*.java When running an experiment, it is necessary to run a baseline test to determine what frequency and workload settings to use. The command we need to run Hartstone is as below: -e <experiment #> -t 1 where <VM> is the name of the Java virtual machine to use, and experiment# is the experiment number (1-4). For instance, with the command java benchmarks.hartstone.hartstone e 1 t 1, the output will be the following: Sleep granularity: 1 ms currenttimemillis() granularity: 1 ms Experiment: Increasing Frequency Experiment (PH1) Raw speed in Work Units Per Second (WUPS): Test 1 characteristics: Task # Freq. (Hz) WU/Period WU/Second Workload util S2F-26

3 % 3. INTERPRETING THE RESULTS Step size: % Test 1 results: Test duration (seconds): 1. Task # Met Missed Skipped Average Late (msec) Most Late When it prints its results, we need to look at the total utilization that the test reached, which should be around 2%. If not, we need to scale the baseline task frequencies or workloads to bring the baseline utilization to about 2%. Otherwise, the experiment will probably take longer time to complete, though the final result should not change. For instance, the above output shows that only about 2% of utilization was reached. Usually it is recommended to scale the workload before scaling the frequency, because all Java VMs are limited to a 1-ms accurate timer, so scaling the frequency too high in the baseline test could cause premature failure due to timer problems. Use -w <n> to scale the workload of each task by a factor of n, and -f <n> to scale the frequencies by a factor of n. Once we figured out suitable frequency and workload scales (f and w), we can run Hartstone as follows: -e <experiment#> -full -f <f> -w <w> As the experiment progresses, results will be printed for the baseline test, the last test that ran with no missed deadlines, the first test with failed deadlines, and the last test ran. Once complete, take note of the total utilization percentage of the last successful test, which is the VM s score for that experiment. That is the utilization at which the VM can run at and still meet hard real-time requirements. Additional options for the command line can be found in [4]. With the machine configuration listed in Appendix A at the end of this paper, we used the following commands to complete the experiment: -e 1 -full -f 2 -w 5 -e 2 -full -f 2 -w 5 -e 3 -full -f 1 -w 1 -e 4 -full -f 1 -w 1 These commands were first applied to Sun JVM and then repeated for PERC JVM, JRockit JVM, and IBM JVM. The first important value output by Java Hartstone benchmark is the raw speed of the system. The raw speed shows the total number of workloads per second the system can handle running the workloads on a single thread. The raw speed represents full utilization of the system. The workload utilization is used to determine the percentage of utilization of the system by dividing the workloads completed by the last test to succeed by the raw speed in workload units per second. The frequency is the number of times per second the workload units will be performed. The most important result for each experiment is the utilization percentage from the last test in the experiment to succeed. This value can be considered a score for that experiment as well as a measurement of how well the VM is able to utilize its full capability for real-time systems. Also important is the work units per period completed by the last successful test. This value is a sum of the total work units that the VM was able to successfully complete for an experiment. The work units per period summed for the last test makes a useful value to measure the performance of the Java virtual machine on an identical machine configuration. Each experiment increases the workload in different ways to test different capabilities of the JVM. Each task set consists of five tasks. Experiment 1 tries to test the system s ability to schedule an increasing number of tasks per second as it increases the frequency of task 5 each test run while leaving the other tasks alone. This means that task 5 will try to schedule more per second. This tests the VM s ability to schedule tasks as an ever decreasing granularity and to switch rapidly between processes as it tries to complete the workload for all the running threads. This experiment s results are related to how much system overhead the VM requires. Experiment 2 increases the frequency of each task in the test set by 1% for each new test until the deadline is missed. Each task does the same workload per period but by increasing the frequency the total number of periods increases, increasing the total workload for each task. This tests the systems ability to handle more work at higher frequency rates. As in experiment 1, with experiment 2 increases the frequency, overhead of scheduling threads is increased. Experiment 3 increases the workloads of each task by 1 work unit each new test and does not change the frequency for the tasks. Unlike the previous experiments, this test does not increase the system overhead of scheduling the workloads, it just schedules more work. This tests the systems ability to handle an increasing workload but with no increasing scheduling overhead. Experiment 4 actually adds new tasks each time it runs. Unlike the other tests, the workload and frequency stays the same while the number of tasks increases each time. This experiment will show how well the system can handle a large number of tasks as the previous experiments only ran five tasks each time. S2F-27

4 4. EXPERIMENTING RESULTS Scaled Frequency Experiment 2 As demonstrated in Table III below, the Increasing Frequency Experiment (experiment 1) as illustrated in Figure 1 showed that the PERC VM had the highest percentage of utilization. But when we inspect the numbers further it is obvious that the IBM VM had the best performance with a raw speed of 5769 and 19.44% utilization for 112 work units per second (WUPS) TABLE III TESTING RESULT IN EXPERIMENT 1 Sun VM JRockit VM Perc VM IBM VM Increasing Frequency Experiment 1 1 FIGURE. 2 THE RESULT FOR EXPERIMENT 2 On the Increasing Workload Experiment (Experiment 3), the IBM VM had the highest percent utilization as well as the highest WUPS, as illustrated in Table V and Figure 3 below. TABLE V TESTING RESULT IN EXPERIMENT 3 Sun VM JRockit VM Perc VM IBM VM Increasing Workload Experiment FIGURE. 1 THE RESULT FOR EXPERIMENT Table IV and Figure 2 below summarize the testing result for the Experiment 2. The Scaled Frequency Experiment also showed the PERC VM with the highest percent utilization but once again the IBM VM was the top performer with 42.44% utilization of over 584,38 raw speed WUPS given it over 248, WUPS utilization. TABLE IV TESTING RESULT IN EXPERIMENT 2 Sun VM JRockit VM Perc VM IBM VM FIGURE. 3 THE RESULT FOR EXPERIMENT 3 The Increasing Tasks Experiment (Experiment 4, Table VI, and Figure 4) showed the PERC VM with the highest percent utilization but once again the IBM VM outperformed it with 368, WUPS utilized out of 584,38.13 raw speed WUPS. S2F-28

5 TABLE VI TESTING RESULT IN EXPERIMENT 4 Sun VM JRockit VM Perc VM IBM VM Increasing Tasks Experiment 4 FIGURE. 4 THE RESULT FOR EXPERIMENT 4 Figure 5 compares the four JVMs in terms of their workload utilizations. If we just compared the percentage of workload utilization, the PERC VM was the winner in all categories by the Increasing Workload Experiment. But the IBM VM was hands down the highest performing VM with the Sun VM closely behind it. The JRockit VM showed good raw speed performance but the workload utilization percentages were the lowest in every experiment. One thing that stood out from the benchmark output for the JRockit was the Sleep Granularity value. JRockit had a 53 ms sleep granularity while the other VM s had only a 1 ms sleep granularity. This likely explains JRockit low performance with thread scheduling. 12 Virtual Machine Utilization 5. CONCLUSIONS AND DISCUSSION The Java Hartstone benchmark provides a useful way to test a Java VM for real-time system handling performance. The IBM VM was clearly the workhorse of all the VM s with higher work units per second than any of the others and higher work units per second utilization. Even at a low percent utilization, the IBM VM was just able to handle more work and more thread scheduling. But one limiting factor to the whole test was the test machine configuration. Windows 2 is far from a real-time operating system with many other background OS threads running while a real real-time operating system (RTOS) would probably only have the single embedded system process running on the whole OS, thus it would be much better at handling thread scheduling request from just its single running process. The Java Hartstone benchmark might be better run on an embedded system version of Java running on an embedded system to truly test a systems real-time performance. In our real-time systems course, students were divided into several groups to testing different Java virtual machines with the same benchmark program. It was interesting to discover that testing results were slightly different due to the fact that different hardware and software platforms were used by different groups. One potential future work is to find a calculating method to sum up and average the results obtained from different groups so that we could reach a statistical and consistent assessment for selected Java virtual machines. Another issue is to make the experiment reproducible. For instance, we ought to compile the Java Hartstone benchmark program using only one compiler and test all the selected Java virtual machines with the same byte-code. Background running processes and user interference, like moving the mouse around, should be minimized during the testing process. Moreover, we should first scale up workload before scaling frequency due to the timing granularity of Java virtual machines. Finally, it is interesting to test more Java virtual machines to have a complete comparison among JVMs in terms of their suitability for hard real-time applications. 1 8 ACKNOWLEDGMENT % Sun VM JRockit VM Perc VM IBM VM The author would like to express his thanks to all the students in CS6283 Real-Time Systems (fall 21), especially Jim Knupp, Venkatesh Baglodi, Yiming Ji, Xing Dong, Mili Naik, Kanchan Nigam, Abraham Jacob, and Li Yi for their well-done project work and experimental effort. Experiment 1 Experiment 2 Experiment 3 Experiment 4 FIGURE. 5 COMPARE THE UTILIZATION S2F-29

6 REFERENCES [1] Weiderman, N., Hartstone: Synthetic Benchmark Requirements for Hard Real-Time Applications, CMU/SEI-89-TR-23, June [2] Donohoe, P., Shapiro, R., and Weiderman, N., Hardstone Benchmark Results and Anaylisis, CMU/SEI-9-TR-7, June 199. [3] Donohoe, P., Shapiro, R., and Weiderman, N., Hardstone Benchmark User s Guide, Version 1., CMU/SEI-9-UG-1, March 199. [4] NewMonics, Inc., PERC3.1.1 User Manual, 21. [5] Laplante P.A., Real-Time Systems Design and Analysis, An Engineer s Handbook, 2 nd ed, IEEE Press, [6] Shaw A.C., Real-Time Systems and Software, John Wiley & Sons, Inc., 21. [7] Burns A. and Wellings A., Real-Time Systems and Programming Languages, Ada 95, Real-Time Java and Real-Time POSIX, 3 rd ed., Addison Wesley, 21. Appendix A Test machine configuration Hardware: IA32 CPU: 1.8 GHz Pentium 4 FPU: Integrated Number of CPUs: 1 Cache size per CPU: 15MB Memory: 1 Gigabyte SDRAM PC133 Disk subsystem: 1 2 Gigabyte EIDE Network interface: NA Software Operating System: Java VM s: Sun JDK IBM 1.3 Perc PVM JRockit 1.3 Java VM s release version Sun 1.3.1_1 IBM 1.3. Perc PVM JRockit Other software: File system type: System Tuning parameters : Background load: System state: Windows 2 Professional Service Pack 2 none NTFS none none Single user (single-user login) S2F-3

Method-Level Phase Behavior in Java Workloads

Method-Level Phase Behavior in Java Workloads Andy Georges, Dries Buytaert, Lieven Eeckhout and Koen De Bosschere Ghent University Presented by Bruno Dufour dufour@cs.rutgers.edu Rutgers University DCS