Runtime Performance Evaluation of Just-In- Time Compiler Enabled J9 Virtual Machine

Transcription

1 Runtime Performance Evaluation of Just-In- Time Compiler Enabled J9 Virtual Machine Yi-Huey Li Arizona State University East Abstract This project evaluates the runtime performance of J9 Virtual Machine (VM) on a TOSHBA Pocket PC. The publicly available benchmark TaylorBench and four other small test progra written by the author were tested on the device. The VM was tested either with or without Just-In-Time compiler (JIT). The JIT options for optimization/non-optimization and when to start JIT were tested also. This project tried to find the reasons why benchmarks may not realize the full performance that can be achieved from the JIT enabled J9 VM and to characterize applications where the JIT will improve performance. I. INTRODUCTION 1. Background The Java programming language is famous of its portability. In order to achieve the Write Once, Run Anywhere philosophy, the Java source are converted into an architectureneutral distribution format, called Java byte code. The byte code sequences are then interpreted by a Java virtual machine (JVM) for each platform. Although its platform-neutrality, flexibility, and reusability are all advantages for a programming language, the execution by interpretation results in a poor runtime performance. The runtime overhead of the byte code instruction fetch and decode is the main reason for the slow down [1], [2]. To improve the Java runtime performance, many techniques have been employed. The Just-In-Time (JIT) compilation is one of them. The JIT compilers promise to improve the runtime performance and still preserve the important portability property of the Java applications. The JIT compiler converts the given byte code sequences on the fly into an equivalent sequence of the native code of the underlying machine [1]. Because the execution of native code is much faster than the execution of byte code, and the compiled code can be cached in the memory for later use, the run time performance can therefore be improved. However, there is one drawback of this approach, the time spent compiling a method is time that could have been spent interpreting and executing its byte code. To gain the overall speed, the JIT compiler needs to recoup the time spent on optimizing and compiling by the subsequent executions of the fast machine code form. Since the total execution time of a Java program is a combination of compilation time and execution time, a JIT compiler needs to balance the time spent optimizing generated code against the time it saves by the optimization [3]. A fast un-optimizing compiler could minimize start-up delays. A slow, optimizing compiler, on the other hand will benefit from faster execution. Moreover, the performance of JIT depends not only on the speed of compilation and the level of optimization but also the type of the Java applications. A program without large loops or recursive methods for example, may not benefit from a slow extensive optimizing JIT compiler. Appropriate benchmarks can help the developers to evaluate and compare the performance of various Java execution techniques. Inappropriate benchmarks on the other hand can be misleading. The author of [4] used the example of HotSpot TM to demonstrate that the way benchmarks are written could really affect the result of benchmarking. With inappropriate benchmark, the HotSpot TM could be 35 times slower than previous JVMs using JIT. After modifying the benchmark to take advantage of HotSpot TM, the program executes better than twice as fast as the previous best run. It showed the importance of choosing benchmarks to evaluate the performance of JIT compilers. 2. Problem Statement With the importance of benchmark in mind, there are however some proble in the benchmarking world. For example, there see to be no benchmark specifically designed for testing the relative quality of JIT. Moreover, most of the benchmarks have no source available. It makes it difficult to evaluate and compare the performance of JIT compilers. Agarawal [5] compared the performance of J9 virtual machine (VM) with/without JIT on Pocket PC using some publicly available benchmarks. In her paper, she found no big difference between JIT and no JIT VMs. It see that using an appropriate benchmark or a specific type of application is the key to take the full advantage of the JIT compiler. Therefore, this project tried to find the reasons why benchmarks may not realize the full performance that can be achieved from the JIT enabled J9 VM and to characterize applications where the JIT will improve performance. 1

2 3. Significance By comparing the performance of J9 with different JIT options, this project intends to help the developers and programmers realize the advantage and limitation of the J9 and its JIT. With the result in mind, the developers can know how to write their applications to take the full advantage of the JIT compiler, and know how to best use JIT. II. METHODOLOGY 1. Devices and Platform All of the tests were done on a TOSHIBA Pocket PC e355. The operation system for this device is Microsoft Pocket PC Version The processor is Intel PXA MHz with MB RAM and 16 MB ROM. WebSphere Studio Device Developer (WSDD) Evaluation version 5.5 [6] running on Windows XP was used to develop the Homemade Test Progra and to launch the progra. 2. VM The IBM J9 is the VM to be tested in this study. It is a highperformance environment with JIT technology available for the deployment of J2ME applications and supports multiple platfor. Together with WSDD Micro Environment, J9 allows developers to quickly and easily create and deploy applications to hand-held computers, PDA's and cellular phones [6]. Although J9 supports both CLDC and CDC and various profiles for Pocket PC, this study only focuses on CLDC 1. and MIDP JIT Options Table 1 shows the seven combinations of JIT options as command line arguments selected for this study and the abbreviations used in this report. TABLE I JIT OPTIONS AND THEIR ABBRIEVIATIONS FOR THIS STUDY Arguments for JIT Abbreviation Definition used in this report - Interpret all methods. -jit:count=, cbcnoopt JIT-compile immediately bcount=, noopt before the first invocation of the method with un-optimized code. -jit:count=, bcount=, bestavailopt -jit:count=1, bcount=1, noopt -jit:count=1, bcount=1, bestavailopt cbcopt JIT-compile immediately before the first invocation of the method with best possible code. c1bc1noopt JIT compile methods without loops after 1 invocations. JIT compile methods with loops after 1 invocation. Generate unoptimized code. c1bc1opt JIT compile methods without loops after 1 invocations. JIT compile methods with loops after 1 -jit:count=1, bcount=2, noopt -jit:count=1, bcount=2, bestavailopt invocation. Generate best possible code. c1bc2noopt JIT compile methods without loops after 1 invocations. JIT compile methods with loops after 2 invocations. Generate unoptimized code. c1bc2opt JIT compile methods without loops after 1 invocations. JIT compile methods with loops after 2 invocations. Generate best possible code. 4. Benchmarks 4.1 TaylorBench 1.1 A very simple benchmark [7], tests the low-level performance of a MIDP-compliant device. It consists of four categories: a. Low-level graphics - tests the performance of lines, rectangles, ellipses, arcs, images, and text rendering b. RMS - tests record creation, reading (enumerated and random), and deletion c. CPU /VM - general CPU test, general Virtual Machine test, and a simple check to make sure random numbers really are randomly distributed d. Com - tests the performance when uploading and downloading data to a remote HTTP server. Also allows testing of file loads from the MIDlet suite. The result represents the time in milliseconds required to complete each test. The test to the remote HTTP server was not performed because the networking device was not available. 4.2 Author-Defined Test Progra A set of small progra created by the author to test the low-level performance of a MIDP-compliant device. It consists of four tests: a. Ellipses tests the performance of low-level graphics by drawing filled ellipses repeatedly on a canvas. The source code that draws filled ellipses at random location was extracted from the class ArcCanvas.java of TaylorBench [7]. b. Text tests the performance of text rendering by drawing Hello Word repeatedly on a canvas. c. QuickSortInt use Quick Sort algorithm to sort an array of integers. The code in QuickSort.java was copied from [8] to provide the methods to sort array of integers. A snippet of the methods that are responsible for sorting is shown below. public void sort(int low, int high) { if (low >= high) return; int p = partition(low, high); sort(low, p); sort(p + 1, high); } private int partition(int low, int high) { // First element 2

3 } int pivot = a[low]; // Middle element int i = low - 1; int j = high + 1; while (i < j) { i++; while (a[i] < pivot) i++; j--; while (a[j] > pivot) j--; if (i < j) swap(i, j); } return j; /** Swaps two entries of the i the first position to j the second position to swap */ private void swap(int i, int j) { int temp = a[i]; a[i] = a[j]; a[j] = temp; } } private int[] a; d. QuickSortString use Quick Sort algorithm to sort an array of String objects. The sort.java class was downloaded from the web site [9] to provide the methods to sort array of String objects. The class Utils.java of TaylorBench was imported into each of the MIDlet suites to provide the methods for timing the test. Each test was written as an independent MIDlet suite to reduce the possible interference among different tests. The result represents the time in milliseconds required to complete each test. 5. Procedure 5.1 TaylorBench 1.1 The steps to create J9 runtime environment and application shortcuts on Pocket PC are described below. 1) Place the device in the cradle for ActiveSync. Use Explore to navigate through the file system on the device. Copy the ive folder from C:\Program Files\IBM\Device Developer\wsdd5.\ive\runtimes\pocketpc\arm\ to the directory under My Pocket PC on the device. This creates J9 runtime environment on Pocket PC. 2) Copy classes.zip from C:\Program Files\IBM\Device Developer\wsdd5.\ive\runtimes\common\ive\lib\jclMidp to \ive\lib\jclmidp directory on the device. These are the class files needed to run the application. 3) Copy taylorbench.jar and taylorbench.jad to \Program Files\ on the device. 4) Create a shortcut for the application to run on the device. Open a text editor and write the following command to the file and save it as taylor_njit.lnk. 1#"\ive\bin\J9.exe" "-" "- Xbootclasspath:\ive\lib\jclMidp.jxe" "-classpath" "\Program Files\TaylorBench.jar;\Program Files;\ive\lib\classes.zip" "- jcl:midp:loadlibrary=swt-win32-214" "javax.microedition.midlet.appmanager" "TaylorBench.jad" 5) Copy taylor_njit.lnk to \Windows\Start Menu\ on the device. 6) An icon with the name taylor_njit should appear on the Progra window and ready to be executed. 7) Create a shortcut for each JIT option by replacing the argument - with different JIT argument listed above like -jit:count=,bcount=,noopt. When testing, the program was restarted for each specific test to avoid possible interference. The steps to run the tests and to capture performance information are described below. 1) Make sure there are no other progra running by selecting Settings from Start menu. Select System tab, then pick Memory icon. In the Running Progra page, a list of running progra will be displayed. Select Stop All to stop all of the running progra. 2) Select Progra from Start menu. Pick the shortcut for the specific benchmark and JIT options. This will start the TaylorBench with the specified JIT options. 3) Select a category from the category list. Specify number of repeats for the tests within the category by selecting Options. 4) Select and start a test. The result will be returned as milliseconds once the test is done. Record the result as. Repeat the test by selecting the same test and record the result for Test2,, and so on. 5) Exit from the benchmark and repeat from step 1) for other tests. 5.2 Author-Defined Test Progra These progra were developed and launched with WebSphere Studio Device Developer Evaluation version 5.5. To create a MIDlet suit for Pocket PC, follow the following steps. 1) Click the Open the New Wizard button. 2) In the left pane, select J2ME. In the right pane, select Create MIDlet Suite. Then click Next. 3) In the MIDlet Suite Creation wizard, fill in the following fields: Project name MIDlet suite name MIDlet name MIDlet class name 4) Click Finish. A template MIDlet class is created for you. 5) Right click on default package under src of the project, select New Class to add a class to the project if needed. 6) Click File Import to import existing classes to the project if needed. 7) With Java Perspective, double click wsddbuild.xml. On the right panel, click Add Build. Pick J9 for PocketPC ARM as platform and click Next. Click Finish on the following 3

4 page. 8) To launch the program to the device, follow the steps below. 9) Put the device on the ActiveSync cradle. Create the device configuration by clicking Devices icon from the WSDD workbench. Select PocketPC Handheld and press New. 1) Specify J9 runtime location, application install location and shortcut install location and press Apply. 11) Click Run icon from the WSDD workbench and select Run to create and run launch configurations. 12) Pick the Pocket PC configuration as the device. Press Apply to save the launch configuration and then press Run to launch the program on the device. 13) Copy the shortcut created by WSDD from the device to a laptop PC. Open it with an editor. Create shortcuts for each JIT option by adding different JIT arguments into the shortcut file the same way was done for TaylorBench. 14) Run the tests by clicking different shortcuts. Before each test, all of the progra running on the device were stopped to ensure the fairness for each test. 5.3 Data Collection and Analysis Data from TaylorBench and Author-Defined Test Progra were separated into two different folders. Inside each folder, a MicroSoft Works spreadsheet was created and maintained for each VM JIT option. For example, the file name cbcnoopt is for the data from testing the VM with JIT option cbcnoopt. The result was recorded into the spreadsheet manually. After all of the data were gathered, they were then copied into a MicroSost Excel spreadsheet. The various comparisons and analysis were made with raw data collected. III. RESULTS The results represent the time required to run each test in milliseconds (). Lower score is better. The, Test2. in the X axis represent the first, second,. time the same test was run. So for example, when the program was started, and the specific test was picked and set to run, then the result from the first time running the test was recorded as. Without exiting the program, the same test can be set to run again. The result from the second time running the test was recorded as Test2. The same test was set to run over and over again and the result was recorded until the desired number of tests was reached. Besides the options of optimization or non-optimization, the count and bcount were set to various values to control when the JIT will be started. Immediate JIT with optimization (count=,bcount=,bestavailopt) will cause the VM to optimize and compile every method it has encountered. Because optimization is pretty expensive, it will make the program extremely slow. Since the benchmark is just timing a small block of code, the time user spend on waiting for the program to start, for the user interface to take user input or to bring up the output is not included as result. The setting cbcnoopt runs a lot faster than cbcopt, because it uses faster JIT, that is non-optimized JIT. But it is still slow. So it is not recommended to set the count to and bcount to for either optimized or non-optimized JIT. However, if you can mark the specific methods to be compiled by using xjit: verbose command, then it will not be that a big problem. Doing JIT immediately in this study is a better way to evaluate the time spent on optimization and compilation. The time spent on the first invocation of the test will include the time spent on optimization and/or compilation. After the first invocation of the test, the subsequent tests will be running with compiled code. Therefore, by comparing the result of the first and the second test, you can estimate the time required for optimization and/or compilation. 1. Results From TaylorBench 1.1 Potential proble of TaylorBench The TaylorBench allows user to pick from a list of four categories. Within that category user can then pick from a list of tests. A specific test can be run repeatedly by clicking the same test over and over again without exiting the program. Different tests can also be run consecutively without leaving the program. It is convenient for the user to run the test, but it posts a potential problem when a VM with JIT is tested. The problem comes from when two or more tests call to the same method. Take Ellipses and Arcs in Low-level Graphics as an example. They both call the method g.fillarc() with different arguments. If Ellipses is run first, then when Arcs is tested later, the method g.fillarc() probably has been compiled if JIT is on. Then the result will not be the same as if Arcs is tested first before Ellipses. The same kind of interference could happen on both API classes and user-defined classes. Because they are all byte code, they are treated by JIT no differently. To avoid this kind of interference among different tests, the benchmark was restarted for each test. 1.2 Runtime performance Among four different categories of TaylorBench, only the tests on low-level graphics show differences between JIT and no JIT options Low-level graphics This category includes the following tests: Lines, Rectangles, Ellipses, Arcs, Small Fonts and Images. They were tested with 2 repeats and 2 repeats. Each test was run 12 times consecutively With 2 Repeats With 2 repeats, the results for Lines, Rectangles, and Images all look similar. Figure 1 shows the result of Lines with 2 repeats. At, both VMs with optimized JIT performed the worst. The VMs with JIT but without optimization performed better than the optimized ones, but still worse than the no JIT one. After paying the start up price, all VMs performed with indistinguishable performance. 4

5 TaylorBench Lines 2 Repeats TaylorBench Small Fonts 2 Repeats Test2 cbcnoopt cbcopt c1bc1noopt c1bc1opt Test6 Test8 1 2 Fig. 1. Result for TaylorBench benchmark. Test on low-level graphics. The time to draw 2 lines in milliseconds was measured. Lower is better. Figure 2 shows the result for Small Fonts in the same category as Figure 1. The VM with option c1bc1opt has extremely high start up cost (341 ). The VM with option c1bc1noopt has much less cost (49 ). The two VMs with option cbc performed similar to the no JIT one (8 ). After, all VMs have indistinguishable performance (8 ). The difference between c1bc1opt and cbcopt is just when to start JIT. When we look more closely inside the code, we can find the reason why a huge difference arises between these two options in the result. Inside the block of code that is timed in this test, there are a few lines of code written as below. g.setgrayscale(random[j++] & xff);.. g.setcolor(random[j++]); The calls to these two methods were also found in the same class just before the block to be timed. Therefore, for cbcopt, it should JIT these two methods the first time it hits them. Then when it gets to the block, and hits the same methods again, they will be executed as compiled code. So the time needed to compile these two methods are timed in of cbcopt. For c1bc1opt, it is a different situation. Since with this option, JIT is not starting before a method is hit 1 times. Therefore the time spent on compiling these methods is timed in of c1bc1opt cbcnoopt cbcopt c1bc1noopt c1bc1opt Test2 Test6 Test8 1 2 Fig. 2. Result for TaylorBench benchmark. Test on low-level graphics. Execution time to draw 2 small fonts in milliseconds was measured. Lower is better. The result for Ellipses is similar to the result of Arcs. Figure 3 shows the result for Ellipses. Both optimized VMs have really high start up cost (27 ) but the execution time was reduced dramatically to around 3 after the first execution. The two VMs with non-optimized JIT start with the similar performance as no JIT one (9 ). The execution time was further reduced to the level of optimized ones (3 ). The no JIT VM, however, stays at around TaylorBench Ellipses 2 Repeats cbcnoopt cbcopt c1bc1noopt c1bc1opt Test2 Test6 Test8 1 2 Fig. 3. Result for TaylorBench benchmark. Test on low-level graphics. Execution time to draw 2 ellipses was measured in milliseconds. Lower is better. 5

6 With 2 Repeats The results for Lines, Rectangles, Images and Small Fonts are similar. The start-up costs for JIT with/without optimization are not as prominent as that for 2 repeats. Although there is no big improvement for optimized JIT over non-optimized JIT, the difference between JIT and no JIT starts to show. Figure 4 shows the result for Small Fonts. The optimized JIT starts out as 1129 and then reduced to around 77. Non-optimized JIT starts out as 868 and then reduced to around 8. The VM without JIT stays at about TaylorBench Ellipses 2 Repeats c1bc1noopt c1bc1opt TaylorBench Small Fonts 2 Repeats c1bc1noopt c1bc1opt 1 Fig. 5. Result for TaylorBench benchmark. Test on low-level graphics. Execution time to draw 2 ellipses was measured in milliseconds. Lower is better RMS The Creation, reading (enumerated and random), and deletion of 1 or 1 of 1 byte records were tested. Start-up cost was observed for all tests with both 1 and 1 records. JIT compilations with or without optimization both did not show to reduce the execution time. Figure 6 shows the result for the creation of 1 records. Optimized JIT has higher startup cost (367 ) than non-optimized JIT (333 ). After the first test, optimized JIT, non-optimized JIT and no JIT all performed no differently (around 3 ). Fig. 5. Result for TaylorBench benchmark. Test on low-level graphics. Execution time to draw 2 small fonts was measured in milliseconds. Lower is better. The results for Ellipses and Arcs are similar. Figure 6 shows the result for Ellipses. The penalty for compilation is not showing anymore because of the increased repetitions. Both JIT with/without optimizations perform a lot better than no JIT one (less than 3 versus 8 ). The optimized VM and non-optimized VM have the same performance at. But optimized code does run a little better than non-optimized code (27 versus 293 ) TaylorBench RMS 1 of 1 Records Creation c1bc1noopt c1bc1opt Test2 Test6 Fig. 6. Result for TaylorBench benchmark. Test on RMS. Execution time to create 1 of 1 byte records was measured in milliseconds. Lower is better Com Reading from one 983 byte local file was tested. No performance difference can be found between JIT and no JIT VMs. The optimized JIT has high compilation cost. 6

7 1.2.4 CPU/VM byte array copies One hundred or five thousand array copies were tested. Not much difference can be found in either 1 or 5 array copies among different JIT options. Figure 7 shows the result for 5 array copies TaylorBench 5 1-byte Array Copies Test2 Test6 Test8 1 c1bc1noopt c1bc1opt 2 Fig. 7. Result for TaylorBench benchmark. Test on CPU. Execution time to copy 5 of 1 byte arrays was measured in milliseconds. Lower is better VM tests Neither 1 nor 5 VM tests show any difference among different JIT options. 2. Results From Author-Defined Test Progra To avoid the proble that TaylorBench has, four small test progra were written and organized into four different MIDlet suites. Each MIDlet suite was tested independently. 2.1 Ellipses The performance of drawing 5 ellipses was tested. The result is shown on Figure 8. All of the options with JIT performed better than the no JIT one even with the compilation overhead. All three optimized JITs have higher penalty than the non-optimized JITs. It is suggested that the cost for JIT with optimization is higher than JIT without optimization. Among the three optimized JIT options, c1bc2opt has the greatest cost. The c1bc1opt comes next and cbcopt has the least cost among three optimized JIT options. The cost for non-optimized JIT options is similar to the optimized group. The start-up cost from the highest to the lowest is c1bc2noopt, c1bc1noopt, and cbcnoopt. Although the difference is not big, the optimized code does run faster than non-optimized code. The question is. Is it worthy to pay that kind of high price to exchange just a little improvement in execution time? The Really high start-up cost of c1bc2opt is another interesting observation. This higher cost is probably due to the extra time used to interpret byte code before it compiled by JIT compiler. With the option of c1bc2opt, the g.drawfillarc method (which is called to draw the ellipses) has to be called 1 times before it gets to be compiled. Therefore, when running this test, the first 1 method calls to g.drawfillarc was interpreted. It is much slower to interpret code than running compiled code. Then the byte code was optimized and compiled by JIT compiler. After that, the rest of 4 method calls can be run faster with compiled code. Therefore, the time spent in of Ellipses actually includes the time to draw 1 ellipses by interpreting the byte code, to optimize and to compile the byte code, and then to run the fast compiled code for the rest 4 ellipses. Comparing the result to c1bc1opt. The c1bc1opt only interpreted the byte code for 1 times and then switch to run the fast compiled code for the rest of 49 ellipses. Therefore, it has much less start-up cost than c1bc2opt Ellipses cbcnoopt cbcopt c1bc1noopt c1bc1opt c1bc2noopt c1bc2opt Test22 Test25 Test28 Fig. 8. Result for Author-Defined Test Progra. Execution time to draw 5 ellipses to a canvas was measured in milliseconds. Lower is better. 2.2 Text Printing the string Hello World to a canvas 5, 1, 5, and 1 times was tested. The results for 5 and 1 Hello World are similar. Figure 9 shows the data for 1 repeats. Both optimized JIT (cbcopt and c1bc1opt) have extremely high start-up cost. The optimized compiled code does run faster than interpreted byte code, but it is not much faster than the un-optimized compiled code. On the other hand, the non-optimized JIT (cbcnoopt and c1bc1noopt) have 7

8 relatively low start-up cost, and the code they produce does not run much slower than the optimized code HelloWorld 1 times cbcnoopt cbcopt c1bc1noopt c1bc1opt Test22 Test25 Test28 Fig. 9. Result for Author-Defined Test Progra. Execution time to print 1 of string Hello World to a canvas was measured in milliseconds. Lower is better. Comparing different number of repeats for a specific JIT option can provide an interesting view to the data. Figure 1 shows the data for cbcnoopt and Figure 11 shows the data for cbcopt with different number of repeats. Since with cbc, the byte code will be compiled immediately. The time required for the includes the time used for optimization/compilation plus the actual execution time. The difference between the first test and the subsequent test can be viewed as the time required for optimization/compilation. Since it takes the same amount of time to optimize or compile the code regardless of the number of repeats, the difference between and Test2 are all the same for a specific JIT option. The compilation time for non-optimized JIT is about 4. The optimization time plus compilation time for optimized JIT is about 27. So, the difference in compilation overhead between optimization and nonoptimization is about 23. For non-optimized JIT, the price is relatively low. It is reasonable for non-optimized JIT to pay the price even with low number of repeats. For optimized JIT, on the other hand, it may not be so wise to pay the price Hello World cbcnoopt 5 Repeats 1 Repeats 5 Repeats 1 Repeats Test22 Test25 Test28 Fig. 1. Result for Author-Defined Test Progra. Printing string Hello World for 5, 1, 5, 1 time was tested. Cbcnoopt is the JIT option for all of the tests HelloWorld cbcopt Test # times 1 times 5 times 1 times Fig. 11. Result for Author-Defined Test Progra. Printing string Hello World for 5, 1, 5, 1 time was tested. Cbcopt is the JIT option for all of the tests. 2.3 QuickSortInt Figure 12, 13, and 14 show the results of sorting an array with 1, 2 or 4 integers, respectively. All three figures show that optimized JIT has higher start-up cost than non-optimized JIT. The start-up time does not increase a lot when the number of integers move from 1, 2, to 4. For cbcopt, the start-up time is 116, 12, and 124 milliseconds for 1, 2, 4 integers respectively. And for cbcnoopt, it is 24, 29, and 39 milliseconds. On the other hand, execution time for no JIT increases a lot from 2, 42, to 9 milliseconds. So, with

9 integers, non-optimized JIT needs a little more time (4 milliseconds more) than no JIT one at. By increasing the number of integers to 2, non-optimized JIT already took less time (13 milliseconds less) than no JIT At. With 4 integers, the savings are even greater. Compare cbcopt to with 1 integers. It spent 96 milliseconds more than at to produce the code that later run 18 milliseconds less than. It does not seem like a good deal. However, as the number of integers increases, the impact of cost decreases and the profit increases. With 4 integers, optimized JIT costs only 3 milliseconds more than at, but saves about 78 milliseconds for the subsequent tests QuickSortInt 1 Integers Test22 Test25 cbcnoopt cbcopt c1bc1noopt c1bc1opt Test28 Fig. 12. Result for Author-Defined Test Progra. Test QuickSortInt. The time to sorting an array of 1 integers is measured in milliseconds. Lower is better QuickSortInt 2 Integers cbcnoopt cbcopt c1bc1noopt c1bc1opt Fig. 13. Result for Author-Defined Test Progra. Test QuickSortInt. The time to sorting an array of 2 integers is measured in milliseconds. Lower is better QuickSortInt 4 Integers 1 3 cbcnoopt cbcopt c1bc1noopt c1bc1opt 5 Fig. 14. Result for Author-Defined Test Progra. Test QuickSortInt. The time to sorting an array of 4 integers is measured in milliseconds. Lower is better. 2.4 QuickSortString Figure 15, 16, and 17 show the result for sorting an array of 2, 3, or 4 String objects, respectively. The result is similar to the result of QuickSortInt. Optimized JIT has extremely high start-up cost with just a little saving for the subsequent tests. The non-optimized JIT, on the other hand, has relatively low start-up cost, but still has some savings over. As the number of String objects increases, the difference in execution time between non-optimized JIT and, and between two JIT options increases as well. But compare to the huge cost imposed to optimized JIT, the saving over for the subsequent tests does not seem significant. Therefore, for this kind of task, optimized JIT may not be the choice

10 QuickSortString 2 String Objects QuickSortString 4 String Objects cbcnoopt cbcopt c1bc1noopt c1bc1opt cbcnoopt cbcopt c1bc1noopt c1bc1opt Fig. 15. Result for Author-Defined Test Progra. Test QuickSortString. The time to sort an array of 2 String Objects is measured in milliseconds. Lower is better. Fig. 17. Result for Author-Defined Test Progra. Test QuickSortString. The time to sort an array of 4 String Objects is measured in milliseconds. Lower is better QuickSortString 3 String Objects cbcnoopt cbcopt c1bc1noopt c1bc1opt Figure 18a and 18b compares the results of QuickSortInt and QuickSortString. Both tests sort 4 ite, integers or String objects. Figure 18a shows that for optimized JIT, QuickSortString (6 ) cost a lot more than QuickSortInt (117 ) at. Figure 18b shows a closer look of the same data. The difference between QuickSortString and QuickSortInt for is bigger than for optimized JIT and non-optimized JIT. For all of the three JIT options, QuickSortString takes more time to execute than QuickSortInt Fig. 16. Result for Author-Defined Test Progra. Test QuickSortString. The time to sort an array of 3 String Objects is measured in milliseconds. Lower is better. 1

11 QuickSort 4 Ite -int cbcnoopt-int cbcopt-int -String cbcnoopt-string cbcopt-string Fig. 18a. Comparison of QuickSortString to QuickSortInt. Solid squares represent data for QuickSortInt. Open squares represent the data for QuickSortString QuickSort 4 Ite -int cbcnoopt-int cbcopt-int -String cbcnoopt-string cbcopt-string Fig. 18b. Comparison of QuickSortString to QuickSortInt. Solid squares represent data for QuickSortInt. Open squares represent the data for QuickSortString 3. Summary of the results For TaylorBench, among the tests of the Low-level Graphics, only Ellipses and Arcs make big differences between JIT and no JIT even with as small as 2 repeats. The nature of these two tests are drawing curves, whereas the tests for Lines, Rectangles, and Small Fonts are all drawing lines. Therefore, it is reasonable to assume that the JIT works better on the methods that draw curves than those that draw straight lines. Result for Image did not show any difference between JIT and no JIT with 2 repeats. As the number of repeats increases to 2, however, JIT starts to perform a little better than no JIT. In the real world, the game progra will probably draw a lot of images. With the increased repetition, the result may be more likely to favor JIT. The rest of the tests in TaylorBench such as RMS, Com, and VM/CPU did not show any difference between JIT and no JIT. A closer look at the nature of these tasks may help to explain the result. For Home Made Test Progra, the test on ellipses shows that JIT reduced the execution time dramatically. But optimization does not further reduce the execution time much more. For the test on text, JIT reduces the execution time a lot, whereas, Optimization only further reduce it a little. Compare to the high price needed for compiled JIT, non-compiled JIT probably is a better choice. JIT works much better than in QuickSortInt. Optimization does bring down the execution time even more, but compare to the price it has to pay, the saving over nonoptimization does not mean anything. So for this type of task, JIT without optimization may be the choice. QuickSortString has similar result to QuickSortInt, but it takes more time to sort String objects than integers. The price for optimized JIT looks even more dramatically. More String objects may need to be added to show greater difference. IV. CONCLUSION AND FUTURE WORK Benchmarking is a useful tool to help people validate and evaluate performance. But misusing benchmarks could produce misleading results. The interference among different tests of TaylorBench would cause a big problem for JIT if not taken care of properly. JIT can reduce execution time with acceptable compilation overhead. In contrast, the result of optimization may not be so useful. In this study, the optimized code runs only a little better than non-optimized code. The overhead of optimization is extensive. Although by increasing the repeats and numbers of iterations, the time spent on optimization and compilation could be recouped. The question is how often does a small personal device perform a task requiring so many iterations? Besides the choices between JIT and no JIT, optimization and non-optimization, when to start JIT has also been explored in this study. The values of count and bcount have a significant impact on runtime performance. The default value 11

12 for count is 1 and for bcount is 1. The programmers and developers should look at their code closely to make a wise decision. This study only used limited varieties of tasks with limited iterations. Testing with wider varieties of tasks such as math with intensive calculation or indexing through an array may provide important information. The benchmarks used in this study are simple progra. They measure performance on some small blocks of code. More complex, close to real world progra may be used in the future to reflect relevant performance. There are some issues to be considered for optimization and compilation, such as garbage collection, class loading, and method inlining. These factors need to be carefully studied and addressed for a better understanding of performance. Compilation speed is also important for the runtime performance. A fast compilation can minimize start-up delays. If the compiler is slow, compilation will dominate the starting phase of the program, and the user must wait [2]. Memory use is another factor to evaluate the JIT compiler. The JIT compiler not only uses the memory itself, it also uses the memory to save the compiled code. On a system with little memory, such as handheld device, the code size could be an important issue. Even if there is enough memory, the extra space may affect the performance because of the increased paging and cache effects. Therefore in future study, the memory usage and compilation speed may need to be considered as well to provide a more complete picture for the Java Virtual Machine. REFERENCES [1] T. Suganuma, T. Ogasawara, M. Takeuchi, T. Yasue, M. Kawahito, K. Ishizaki, H. Komatsu, and T. Nakatani,, Overview of the IBM Java Just-in-Time Compiler, IBM Syste Journal, 2, Vol. 39, No. 1, Java Performance. [2] T. Cramer, R, Friedman, T. Miller, D. Seberger, R. Wilson, M. Wolczko, Compiling Java Just In Time, IEEE Micro, 1997, May/June, pp [3] I. H. Kazi, H. H. Chen, B. Stanly, D. J. Lilja, Technique for Obtaining High Performance in Java Progra, ACM Computing Surveys, 2, Vol. 32, No. 3, September, pp [4] T. Spencer, Benchmarking on HotSpot. Available: _IDX/ html [5] S.Agarawal, Performance Analysis and Benchmarking of Embedded Java Virtual Machine, Master of Technology Final Project Report, Arizona State University, 23, December. [6] WebSphere Studio Device Developer (WSDD) Evaluation version 5.5. Available: [7] TaylorBench. Available: [8] C. Horstmann, QuickSort.java. Source available: csc31/source/quicksort.java.html [9] Sort.java. Available: software/net/dist/sort.java 12