Salable Parametr Runtme Montorng Dongyun Jn Patrk O Nel Meredth Grgore Roşu Department of Computer Sene Unversty of Illnos at Urbana Champagn Urbana, IL, U.S.A. {djn3, pmeredt, grosu}@s.llnos.edu Abstrat Runtme montorng s an effetve means to mprove the relablty of systems. In reent years, parametr montorng, whh s hghly sutable for objet-orented systems, has ganed sgnfant traton. Prevous work on the performane of parametr runtme montorng has foused on the performane of montorng only one spefaton at a tme. A realst system, however, has numerous propertes that need to be montored smultaneously. Ths paper ntrodues salable tehnques to mprove the performane of one of the fastest parametr montorng systems, JavaMOP, n the presene of multple smultaneous propertes, resultng n average runtme overheads that are less than the summaton of the overheads of the propertes run n solaton. An extensve evaluaton shows that these tehnques, whh were derved followng a thorough nvestgaton and analyss of the urrent bottleneks n JavaMOP, mprove ts runtme performane n the presene of multple propertes by up to two tmes and the memory usage by 34%. Categores and Subjet Desrptors D.2.1 [Software Engneerng]: Requrements/Spefatons; D.2.4 [Software Engneerng]: Software/Program Verfaton; D.2.5 [Software Engneerng]: Testng/Debuggng Languages, Performane, Relablty, Ver- General Terms faton Keywords salablty, parametr montorng, runtme verfaton, runtme montorng, testng, debuggng, aspetorented programmng 1. Introduton Runtme montorng s an effetve tehnque to nrease software relablty, by enablng more effetve testng, debuggng, and reovery from norret program behavor. Parametr propertes are propertes that desrbe behavors of objets (parameters), whh a program should onform durng ts exeuton. They an desrbe use protools for lasses, pre-ondtons for usng lasses, prohbted atvtes, and so fourth. Typestates [30] are a smlar onept, but only allow one sngle parameter. Parametr propertes n general an desrbe propertes about any number of parameters. For example, Fgure 1 shows the Map UnsafeIterator spefaton from [26], whh formalzes the parametr property onernng Map, Colleton, and Iterator parameters, that a map should not be updated whle usng the terator nterfae to terate over ts keys or values. Ths spefaton defnes fve parametr events wth the orrespondng AspetJ pontuts. The property s formalzed usng an extended regular expresson (ERE), as spefed by the ere keyword. If a program behavor mathes ths pattern, volatng the property from the Java API doumentaton, the defned handler ontanng the user-defned Java ode wll be exeuted; here we smply prnt out an error message n the handler. Handler an be any ode, from loggng to reovery. Many runtme propertes an be enfored wth parametr montorng. Parametr spefatons are espeally effetve for formalzng propertes that arse n objet-orented programmng. Several parametr montorng systems suh as Hawk/Eagle [16], J-Lo [8, 9, 29], JavaMaC [25], Java- MOP [13 15], JPaX [23], Pal [12], PoET [19], PQL [27], PTQL [21], RuleR [6], QVM [4], SpoX [22], Temporal- Rover [17], and Traemathes [3, 5] have been proposed n reent years. Among those systems, JavaMOP s the only formalsm-ndependent parametr montorng system. Desgnng and developng a system whh an effently montor parametr propertes s not trval. Many advaned montorng systems often show prohbtve overhead for omplex spefatons or for omplex applatons. Ths s beause the parameters are dynamally bound to objets at runtme, resultng n a potentally unlmted number of parameter bndngs [24]. Sne the property needs to be 1
Map UnsafeIterator(Map m, Colleton, Iterator ) { reaton event getset after(map m) returnng(colleton ) : (all(set Map+.keySet()) all(colleton Map+.values())) && target(m) {} event getter after(colleton ) returnng(iterator ) : all(iterator Iterable+.terator()) && target() {} event modfymap before(map m) : (all(* Map+.lear*(..)) all(* Map+.put*(..)) all(* Map+.remove(..))) && target(m) {} event modfycol before(colleton ) : (all(* Colleton+.lear(..)) all(* Colleton+.offer*(..)) all(* Colleton+.pop(..)) all(* Colleton+.push(..)) all(* Colleton+.remove*(..)) all(* Colleton+.retan*(..))) && target() {} event useter before(iterator ) : (all(* Iterator.hasNext(..)) all(* Iterator.next(..))) && target() {} ere : getset (modfymap modfycol)* getter useter* (modfymap modfycol)+ useter @math { System.err.prntln("a volaton deteted!"); } } Fgure 1. Map UnsafeIterator spefaton n JavaMOP heked for eah parameter bndng ndvdually, the runtme and memory overhead of montorng a parametr property an be arbtrarly large. There has been a sgnfant amount of researh on mprovng the runtme and the memory performane of parametr montorng [5, 11, 24, 28]. Thanks to these efforts, parametr montorng has beome relatvely pratal: n most ases the runtme/memory overhead s not noteable; and t generally osts less than 15% runtme overhead to montor even omplex objet systems and/or parametr propertes; and even n extreme ases the runtme overhead s reasonable [24]. Stat optmzaton tehnques, suh as [11], an mprove performane even further (but we do not nvestgate stat analyses n ths paper). For the rest of ths paper, whenever we say montorng we mean parametr montorng. To the best of our knowledge, all earler efforts on parametr montorng have been fousng on better performane when montorng a sngle spefaton. In realty, t s qute lkely to have many spefatons for a gven program. A natural queston then s: an we do better than the sum of the parts? That s, an we montor multple spefatons at the same tme wth less overhead than the sum of the overheads of eah ndvdual property? Theoretally, f all spefatons are ndependent from eah other wthout any overlap n delared events or parameter types, there s no way to montor them more effently. However, n prate, there are lkely multple spefatons on the same lass, often sharng some events and parameter types. Among 137 spefatons from [26], only 42 spefatons are totally ndependent from all the other spefatons. In ths paper, we present salable parametr montorng tehnques for montorng multple smultaneous spefatons more effently n the presene of some overlaps between spefatons. The man dea of the salable tehnques s to share resoures for montorng between spefatons, redung the memory usage and utlzng the ahes more often. Sne our salable tehnques are formalsmndependent and address general ssues n the ndexng tree tehnque, they an be appled to other parametr montorng systems that use smlar ndexng tree strutures. Also, they are orthogonal to other optmzaton tehnques lke stat optmzaton [10, 11, 18, 27], whh redue runtme and memory overhead sgnfantly. However, we delberately dsabled stat optmzatons n ths paper to measure the effetveness of our salable tehnques properly. For the evaluaton of our work on salablty, we use the 137 spefatons from [26]. These spefatons are based on the Java 6 API doumentaton onernng three man pakages: java.o, java.lang, and java.utl. Sne there s no other parametr montorng tool whh s apable of pratally montorng the 137 spefatons smultaneously, we ompare our work on salablty to the prevous verson of JavaMOP. The average runtme overhead of the salable JavaMOP run on verson 9.12 of the DaCapo [7] benhmark sute for the 137 smultaneous spefatons s 147%, whh s almost half of the 262% overhead that the prevous JavaMOP shows. Moreover, ths runtme overhead s smaller than the sum of the overheads from montorng them ndvdually, whh s 178% on average, whle the prevous verson of JavaMOP shows more overhead when runnng all propertes together than the sum of the overheads when run ndvdually, whh s 243% on average. The rest of ths paper s strutured as follows: Seton 2 provdes bakground on parametr montorng as used n the prevous verson of JavaMOP; Seton 3 presents a thorough proflng of urrent runtme overheads from montorng, and dsusses the man urrent bottleneks n montorng; Seton 4 dsusses our salable parametr montorng tehnques n detal; Seton 5 presents our evaluaton results for the 137 spefatons; Seton 6 dsusses some neffetual approahes that we have tred; and Seton 7 onludes. Contrbutons Ths paper s ontrbutons are as follows: Thorough Proflng on overhead from montorng; Salable Montorng Tehnques that redue runtme and memory overhead sgnfantly when montorng multple spefatons smultaneously; Large Sale Evaluaton that montors 137 parametr spefatons smultaneously for the frst tme. 2
montor(s) montor(s) ahe ht found event (p 1,, p n ) Indexng Cahe ahe mss Indexng Tree m m Map Montor not exst Set of Montors montor(s) oped Copy States from smaller parameter nstane Fgure 3. Some ndexng trees for Map UnsafeIterator montor Update a reaton event? reated Create a New Montor nothng to opy yes Stop Fgure 2. Overvew of the prevous montorng mehansm 2. Bakground To explan salable parametr montorng tehnques, we frst ntrodue some bakground on the prevous parametr montorng algorthm, ts mplementaton, and data strutures for t. A more n-depth dsusson of these detals an be found n [24, 28]. In parametr montorng, a spefaton montors a program exeuton trae onsstng of parametr events. The spefaton must hold for eah parameter nstane, whh s a partal mappng from parameters to objets. A parametr event e wth parameter nstane p s denoted as e p. For example, the Map UnsafeIterator spefaton (Fgure 1) should hold for all ombnatons of a Map nstane, a Colleton nstane, and an Iterator nstane that are related by spefaton events. We sle the program exeuton trae for eah parameter nstane so that a montor for eah parameter nstane an forget about parameters and fous on the property. In ths way, a montor s ndependent from parameters, resultng n a formalsm-ndependent parametr montorng system. In ths paper, we omt the detaled algorthm of the parametr montorng, but summarze t n Fgure 2 sne the detaled algorthm s not requred n explanng our work; the nterested readers may refer to [28]. Upon an event, t retreves the montor(s) from the ndexng tree. If there s no montor that an aept a gven parametr event, e p, ether an exstng montor from a subset of p s oped, or f no suh montor exsts and e s a reaton event, a new montor s reated. For better performane, t uses an ndexng ahe that stores the prevously aessed montor(s). no The ndexng tree s an effent means to loate the montors for a gven parameter nstane. The ndexng tree s mplemented as a mult-level map that, at eah level, ndexes eah parameter objet of the parameter nstane. For example, Fgure 3 shows two ndexng trees out of sx n the Map UnsafeIterator spefaton. The ndexng tree for Map, Colleton, Iterator (left tree) s a 3-level map. Wth a map, a olleton, and an terator, we an retreve the related montor. The ndexng tree for Map, Colleton (rght tree) s a 2-level map. For a map and a olleton, ths ndexng tree returns a set of montors, beause there an be multple montors for the gven map and olleton (one montor for eah terator). If an ndexng tree stores all parameter objets dretly, t wll blok them from beng garbage olleted, leadng to a memory leak. Instead of storng parameter objets dretly, the ndexng tree uses the WeakReferene lass from the Java API. WeakReferene allows a referene to an objet that wll not dsallow garbage olleton for sad objet. When the objet s garbage olleted, the JVM hanges the referent feld of all weak referenes referrng to t to null. In ths way, parameter objets an be garbage olleted wthout any nterferene from montorng. Mappngs n the ndexng tree an be broken when parameter objets are garbage olleted and ther weak referenes pont to null. The Java API provdes a way to queue weak referenes of garbage olleted objets nto a RefereneQueue objet. By usng ths feature, broken mappngs an be easly removed from the ndexng tree. However, usng ths feature slows down the system sgnfantly, beause queung weak referenes nvolves synhronzaton. Whle other general data strutures lke the Apahe Commons Colletons Lbrary [20], use ths feature, our mplementaton of the ndexng tree does not use t for the performane reasons. Instead of usng RefereneQueue, we terate through mappngs and remove broken ones. Surprsngly, teratng through mappngs s sgnfantly faster than usng the queung feature from the Java API. Ths self-leanng feature of our ndexng tree also allows for effent garbage olleton of unneessary montors [24]; when we terate through the mappngs, we smply hek whether any of the montors have beome unneessary. 3
Overhead Fraton Method Name 355% Orgnal Program 281% MOPSet.event 205% MOPMap.leanup 130% System.denttyHashCode 69% MOPMap.get 67% MOPSet.sze 51% MOPMap.endObjet 28% Aspet Code 27% MOPMap.full leanup 22% MOPSet.endObjet Table 1. Overhead dstrbuton when montorng bloat (total overhead: 1330%) Overhead Fraton Method Name 479% Orgnal Program 90% MOPSet.event 56% MOPMap.leanup 28% System.denttyHashCode 25% MOPSet.sze 13% MOPMap.get 8% MOPMap.full leanup 7% MOPMap.endObjet 6% WeakReferene nt 5% MOPSet.endObjet Table 2. Overhead dstrbuton when montorng pmd (total overhead: 831%) 3. Overhead Analyss In ths seton, we analyze the overhead of montorng to fnd the man bottleneks n montorng. For ths analyss, we have seleted 9 spefatons 1 that have aused the most overhead n prevous evaluatons. We run the spefatons on the bloat and pmd benhmarks beause they have shown the largest overheads among the benhmarks n our evaluaton (Seton 5). We use the same system settngs from the evaluaton, and HPROF, the Heap/CPU proflng tool nluded n the Sun JDK [2] s used to obtan performane statsts. There are two modes for CPU usage analyss n HPROF: the CPU Usage Tmes Profle and the CPU Usage Samplng Profle. The CPU Usage Tmes Profle adds a onsderable amount of overhead, obstrutng the analyss of the atual bottleneks. Moreover, we do not need to know the exat tme dstrbuton to fgure out where bottleneks our. The CPU Usage Samplng Profle, whh auses less performane degradaton, s good enough for ths analyss. Sne the CPU Usage Samplng Profle does not ombne the results for the same method of dfferent objet nstanes, we manually ombne them and ategorze. Tables 1 and 2 summarze the proflng results for montorng bloat and pmd. The results for bloat show total overhead of 1330%; that s 1430% total exeuton tme ompared to the orgnal non-montored bloat. In the same way, montorng pmd shows a total overhead of 831%. Beause proflng an hange the program behavor, numbers may ontan errors, so they should be onsdered as rough estmatons. The MOPSet.event entry n Table 1 shows the overhead spent updatng montor states when events our. Ths omponent s formulated from the property of the spefaton, and s already optmzed well. MOPMap.leanup and MOPMap.full leanup remove mappngs of garbage olleted parameter objets and montors. The dfferene s whether t partally or fully sans the map. These leanup 1 Map UnsafeIterator, Colleton UnsafeIterator, Iterator HasNext, Colletons SynhronzedColleton, NavgableMap Modfaton, Colletons SynhronzedMap, Iterator RemoveOne, Lst UnsynhronzedSubLst, Colletons SortBeforeBnarySearh Peak Young Garbage Full Garbage Desrpton Memory Usage Colleton Tme Colleton Tme Orgnal bloat 5MB 6% 2% Orgnal pmd 21MB 7% 8% Montorng bloat 970MB 278% 258% (out of 1330% overhead) Montorng pmd 603MB 172% 181% (out of 831% overhead) Table 3. Memory usage analyss methods are well tuned so that they are unlkely to be mproved sgnfantly. The methods MOPMap.endObjet and MOPSet.endObjet propagate nformaton about garbage olleted parameters. They onsst of smple statements and have already been thoroughly optmzed [24]. System.denttyHashCode s the system default hashng funton provded n the Java API, whh s based on referene dentty nstead of the equals method provded by lasses. It returns the same hash ode for objets a and b f a == b, and tres to return dfferent odes otherwse, but unqueness s not guaranteed. Although ths s just one of several statements n the MOPMap.get method that retreves montor(s) for a parameter nstane, t produes more overhead than all other methods ombned. Callng ths method s unavodable sne t s used to retreve keys n the MOPMap mplementaton. However, we need to all ths method as lttle as possble. Whle many montorng omponents show sgnfant overhead, t s notable that the orgnal program omponents are also slower when montorng s present (.e. 100%). To understand ths stuaton, we analyze the memory usage when montorng, usng Java Management Extensons (JMX) [1]. Table 3 summarzes the memory usage analyss. Montorng trggers huge memory overheads, resultng n sgnfantly more garbage olleton tme. Wth respet to the orgnal program exeuton tme (100%), n montorng bloat, young objet garbage olleton takes 278% and full garbage olleton takes 258%. In total, garbage olleton takes 536% when montorng bloat and 353% when montorng pmd. Ths explans why the orgnal omponents of the ode run far slower when montorng s present. 4
We must onlude that the man remanng bottlenek to runtme performane n montorng s exessve memory usage. Huge memory overhead auses more frequent and longer garbage olletons, resultng n larger runtme overhead. We should redue memory overhead to optmze runtme performane. For example, n Table 2, WeakReferene objet ntalzatons show 6% overhead, whle there s no other lass ranked n the result. Ths s beause there s a very large number of weak referenes. We need to redue the number of objets reated for montorng purposes, espeally weak referenes. 4. Optmzatons for Salablty The more spefatons that we montor smultaneously, the more overhead. Our goal s to mprove the overhead n the presene of multple spefatons by fndng strutures and parts of the montorng algorthm that may be shared between dfferent spefatons. If no spefatons overlap wth others, n terms of delared events or parameters types, there s nothng muh we an mprove. Theoretally, the overhead n ths ase wll be the sum of overheads from montorng them ndvdually. When the memory overhead s exessve, t an be worse than the sum beause of the garbage olleton behavor. However, n prate, there are generally multple spefatons for eah lass, often sharng some events. Among 137 spefatons from [26], only 42 spefatons are totally ndependent from all other spefatons. Another 95 spefatons share parameters or events wth some of other spefatons. By sharng resoures between overlappng spefatons we an aheve a truly salable parametr runtme montorng system. In ths Seton, we explan new tehnques for nreasng runtme and memory performane frst, then we fous on the bg pture of the new montorng mehansm, frst presented here, n omparson wth the prevous montorng mehansm (Fgure 2). Our tehnques are formalsm-ndependent and general so that they an be appled to other parametr montorng systems that use smlar ndexng tree strutures. 4.1 Global WeakReferene Table As explaned n Seton 2, WeakReferene s a referene lass that refers to an objet wthout blokng t from garbage olleton. The ndexng tree uses weak referenes to store parameter objets n ts mappngs, wthout blokng garbage olletons. In prevous versons of JavaMOP, there was no ollaboraton between spefatons, so eah spefaton reated a weak referene objet for eah parameter objet. Thus, multple weak referenes were potentally reated for the same parameter objet, f t appeared n dfferent spefatons. There s no need to have multple opes of WeakReferene; t smply wastes memory. We ntrodue a global WeakReferene table, mplemented n the lass GlobalWeakRefTable, for eah parameter type, whh all spefatons share. Ths table takes a parameter objet as an nput and outputs a weak referene. If there s no weak referene n the table for the nput objet, the table wll reate one. Thus, weak referenes wll be reated only by ths table and there wll be exatly one opy for one parameter objet. Also, upon a non-reaton event, we an query the exstene of the weak referene wthout reatng one. If there s no weak referene for the parameter objet n the table, then there s no montor n any spefaton for the parameter objet. Thus, we an skp the rest of the steps for hekng the exstene of montors for the non-reaton event. The funtonalty of the GlobalWeakRefTable s smlar to HashMap from the Java API, but ts mplementaton s totally dfferent. If the GlobalWeakRefTable stores keys (parameter objets) and values (weak referenes) n ts nternal table lke HashMap, t wll ause memory leaks. Instead, the GlobalWeakRefTable stores only weak referenes. Sne weak referenes an refer to the orgnal objets, we an retreve the weak referene for an objet by hekng f the weak referene ponts to the objet. Although the GlobalWeakRefTable ntrodues one more step n the montorng mehansm, t redues not only memory overhead by redung the number of weak referenes, but also runtme overhead. From the analyss n Seton 3, we know that System.denttyHashCode() auses the most runtme overhead n the ndexng trees. Instead of allng ths method n eah ndexng tree, eah GlobalWeakRefTable alls ths method and stores the result n weak referenes so that ndexng trees an reuse t. To allow ths, we mplement MOPWeakReferene, a sublass of WeakReferene whh has a hashode feld, and hange the ndexng tree to take MOPWeakReferene as nput rather than parameter objets. Wth ths hange, ndexng trees no longer all the System.denttyHashCode() method, removng the man overhead n aessng them. The GlobalWeakRefTable alls ths method at most one for eah parameter objet n an event, mnmzng the number of the method alls to System.denttyHashCode(). The GlobalWeakRefTable s essentally the same as the ndexng tree exept that t does not return montors. It leans up referenes to garbage olleted objets and expands the nternal data struture just lke the ndexng tree does [24]. We an redue overhead even more by ombnng GlobalWeakRefTables wth relevant ndexng trees, redung the number of tables and maps. If there s an ndexng tree that has the same parameter type at the frst level as the GlobalWeakRefTable, they an be ombned nto one data struture. In the majorty of ases, GlobalWeakRefTables an be ombned wth ndexng trees. Among the many GlobalWeakRefTables for the 137 spefatons from [26], there are only two GlobalWeakRefTables that annot be ombned nto ndexng trees when montorng ndvdually, and all GlobalWeakRefTables an be ombned nto ndexng trees when montorng them smultaneously. 5
m m m m Map Montor Set of Montors Map Montor Set of Montors Fgure 4. Indexng trees for Map UnsafeIterator before ombnng Fgure 5. Indexng trees for Map UnsafeIterator after ombnng 4.2 Cahes for Global WeakReferene Table Under our new tehnque, the GlobalWeakRefTable s the most frequently aessed data struture n montorng sne all events should query ths table before aessng any ndexng tree. Therefore, t s mportant to optmze ths table. One natural and ommon method of optmzaton s ahng. In the prevous approah, there was already an ndexng ahe (Fgure 2). After addng GlobalWeakRefTables, t ahes not only a montor but also weak referenes for the montor. Thus, t ats as a ahe for both the ndexng tree and the GlobalWeakRefTable. Although the ndexng ahe provdes a good ahe ht rato wthn a spefaton, t s not good enough when montorng multple spefatons. Frst, sne there are multple events from dfferent spefatons for the same objet, t s lkely that multple spefatons onseutvely aess GlobalWeakRefTables for the same objet, when ther ndexng ahes mss. Seond, the ndexng ahe s a one-entry ahe whh s fragle f more than two objets are frequently used together n an nterleaved way. To mprove the performane upon ths observaton, we now use a one-entry level-1 ahe to handle the frst ase and a mult-entry level-2 ahe to handle the seond ase. On a query to the table, we frst hek the one-entry ahe and when t msses, we hek the mult-entry ahe. However, f we lnearly searh n the mult-entry ahe, the overhead wll nrease lnearly wth the number of entres n the ahe. Thus, we use a mappng so that we an hek only one entry at a tme. Beause eah nstrumentaton pont tends to aess the same objet onseutvely, we ndex the mult-entry ahe by a few least sgnfant bts of the unque d number for nstrumentaton ponts, provded by AspetJ. In ths way, the mult-entry ahe s mplemented effently. The beneft of the ahes surpasses the overhead from mantanng the ahes n most ases. 4.3 Combnng Indexng Trees The ndexng tree s one of the major bottleneks n terms of both runtme and memory performane. It ontans all of the mappngs from parameter objets to montors. The sze of the ndexng tree grows as the spefaton reates more montors. Addtonally, the ndexng tree leans up mappngs of garbage olleted parameter objets and montors by tself. Therefore, we an redue runtme and memory overhead by ombnng ndexng trees. We an ombne ndexng trees f ther defned parameter types share the same prefx. For example, ndexng trees for Colleton, Iterator and Colleton an be ombned but ndexng trees for Map, Colleton, Iterator and Colleton, Iterator annot be ombned sne the frst parameter type, Map, appears only n the frst. Combnng ndexng trees between dfferent spefatons s also possble as long as they satsfy the ondton for ombnng. However, t s usually neffent beause there s nsuffent mappng overlap between spefatons (Seton 6). Thus, we ombne ndexng trees only wthn eah spefaton. Combnng ndexng trees n eah spefaton mproves not only the performane of montorng multple spefatons but also the performane of montorng eah spefaton. For example, Fgure 4 shows all ndexng trees for Map UnsafeIterator before ombnng them. There are sx ndexng trees for: 6
1. Map, Colleton, Iterator 2. Map, Colleton 3. Map 4. Colleton, Iterator 5. Colleton 6. Iterator Among sx ndexng trees, the frst three ndexng trees an be ombned nto one, and the fourth and ffth ndexng trees an be ombned as well. As a result, three ndexng trees wll reman (Fgure 5). 4.4 Spefaton Atvator In montorng multple spefatons, suh as the 137 spefatons from [26], t s ommon that only some of them are atvely montored when appled to a gven program. Ths s beause one program generally does not over every spefaton n suh a large set of standardzed spefatons. When a spefaton does not have any reaton event durng the exeuton of a program, t does not need to montor the program at all. We keep a boolean value as an atvator for eah spefaton and atvate t when there s at least one reaton event. When the spefaton s not atvated, we gnore all non-reaton events, suppressng the unneessary overhead. If there s no reaton event at all durng the exeuton, all non-reaton events wll be gnored. Ths smple tehnque suessfully deatvates unneessary spefatons durng the exeuton of a program, redung unneessary runtme overhead. Even n montorng a sngle spefaton, t an effetvely remove unneessary overhead. In our evaluaton (Seton 5), some spefatons are effetvely deatvated and show no overhead at all. The overhead of mantanng spefaton atvators s essentally unnoteable, far less than the error range of our evaluaton (up to 3%). 4.5 Summary of New Montorng Tehnques Fgure 6 summarzes the salable parametr montorng mehansm usng tehnques ntrodued n ths seton. Compared to the prevous montorng mehansm (Fgure2), there s an atvator at the begnnng and the GlobalWeakRefTable before the ndexng tree. Also, nstead of parameter objets, t uses weak referenes n aessng ndexng trees. The man dea of our salable parametr montorng s that the GlobalWeakRefTable allows sharng of weak referenes, redung memory overhead. Also, ahng on ths table redues runtme overhead over all spefatons usng t. Moreover, there are fewer ndexng trees and there s no hash method all from the ndexng tree. Thus, the overhead from ndexng trees has been dramatally dereased. Sne the Copy State omponent and the Create a New Montor omponent also aess the ndexng tree to add new montors, overheads from both omponents derease as well. montor(s) montor(s) montor(s) montor Update Weak Referenes a reaton event? ahe ht montor exsts ahe ht, no montor found oped event (p 1,, p n ) yes, atvate a reaton event? reated Indexng Cahe GlobalWeakRefTable Indexng Tree Copy State from smaller parameter nstane Create a New Montor Stop atvated? no ahe mss found or reated not exst yes nothng to opy yes no no not exst Fgure 6. Overvew of the salable parametr montorng mehansm 5. Evaluaton In ths seton, we evaluate JavaMOP wth the presented salablty mprovements on 137 spefatons from [26]. Even before ths work, JavaMOP had the best runtme performane of any montorng system, whle mantanng ompettve memory performane [24]. Also, to the best of our knowledge, there s no other parametr montorng tool whh s apable of pratally montorng 137 spefatons smultaneously. Thus, we ompare our work on salablty to the prevous verson of JavaMOP (JavaMOP 2.3). 5.1 Expermental Settngs For our evaluaton, we used a Pentum 4 2.66GHz / 2GB RAM / Ubuntu 9.10 mahne and Sun JVM 1.6.0 10. For nstrumentng benhmark programs wth JavaMOP montorng ode, we used verson 1.6.11 of the AspetJ ompler (aj). We montor 137 spefatons for verson 9.12 of the DaCapo (DaCapo 9.12) benhmark sute. We also present the result from the bloat benhmark n the old verson of the DaCapo (DaCapo 2006-10) benhmark sute, 7
java.o java.lang java.utl All # of spes 30 49 58 137 Prevous Salable Prevous Salable Prevous Salable Prevous Salable Org (se) Sum Together Sum Together Sum Together Sum Together Sum Together Sum Together Sum Together Sum Together bloat 14.4 6-2 -3-3 289 327 300 339 1203 1493 719 632 1498 1950 1016 1033 avrora 13.9 7 8 0 5 19 10 12 9 468 336 273 215 494 364 285 235 batk 3.5 0 4 0 2 0 3 0 1 50 37 53 26 50 40 53 26 elpse 79.5 0-1 0 2 0 1 42 1 0 1 7 0 0-2 49 2 fop 2.0 8 7 18 7 96 56 52 55 584 450 380 380 688 N/A 450 N/A h2 18.7 0 0 13 2 17 24 34 19 71 54 86 47 88 73 133 62 jython 13.6 10-1 0 1 27 21 18 18 112 90 79 75 149 121 97 95 lundex 2.9 9 5 5 5 5 5 12 3 3 5 5 4 17 9 22 12 lusearh 25.3 14 13 17 16 28 34 21 32 29 26 30 30 71 75 68 79 pmd 8.4 0-1 0 3-3 8 0 4 858 898 657 476 855 988 657 486 sunflow 32.5 0 1 0 0 0 1 0 1 4 8 13 6 4 10 13 7 tomat 14.1 0-1 0 0 0 0 0 0 0 0 0-1 0 0 0 1 tradebeans 45.7 40 12 11 2 33 3 26 1 51 1 50 3 124-1 87 2 tradesoap 95.0 0 2 11 0 16 2 9 1 12 0 4 0 28 0 24 0 xalan 20.9 6 12-7 30-17 -12-32 -16 38 52 31 61 27 34-8 24 Table 4. Average perent runtme overhead for Prevous JavaMOP (JavaMOP 2.3) and Salable JavaMOP (JavaMOP 3.0) (onvergene wthn 3%, N/A: nstrumentaton rashes) beause t generates large overheads and t s mssng n the new verson. We used the default data nput sze, and the -onverge opton so that the exeuton tme result onverges wthn 3%. AspetJ nstrumentaton an ause the ode to run dfferently, sometmes resultng n negatve overheads even wthout montorng. Also, montorng affets the garbage olleton behavor wth more memory pressure, often mprovng garbage olleton tme; ths also aounts for the negatve overheads. All 137 spefatons from [26] are based on the Java 6 API doumentaton onernng three man pakages: 30 spefatons for java.o, 49 spefatons for java.lang, and 58 spefatons for java.utl. Some spefatons are related to the end of the program exeuton. However, two versons of DaCapo terate a benhmark program n one exeuton untl the exeuton tme onverges. Therefore, we modfed those spefatons slghtly so that they ath the end of teraton of a benhmark program. 5.2 Results and Dsussons Table 4 and 5 summarze the results of the evaluaton on the two versons of JavaMOP. Montorng 137 spefatons smultaneously s a onsderably hallengng task. Whle montorng 137 spefatons wth bloat, there are 839,575,093 events and 27,826,935 montors reated. Wth pmd, there are 68,438,904 events and 9,510,880 montors reated. Also, n JavaMOP 2.3, 129 ndexng trees are requred, but the ndexng tree ombnaton tehnque (Seton 4.3) redues the number of ndexng trees to 105. Therefore, t s not surprsng to see a huge overhead. Although the prevous verson of JavaMOP was the most effent parametr montorng system untl ths paper, t shows more than 100% overhead on fve benhmarks out of 15, nludng fop. For fop, the nstrumentaton rashes beause the added nstrumentaton results n a method larger than the 64KB lmt for Java methods. The method sze was already too bg before the nstrumentaton, and our nstrumentaton makes t exeed the lmt. In regular programmng, the lmt of 64KB seems reasonable; any method over 64KB should be re-desgned and dvded nto several methods. However, for proedurally generated ode, ths lmt mposed by Java seems too harsh. Whle we were unable to obtan overhead for fop wth 137 smultaneous spefatons n ether verson of JavaMOP, we do have numbers for montorng the spefaton of eah pakage separately. Table 4 shows the average perent runtme overhead of the two versons of JavaMOP. It shows the sum of overheads for montorng eah spefaton ndvdually, and the overhead of montorng them smultaneously, for eah benhmark. To avod the error aumulaton, we exlude overheads under 3% for the summaton. Overall, Salable Java- MOP shows sgnfantly less runtme overhead than the prevous verson of JavaMOP. In montorng multple spefatons, the prevous verson of JavaMOP shows hgher overheads than the sum of overheads n many plaes. Ths s beause heavy memory pressure from multple spefatons trggers garbage olleton more often. However, Salable JavaMOP shows muh less overhead than the sum of overheads n most ases. The prevous JavaMOP shows 1950% overhead when montorng all 137 spefatons for bloat, whle the sum of overheads s 1498%. For pmd, t shows 988% overhead when all spefatons are montored, whle the sum of overheads s 855%. However, Salable JavaMOP shows 1033% and 486% overheads for bloat and pmd, respetvely, when all spefatons are montored. These overheads are almost half of what the prevous verson showed. Also, they are less than the sums of overheads n the Salable JavaMOP, whh are 1016% and 657%, respetvely. Note that Salable Java- MOP also mproves the runtme performane of montorng a sngle spefaton, resultng n smaller sums. 8
java.o java.lang java.utl All # of spes 30 49 58 137 Prevous Salable Prevous Salable Prevous Salable Prevous Salable Org Sum Together Sum Together Sum Together Sum Together Sum Together Sum Together Sum Together Sum Together bloat 4.9 5.0 5.7 5.0 4.9 559.2 626.5 586.9 626.8 330.2 1011.2 131.7 282.1 884.6 1500 714.5 1466.4 avrora 4.7 4.7 4.5 4.7 4.4 10.9 12.3 7.3 12.5 44.8 73.1 29.5 53.9 51.0 737.2 32.1 198.4 batk 77.3 77.3 76.3 77.3 79.2 77.3 75.1 77.3 72.5 99.2 166.2 88.3 101.7 99.2 166.7 88.3 105.6 elpse 101.0 101.0 100.0 101.0 99.4 101.0 103.2 101.0 109.1 101.0 113.8 101.0 104.8 101.0 108.0 101.0 113.5 fop 23.9 22.9 25.8 22.0 25.9 79.0 73.2 55.8 58.0 341.4 402.6 255.3 185.2 395.5 N/A 285.5 N/A h2 267.1 267.1 265.3 267.1 260.9 303.5 327.1 308.0 357.7 2307.9 1176.0 1468.6 957.1 2344.3 1343.5 1509.5 1118.4 jython 21.9 22.1 23.0 21.9 22.9 57.0 76.1 69.2 86.5 91.8 191.8 81.1 71.2 127.1 240.2 128.4 141.2 lundex 6.8 5.7 7.9 8.1 7.0 8.1 18.8 7.0 19.7 6.4 8.8 6.7 12.8 6.6 20.8 8.2 20.8 lusearh 4.6 4.4 4.7 4.0 4.6 4.4 7.5 5.1 7.0 4.8 4.5 5.1 4.9 4.4 7.4 5.0 8.3 pmd 22.3 22.3 25.1 22.3 26.3 87.1 38.5 68.9 38.0 430.5 1474.9 405.4 424.0 495.3 1457.4 452.0 450.8 sunflow 4.5 4.5 4.5 4.5 5.0 4.5 6.6 4.5 6.3 4.7 5.5 4.3 4.4 4.7 7.5 4.3 7.0 tomat 11.7 11.7 11.7 11.7 12.3 11.7 11.8 11.7 12.3 11.7 11.4 11.7 11.7 11.7 11.9 11.7 11.5 tradebeans 62.9 64.3 63.3 67.6 63.2 66.6 63.1 65.8 63.1 66.3 63.0 64.4 63.9 71.4 63.1 72.0 62.9 tradesoap 63.9 63.9 64.2 63.9 64.1 69.6 62.1 66.0 64.4 68.3 65.4 66.0 65.4 74.0 64.7 68.1 61.4 xalan 4.9 4.9 5.0 5.0 4.9 20.4 21.4 21.3 21.9 4.9 5.0 4.6 4.9 20.4 22.9 21.1 23.7 Table 5. Peak memory usage (n MB) for Prevous JavaMOP (JavaMOP 2.3) and Salable JavaMOP (JavaMOP 3.0) (durng 5 teratons, N/A: nstrumentaton rashes) Table 5 summarzes the peak memory usage durng 5 teratons. In a smlar way to the runtme result, t shows the sum of memory overheads from montorng eah spefaton ndvdually. In the sum of the peak memory usage, the orgnal peak memory usage s ounted only one. For example, on bloat, whh shows 4.9MB peak memory usage, f two spefatons show 5.5MB and 6.2MB peak memory usage, respetvely, the sum of peak memory usage s 6.8MB. Overall, the Salable JavaMOP shows sgnfantly less memory overhead than the prevous verson of Java- MOP. Smlar to runtme performane, Salable JavaMOP uses less memory, not only when montorng multple spefatons smultaneously, but also when montorng them ndvdually. In montorng spefatons ndvdually, n total, Salable JavaMOP uses 25% less memory than the prevous verson. In montorng multple spefatons smultaneously, for avrora and pmd, Salable JavaMOP shows 3.7 tmes and 3.2 tmes less peak memory usage than the prevous verson, respetvely. Also, n total, Salable JavaMOP uses 34% less memory than the prevous verson, n montorng multple spefatons smultaneously. Montorng a large number of spefatons shows dfferent memory usage from montorng a sngle spefaton. Durng montorng proess, a large number of objets s generated for the purposes of montorng. Many of these montorng objets must be garbage olleted. Sne the JVM ontrols the garbage olleton throughput so that t does not overwhelm the entre exeuton tme, the garbage olleton mght not be able to lean up all garbage objets on tme. Ths an ause parameter objets to lve longer than usual, delayng aompaned montorng resoures from beng garbage olleted. In ths ase, the JVM smply onsumes more memory as long as there s more spae left. After reahng the memory lmt, t starts spendng more tme for garbage olleton. Ths explans for avrora and others, why montorng multple spefatons smultaneously shows more peak memory usage than the sum of peak memory usages of ndvdual montorng and the sum of peak memory usages of montorng spefatons n eah pakage. For example, for avrora and the Salable Java- MOP, montorng all spefatons n java.o, java.lang, and java.utl shows 4.4MB, 12.5MB, and 53.9MB, but montorng all of the spefatons smultatenously shows 198.4MB memory usage at peak. 6. Ineffetual Approahes In ths seton, we dsuss some neffetual approahes that we have tred whle mprovng the salablty of parametr montorng. Although they turn out to be neffetual n parametr montorng, some of them mght be useful n dfferent settngs or they mght nspre new effetual deas. Combnng Indexng Trees between Spefatons As mentoned n Seton 4.3, we ombne ndexng trees only wthn eah spefaton. If we ombne ndexng trees for dfferent spefatons, as well, we an redue the number of ndexng trees even more. However, there s a lot of wasted spae n the ombned ndexng tree. For example, an ndexng tree A maps p 1 to m 1 and p 2 to m 2, and another ndexng tree B maps p 2 to m 3 and p 3 to m 4. The ombned ndexng tree of A and B wll map p 1 to (m 1, ), p 2 to (m 2, m 3 ), and p 3 to (, m 4 ). All empty spaes ndated by wll be wasted whle the ndexng trees A and B do not have empty spae. More memory overhead from wasted spae trggers more garbage olleton, slowng down the montorng. Enhaned Indexng Cahe The ndexng ahe provdes faster retreval of montors from the ndexng tree. There are several deas to mprove ts ht rato. We an apply a mult-entry ahe from Seton 4.2. Also, we an ahe not only montors but also lak thereof to save searhng the ndexng tree for nothng. However, sne the ndex- 9
ng ahe provdes already a hgh ht rato and the ost to aess the ndexng tree s already dereased by the GlobalWeakRefTable, these enhanements to the ndexng ahe do not mprove the performane. Certanly those deas nrease ahe ht rato, but ther benefts are anelled out by the overheads neessary to support them. Indexng Tree Cleanng by GlobalWeakRefTables Sne we an manage all weak referenes for eah parameter type n one plae, the GlobalWeakRefTable, we an let the GlobalWeakRefTable lean up the ndexng trees. In ths way, we an remove garbage olleted parameter objets from all ndexng trees at one, elmnatng the need for partal leanups. Note that partal leanups ould our even when there s no garbage olleted parameter objet. We an also have a bt map n the weak referene to ndate to whh ndexng trees the referent belongs so that we need hek only the ndexng trees that atually ontan t. However, ths approah only moves leanup osts from ndexng trees to the GlobalWeakRefTable, showng no mprovement. The leanup by the GlobalWeakRefTable s more effetve beause t knows whh weak referenes should be removed. However, leanng up from outsde of the ndexng tree osts more beause we must loate the entry before we an remove t. Statsts-Based Indexng Tree Cleanng As mentoned prevously, partal leanups at ndexng trees an our even when there are no garbage olleted parameter objets. Sne we have the GlobalWeakRefTable, we an keep statsts about garbage olleted parameter objets and use t for dedng whether to trgger a partal leanup. However, n most ases, there are garbage olleted parameter objets. Savng a relatvely small number of partal leanups does not ompensate the overhead neessary. Event Atvator Smlar to the spefaton atvator (Seton 4.4), non-reaton events an be skpped f there s no montor reated for the parameter of the event. However, ths approah does not mprove the performane beause the spefaton atvator already works effetvely and the GlobalWeakRefTable already returns no weak referene f there was no reaton event for the parameter objet. Thus, ths approah only ntrodues an overhead of mantanng atvators (boolean varables), although the overhead s too small to be notable. 7. Conluson Parametr montorng s a tehnque for mprovng the relablty of software that has reeved an ever nreasng amount of attenton. Prevous work on parametr montorng has foused on the performane of montorng sngle propertes n solaton. Realst uses of montorng, however, nvolve montorng many propertes smultaneously, as the large number of propertes from [26] an attest. In ths paper we have mproved the effeny of JavaMOP wth respet to montorng multple smultaneous propertes; as an added bonus, we also mproved performane n the ase of a sngle property. We preformed a thorough analyss of the remanng bottleneks n the JavaMOP system, and we addressed those that ould be addressed wthout addng more runtme overhead than they save. The ases that were neffetual, presented n ths paper, show that sometmes t s more expensve to address an neffeny than to let t be. The remanng ases produed real, tangble performane enhanements, n some ases halvng overhead n a system that was already heavly optmzed. Aknowledgments Ths work s supported by NSF grant CCF-0916893. Referenes [1] Java Management Extensons. http://www. orale.om/tehnetwork/java/javase/teh/ javamanagement-140525.html. [2] Hprof: A heap/pu proflng tool n j2se 5.0. http: //java.sun.om/developer/tehnalartles/ Programmng/HPROF.html. [3] C. Allan, P. Avgustnov, A. S. Chrstensen, L. J. Hendren, S. Kuzns, O. Lhoták, O. de Moor, D. Seren, G. Sttampalam, and J. Tbble. Addng trae mathng wth free varables to AspetJ. In Objet-Orented Programmng, Systems, Languages and Applatons (OOPSLA 05), pages 345 364. ACM, 2005. [4] M. Arnold, M. Vehev, and E. Yahav. Qvm: an effent runtme for detetng defets n deployed systems. In Objet- Orented Programmng Systems, Languages, and Applatons (OOPSLA 08), pages 143 162. ACM, 2008. [5] P. Avgustnov, J. Tbble, and O. de Moor. Makng trae montors feasble. In Objet Orented Programmng, Systems, Languages and Applatons (OOPSLA 07), pages 589 608. ACM, 2007. [6] H. Barrnger, D. Rydeheard, and K. Havelund. Rule systems for run-tme montorng: from EAGLE to RULER. J. Log Computaton, November 2008. [7] S. M. Blakburn, R. Garner, C. Hoffman, A. M. Khan, K. S. MKnley, R. Bentzur, A. Dwan, D. Fenberg, D. Frampton, S. Z. Guyer, M. Hrzel, A. Hoskng, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovć, T. VanDrunen, D. von Dnklage, and B. Wedermann. The DaCapo benhmarks: Java benhmarkng development and analyss. In Objet-Orented Programmng, Systems, Languages and Applatons (OOPSLA 06), pages 169 190. ACM, 2006. [8] E. Bodden. J-LO, a tool for runtme-hekng temporal assertons. Master s thess, RWTH Aahen Unversty, 2005. [9] E. Bodden and V. Stolz. Traeheks: Defnng semant nterfaes wth temporal log. In Software Composton, pages 147 162, 2006. 10
[10] E. Bodden, F. Chen, and G. Roşu. Dependent adve: A general approah to optmzng hstory-based aspets. In Aspet-Orented Software Development (AOSD 09), pages 3 14. ACM, 2009. [11] E. Bodden, P. Lam, and L. Hendren. Clara: A framework for partally evaluatng fnte-state runtme montors ahead of tme. In Runtme Verfaton (RV 10), volume 6418 of LNCS, pages 183 197. Sprnger, 2010. [12] S. Chaudhur and R. Alur. Instumentng C programs wth nested word montors. In Model Chekng Software (SPIN 07), volume 4595 of LNCS, pages 279 283. Sprnger, 2007. [13] F. Chen and G. Roşu. Towards montorng-orented programmng: A paradgm ombnng spefaton and mplementaton. In Runtme Verfaton (RV 03), volume 89 of ENTCS, pages 108 127. Elsever, 2003. [14] F. Chen and G. Roşu. Java-MOP: A montorng orented programmng envronment for Java. In Tools and Algorthms for the Construton and Analyss of Systems (TACAS 05), volume 3440 of LNCS, pages 546 550. Sprnger, 2005. [15] F. Chen and G. Roşu. MOP: An effent and gener runtme verfaton framework. In Objet-Orented Programmng, Systems, Languages and Applatons (OOPSLA 07), pages 569 588. ACM, 2007. [16] M. d Amorm and K. Havelund. Event-based runtme verfaton of Java programs. ACM SIGSOFT Software Engneerng Notes, 30(4):1 7, 2005. [17] D. Drusnsky. The Temporal Rover and the ATG Rover. In Model Chekng and Software Verfaton (SPIN 00), volume 1885 of LNCS, pages 323 330. Sprnger, 2000. [18] M. Dwyer, R. Purandare, and S. Person. Runtme verfaton n ontext: Can optmzng error deteton mprove fault dagnoss. In Runtme Verfaton (RV 10), volume 6418 of LNCS, pages 36 50. Sprnger, 2010. [19] U. Erlngsson and F. B. Shneder. Irm enforement of java stak nspeton. In Symposum on Seurty and Prvay (SP 00), pages 246. IEEE, 2000. [20] T. A. S. Foundaton. The Apahe Commons Colletons. http://ommons.apahe.org/olletons/. [21] S. Goldsmth, R. O Callahan, and A. Aken. Relatonal queres over program traes. In Objet-Orented Programmng, Systems, Languages and Applatons (OOPSLA 05), pages 385 402. ACM, 2005. [22] K. W. Hamlen and M. Jones. Aspet-orented n-lned referene montors. In Programmng languages and analyss for seurty (PLAS 08), pages 11 20. ACM, 2008. [23] K. Havelund and G. Roşu. Montorng Java programs wth Java PathExplorer. In Runtme Verfaton (RV 01), volume 55 of ENTCS. Elsever, 2001. [24] D. Jn, P. O. Meredth, D. Grffth, and G. Roşu. Garbage olleton for montorng parametr propertes. In Programmng Language Desgn and Implementaton (PLDI 11), pages 415 424. ACM, 2011. [25] M. Km, M. Vswanathan, S. Kannan, I. Lee, and O. Sokolsky. Java-MaC: A run-tme assurane approah for Java programs. J. Formal Methods n System Desgn, 24(2):129 155, 2004. [26] C. Lee, D. Jn, P. O. Meredth, and G. Roşu. Towards ategorzng and formalzng the JDK API. Tehnal Report http://hdl.handle.net/2142/30006, Department of Computer Sene, Unversty of Illnos at Urbana-Champagn, Marh 2012. [27] M. Martn, V. B. Lvshts, and M. S. Lam. Fndng applaton errors and seurty flaws usng PQL: a program query language. In Objet Orented Programmng, Systems, Languages and Applatons (OOPSLA 07), pages 365 383. ACM, 2005. [28] P. O. Meredth, D. Jn, D. Grffth, F. Chen, and G. Roşu. An overvew of the MOP runtme verfaton framework. Internatonal Journal on Software Tehnques for Tehnology Transfer, 2011. [29] V. Stolz and E. Bodden. Temporal Assertons usng AspetJ. In Runtme Verfaton (RV 05), volume 144 of ENTCS, pages 109 124. Elsever, 2005. [30] R. E. Strom and S. Yemen. Typestate: A programmng language onept for enhanng software relablty. IEEE Transatons on Software Engneerng, 12:157 171, January 1986. 11