RV-Monitor: Efficient Parametric Runtime Verification with Simultaneous Properties

Transcription

1 RV-Montor: Effent Parametr Runtme Verfaton wth Smultaneous Propertes Qngzhou Luo 1, Y Zhang 1, Choonghwan Lee 1, Dongyun Jn 2, Patrk O Nel Meredth 1, Traan Florn Şerbănuţă 3, and Grgore Roşu 1 1 Unversty of Illnos at Urbana-Champagn, Urbana IL, USA 2 Samsung Eletrons, Suwon, Korea 3 Unversty of Buharest, Buharest, Romana {qluo2,yzhng173,grosu}@llnos.edu, {lnjus.y, pmeredt}@gmal.om, dongyun.jn@samsung.om, traan.serbanuta@fm.unbu.ro Abstrat. Runtme verfaton an effetvely nrease the relablty of software systems. In reent years, parametr runtme verfaton has ganed a lot of traton, wth several systems proposed. However, lak of real spefatons and prohbtve runtme overhead when hekng numerous propertes smultaneously prevent developers or users from usng runtme verfaton. Ths paper reports on more than 150 formal spefatons manually derved from the Java API doumentaton of ommonly used pakages, as well as a seres of novel tehnques whh resulted n a new runtme verfaton system, RV-Montor. Experments show that these spefatons are useful for fndng bugs and bad software prate, and RV- Montor s apable of montorng all our spefatons smultaneously, and runs substantally faster than other state-of-the-art runtme verfaton systems. 1 Introduton Runtme verfaton (RV) an nrease the relablty of software systems by dynamally verfyng property spefatons durng program exeuton. Runtme verfaton has ganed sgnfant nterest from the researh ommunty and there are nreasngly broad uses of Runtme Verfaton (RV) n software development and analyss, as refleted, for example, by abundant approahes proposed reently ([14, 10, 4, 19, 1, 6, 7, 13, 8, 20] among others). Runtme verfaton systems typally take a possbly unsafe system as nput together wth spefatons of events and desred propertes, and yeld as output a modfed verson of the nput system whh heks the nput spefaton durng ts exeuton, possbly trggerng reovery ode f spefatons are volated. Fg. 1 llustrates ths by means of an example usng the state-of-the-art montorng system JavaMOP [20, 15]. An event s an abstraton of an aton of nterest (method nvoaton, feld aess, et.) that ours durng program exeuton. A trae s a fnte sequene of events, and a property (spefaton) s a formal desrpton of a set of (desred or undesred) traes. The spefaton n Fg. 1 has three parameters (lne 1): m, and. These are bound to onrete objets provded by events at runtme. For example, the event geti defned on lnes 4 5 s fred when the terator() of an Iterable mplementaton s nvoked, and arres two onrete objets that and are bound to. The property, gven as an extended

2 1 Map_UnsafeIterator(Map m, Colleton, Iterator ) { 2 reaton event getc after(map m) returnng(colleton ) : 3 ( all(set Map+.keySet()) ) && target(m) {} 4 event geti after(colleton ) returnng(iterator ) : 5 all(iterator Iterable+.terator()) && target() {} 6 event modfym before(map m) : 7 ( all(* Map+.lear*(..)) ) && target(m) {} 8 event modfyc before(colleton ) : 9 ( all(* Colleton+.lear(..)) ) && target() {} 10 event usei before(iterator ) : 11 ( all(* Iterator.hasNext(..)) 12 all(* Iterator.next(..)) ) && target() {} ere : getc (modfym modfyc)* geti usei* 15 (modfym modfyc)+ usei 16 { prnt("map was modfed whle beng terated"); } 18 } Fg. 1. A JavaMOP spefaton Map_UnsafeIterator. regular expresson (ERE) (lnes 14 15), spefes bad behavors. handler (lne 17) says what to do when a trae mathes the pattern,.e., when a map s modfed whle beng terated. Handlers an ontan any ode, for example, error reovery ode. Despte the usefulness of runtme verfaton, we beleve there are a few hurdles that make developers or users relutant to use RV systems n prate: 1. It s not easy to wrte spefatons for RV systems to hek aganst; 2. Even f the spefatons are avalable, exstng RV systems usually nur large overhead when hekng multple propertes smultaneously n real world software. We next ntrodue bakground knowledge and elaborate on the ponts above n detal. Lak of spefatons: spefatons are key to runtme verfaton. However t s not easy to produe orret spefatons, as t requres a deep understandng of the underlyng system. There are several hand-wrtten spefatons shpped wth montorng systems, as well as a few spefatons produed by automated spefaton mnng approahes. These fae a few ommon problems: 1) Corretness. Most spefatons are wrtten n ad-ho ways wthout rgorous proedures, whh make ther orretness questonable; 2) Coverage. Most exstng spefatons only fous on lmted overage of the software system; 3) Reusablty. Most exstng spefatons are spef to the underlyng software system. They annot be appled to dfferent software systems. Hgh montorng overhead: produng a hghly relable system whle mantanng a low overhead was always a major onern of runtme verfaton. Most state-of-theart montorng systems employ parametrty to ensure orretness of the montored behavors. Coneptually, parametr montorng systems mantan a separate trae for eah parameter bndng and separately hek f eah trae mathes the pattern. As an example n Fg. 1, a usei event wth an Iterator objet 1 does not affet traes that orrespond to the parameter bndng where s bound to 2 whle m and an be bound to any objet, denoted as m,, 2. Instead of mantanng traes, an effent montorng system, suh as JavaMOP, atually uses a montor nstane for eah parameter bndng; when an event ours, t s dspathed to every montor nstane

3 m m m Map Montor Set of Montors Fg. 2. Indexng trees for Map_UnsafeIterator. whose parameter bndng ontans onrete objets arred by the event. If no suh nstanes exst and the event s a reaton event, a new montor nstane s reated. For fast lookup of montor nstanes, JavaMOP employs ndexng trees, whh presely return all the related montor nstanes. An ndexng tree s a mult-level map that, at eah level, ndexes eah onrete objet of the parameter bndng. For example, when a parameter bndng s m m1, 1, 1 (m s bound to m1, s bound to 1 and s bound to 1), one an retreve the related montor nstane by searhng for m1, 1 and 1 at eah level n the 3-level map, shown n the top-left of Fg. 2. However, not all parameters are always bound; e.g., geti n Fg. 1 does not arry m, whh makes t neffetve to retreve all the related nstanes usng ths map. To handle suh ase effently, another map, shown n the top-rght of Fg. 2, s onstruted as well. Unlke the 3-level map, where a leaf orresponds to one montor nstane, ths 2-level map has a set of nstanes at eah leaf, beause there an be multple parameter bndngs that bnd and to the same Colleton and Iterator. If an ndexng tree holds a strong referene (.e., an ordnary Java referene) to a onrete objet, ths objet beomes nelgble for garbage olleton, whh leads to a memory leak. To avod ths, ndexng trees store only weak referenes, whh enables the garbage olletor to relam the referents. Sne a weak referene gves ndaton that the referent has been relamed by returnng null, a montorng system an detet broken mappngs n the ndexng trees and lean them up. In addton to ndexng trees, an ndexng ahe that stores the prevously aessed montor nstane(s) s used [20]. Even wth the use of ndexng trees and weak referenes, the runtme overhead s stll large when montorng multple parametr propertes smultaneously. From a thorough proflng of JavaMOP, [16] dentfes the man remanng bottlenek to runtme performane s the exessve memory usage, whh s aused by the large sze/number of ndexng trees. Indeed, when there are hundreds of propertes beng montored and eah property has a few parameters, the total memory and runtme overhead assoated wth all the ndexng trees wll domnate the program exeuton tme. Ths urges us to desgn new tehnques to further redue the number and memory overhead of ndexng trees. In ths paper, we am to address the above problems n order to make runtme verfaton pratal. Ths paper makes the followng spef ontrbutons:

4 Comprehensve formal Java API spefatons: We presented a omprehensve set (179 n total) of formal spefatons whh over four of the most wdely used pakages of the Java API. These spefatons were manually produed followng rgorous proedures, and are publly avalable [3]. Novel optmzaton tehnques: We re-mplemented JavaMOP, separatng ts montorng (as a new RV system, RV-Montor) from ts event frng (as an AspetJ front-end of RV-Montor). We employ a seres of tehnques to make RV-Montor more effent, espeally when montorng multple propertes smultaneously. Our tehnques an also be easly adopted by other RV systems, suh as MOPBox. Large sale evaluaton: We montored all the Java API spefatons we produed wth RV-Montor aganst the DaCapo [5] benhmark sute. Results show that RV- Montor s apable of fndng property volatons (potental bugs) n DaCapo whle montorng hundreds of propertes at the same tme. Our omparson also shows that RV-Montor runs substantally faster than other montorng systems. 2 Formal Spefatons from the Java API Spefaton Almost all Java programs make use of the Java API. Moreover, msuses of the Java API an result n runtme errors or nondetermnst behavor. It s therefore mportant to obey the usage protools of the Java API. In ths seton we desrbe our formalzaton of several Java API spefatons. All our tools and spefatons are publly avalable and an be used and verfed by any RV systems [3]. 2.1 Java API Spefaton A Java platform, suh as Java Platform Standard Edton 6, mplements varous lbrares that are ommonly needed to mplement applatons, suh as data strutures (e.g., Lst and HashMap), and I/O funtons (e.g., FleInputStream and FleOutputStream). Besdes suh lbrary mplementatons, a Java platform provdes the API Spefaton, whh desrbes all aspets of the behavor of eah method on whh user s programs may rely. For example, the API spefaton for the Map nterfae states: If the map s modfed whle an teraton over the set s n progress the results of the teraton are undefned. We beleve the Java API doumentaton s a good soure for formal spefatons. Frst of all, the Java API s ommonly used by vrtually all Java programs. Thus the use protools of those API methods should always be obeyed. Seond, the Java API spefaton s well wrtten, learly statng what the orret/norret usage of API methods s. For example, n the above Map spefaton t an be nferred that any methods that modfy the Map (put(), remove(), et.) should not be alled before the teraton of the Map s done. Volatons of suh propertes usually ndate a bug or bad programmng prate. We have arefully read and analyzed the omplete Java API doumentaton for four of the most ommonly used pakages: java.o, java.lang, java.net and java.utl. We employed a mult-step approah to formalzng all the spefatons n these four pakages and produed 179 formal spefatons. Next we desrbe our approah.

5 2.2 Separatng Spefaton-mplyng Text Although the API spefaton nludes ontrats (Se. 2.1), sentenes referrng to dfferent purposes are mngled n API spefatons. As a preparaton step, we marked spefaton-mplyng text versus desrptve text usng a speally defned javado tag. Ths faltates the revew of relevant parts when wrtng spefatons (Se. 2.3). Whle t mght seem obvous to dstngush between desrptve and spefatonmplyng text, there are unlear ases. One ase s the desrpton of ondtons nvolvng external envronments. Here s an example for the FleOutputStream onstrutor: If the fle exsts but s a dretory rather than a regular fle, does not exst but annot be reated, or annot be opened for any other reason, then a FleNot- FoundExepton s thrown. We deded to not formalze ths property, beause the state of the fle system an externally and dynamally hange wthout notfyng the RV system and, onsequently, t s mpossble to relably hek whether a fle an be reated or opened. It s dffult to formalze a spef set of rules that resolves all of the unlear ases, but the rule of thumb was that a hunk of text s spefaton-mplyng only f a desrable or undesrable behavor s apparent and t s defned n terms of noteable events, suh as lass loadngs, method nvoatons and feld aesses. 2.3 Wrtng Formal Spefatons For eah hunk of text that we marked as spefaton-mplyng, we wrote JavaMOP spefatons. We hose the JavaMOP spefaton syntax beause JavaMOP was the most effent and expressve, n that t allows us to wrte a property n varous formalsms. As shown n Fg. 1, a typal spefaton ontans three parts: a set of event defntons, a desrable or undesrable behavoral pattern (.e., property), and a handler for volatons. An event n our spefatons s mostly a method nvoaton, a feld aess, an end of an exeuton, or a onstruton of an objet. Dependng on the pattern, we hose the most ntutve formalsm, among an ERE, a fnte-state mahne (FSM) or a lnear temporal log (LTL) formula, for expressng a property. Our handlers smply output a warnng message n ase of a volaton. There are some ases where we ntentonally dd not wrte formal spefatons. Below we explan suh ases wth ratonales. Non-montorable behavors We onsdered only runtme-montorable spefatons beause we ntended to use a RV system; e.g., onsder the followng: The mplementor must also ensure that the relaton s transtve: (x.ompareto(y) > 0 && y.ompareto(z) > 0) mples x.ompareto(z) > 0. Although ths mples a ertan behavor, hekng whether t holds s nfeasble at runtme. Not havng a means of desrbng and hekng t, we dd not wrte formal spefatons for suh ases.

6 Already enfored behavors We dd not wrte formal spefatons f the desrable behavor s already enfored by omplers. For example, onsder the followng, whh states the requrement of InputStream s sublasses: Applatons that need to defne a sublass of InputStream must always provde a method that returns the next byte of nput. Java omplers enfore the requrement beause read(), the method mpled by the omment, s an abstrat method. Suh guarantee obvates the need for addtonal runtme spefatons; thus, we dd not formalze suh ases. Internal behavors When all the events n the mpled spefaton are never exposed to lents (e.g., they are prvate method nvoatons or feld aesses), we dd not wrte spefatons. For example, onsder the API spefaton for GregoranCalendar. getyearoffsetinmlls(): Ths Calendar objet must have been normalzed. A spefaton on ths s feasble but useless beause user s programs annot nvoke t anyway, due to the aess ontrol t s defned as prvate. We dd not wrte spefatons n suh ases beause there s no beneft from a user s perspetve. 2.4 Classfyng Formal Spefatons Not all volatons are equally mportant; volatons of some spefatons are harmless n ertan applatons and users may want to suppress them. To faltate suh flterng, we lassfed our spefatons nto three groups: suggeston, warnng and error. We use suggeston f a volaton s merely a bad prate. If a volaton s not neessarly erroneous but potentally wrong, we use warnng. We use the last group, error, f a volaton ndates an error. 3 Salable Runtme Verfaton Many RV systems are able to montor one property effently. However, to make runtme verfaton pratal, an RV system should be able to effently montor multple propertes smultaneously. In ths seton we present our optmzatons to redue runtme overhead. Sne JavaMOP was the most effent RV system when ths paper was wrtten, we bult upon several of ts optmzatons, espeally upon ts ndexng tree dea. Most of our tehnques fous on makng the ndexng tree aesses faster, whh redue the montorng overhead when multple propertes and montor nstanes are present. Note that the dea of usng ndexng trees and ahes for managng multple montor nstanes s ommonly used among other RV systems. For example, MOPBox uses JavaMOP s ndexng algorthm to map varable bndngs to montors. We beleve our mprovement of JavaMOP, suh as the mergng of ndexng trees, an also be dretly appled to other RV systems and an ndretly nspre other systems, suh as the more general RuleR or Eagle [4, 9]. Besdes, some other systems propose stat analyss on top of the JavaMOP tehnque; those systems wll also beneft from our tehnques.

7 3.1 Global Weak Referene Table Even montorng a sngle spefaton auses multple weak referenes for a sngle objet n JavaMOP beause t onstruts multple ndexng trees for a spefaton and eah ndexng tree stores ts own opy of the weak referene. From the spefaton shown n Fg. 1, for example, JavaMOP an reate four weak referenes for a sngle Colleton objet: one at the seond level of the top-left tree, another at the frst level of the bottom-left tree, and so forth n Fg. 2. When multple spefatons are montored, redundany s more severe beause JavaMOP reates separate ndexng trees for eah spefaton. That s, ndexng trees for another spefaton reate ther own (possbly multple) weak referenes for the same objet, n addton to those already reated for the frst spefaton. For example, both Map_UnsafeIterator and Iterator_HasNext t states that Iterator.next() should not be alled wthout hekng for the exstene of the next element reate ther own weak referenes to the same Iterator objets. We avod suh ntra/nter-spefaton redundany by employng a global weak referene table (GWRT) that stores only one weak referene for eah dstnt objet. As n strng nternng, however, keepng only one opy requres extra omputaton for reaton t s neessary to hek whether or not a weak referene for the gven objet has been prevously reated, n order to guarantee unqueness of weak referene. To elmnate redundany wth lttle performane ompromse, we use a separate GWRT for eah parameter type; e.g., three GWRTs wll be used by Map_UnsafeIterator. A GWRT funtons as a dtonary where a onrete objet s used as a key and a weak referene s used as a value. That s, ths table allows one to retreve the weak referene assoated wth the onrete objet. Sne ths table s globally shared among all spefatons and a key an appear at most one n t as n other typal dtonares, t allows only a sngle opy of weak referene for eah objet. Whle the GWRT may seem to be the same as any other hashtable wth weak referene values, ts nternal struture s further optmzed to redue memory onsumpton. Frst, t does not hold a strong referene to a onrete objet, whh would ause memory leaks. More mportantly, ths table does not atually store pars of objets and ther weak referenes; only weak referenes are stored beause one an retreve the objet from the weak referene by nvokng WeakReferene.get(). Fast lookup, aheved usng a hash funton, s stll vable by loatng the slot based on the hash value of the objet and searhng weak referenes n the slot. We further redue the memory overhead nurred from GWRTs by mergng them wth the frst levels of ndexng trees. The frst level of an ndexng tree holds all the weak referenes to the objets of a ertan type, as does the GWRT for that type. Therefore, the GWRT an be embedded n the frst level of the ndexng tree, wthout losng any weak referene, ntrodung wasted spae, or ompromsng fast lookup. The GWRT for Map, for example, s embedded n the frst level of the top-left tree n Fg. 2, whh holds entres for all the weak referenes, one per eah Map objet. Embeddng elmnates all memory overhead of sad GWRT. Most GWRTs an be embedded beause RV-Montor needs to reate ndexng trees wth many dfferent types. For example, RV-Montor reates at least one ndexng tree for eah parameter type for Map_UnsafeIterator, as shown n Fg. 2, whh allows the three GWRTs to be embedded. In our experments, all GWRTs

8 were embedded when the 137 spefatons from [18] were montored smultaneously. However, there are some theoretal ases where embeddng s not possble. Moreover, redung the number of nvoatons of System.denttyHashCode() s also benefal. Therefore, a sublass of the WeakReferene lass, Java s weak referene mplementaton, was reated n suh a way that the hash value of the referent s omputed at most one and then stored n a dedated feld by the onstrutor. Ths sublass s used n plae of the WeakReferene lass n both GWRTs and ndexng trees. Wth ths sublass, alls to System.denttyHashCode() an be replaed by a feld aess, whh s less expensve, at the ost of addng one nt feld to eah weak referene. Addng ths feld to eah weak referene does not ause a sgnfantly larger expendture of memory beause only a sngle weak referene s reated for eah dstnt objet thanks to GWRTs. Another beneft of GWRTs s that t s easer to hek f there exsts a montor nstane referrng to a gven objet. Beause a GWRT holds all the weak referenes of one type, the lak of the weak referene for an objet mples that none of spefatons have reated any montor nstane for that objet. For ths reason, when handlng a non-reaton event (see Seton 1), we an safely skp the other steps of montorng f there are no weak referenes for every parameter arred by the event. In ontrast, JavaMOP, where ndexng trees are separately reated for eah spefaton, must hek ndexng trees for eah spefaton that shares a gven event. 3.2 Cahes for the Global Weak Referene Table GWRTs are frequently aessed beause weak referenes for eah parameter objet bound n a gven event must be retreved. As a result, optmzng ths data struture has a large mpat on performane. We frst redue the number of aesses to a GWRT by extendng eah ndexng ahe. Temporal loalty s often exhbted n the aess patterns of objets related to montorng a gven spefaton. For example, onsder a spefaton that formalzes a resoure management pattern on the Reader lass: a reader an perform the read operaton untl t s losed. Whle montorng ths spefaton, t s lkely that there would be a long sequene of read() alls on a partular Reader objet untl lose() s alled. To utlze suh loalty, JavaMOP already has an ndexng ahe that stores the reently aessed montor nstane(s). Whle ths ahe allows for quk retreval of the montor nstane(s), weak referenes for all of the parameter objets assoated wth those nstane(s) are often neessary durng the proess of updatng sad nstane(s). Quk retreval of a montor nstane followed by retrevng weak referenes for every sngle parameter objet from the GWRT would be ontrary to the whole purpose of the ndexng ahe. We therefore extended the ndexng ahe n suh a way that, when t ahes montor nstane(s), t also ahes the weak referenes that pont to the parameter objets bound by those montor nstane(s). As a result, when an event arres the reently handled objets, the ndexng ahe an return not only the relevant montor nstane(s) but also weak referenes, wthout the need to onsult the GWRT for eah bound parameter objet. Seond, we add to eah GWRT a ahe, alled the GWRTCahe, whh s used to store reently requested weak referenes. As stated prevously, multple spefatons ommonly share event defntons, thus a sngle method nvoaton an trgger multple

9 m Map Montor Set of Montors Fg. 3. Indexng trees for Map_UnsafeIterator after ombnng. events, one per eah spefaton. For example, an nvoaton of Reader.read() trggers not only an event of the aforementoned resoure management spefaton but also the mark/reset spefaton: a reader s mark poston, set by Reader.mark(), s nvaldated after readng the spefed number of bytes. Wthout the GWRTCahe, when read() s nvoked on an unpreedented Reader objet, ndexng ahes for both spefatons wll mss whle handlng the two events and both ahes would query the GWRT for the Reader objet, n order to reate or retreve the weak referene (whhever request ours frst wll reate the weak referene). Usng the GWRTCahe, however, the frst mss auses ths ahe to hold the reated Reader objet and, onsequently, a hash lookup n the GWRT for eah subsequent request s replaed by a lookup n ths ahe, whh s faster. 3.3 Combnng Indexng Trees As we mentoned n Seton 1, montorng a real program auses mllons of montor nstanes and, onsequently, the sze of ndexng trees beomes large beause they store referenes to montor nstanes at leaves. In the presene of hundreds of spefatons, the sze of ndexng trees s lkely to beome even larger, ausng exessve memory overhead. To mtgate suh overhead, we ombne ndexng trees orgnatng from the same spefaton, f they share the same prefx; e.g., ndexng trees for Colleton, Iterator and Colleton are ombned, but trees for Map, Colleton, Iterator and Colleton, Iterator are not. Consder the ndexng trees for Colleton, Iterator and Colleton, shown n Fg. 2. Sne both trees ndex all the Colleton objets at the frst level, ther frst level maps ontan the exatly same keys (.e., weak referenes); the only dfferene s that, by the same key, a seond level map s mapped to n the frst tree, whereas a set of montor nstanes s mapped to n the seond tree. We ombned these two trees by allowng a seond level map and set of montor nstanes par to be mapped to by a key, as shown at the bottom tree of Fg. 3. Choosng whh part of the par an be done dependng on whether or not a Colleton objet s arred by an event. Three ndexng trees, shown at the top of Fg. 2, are also ombned nto one tree, shown at the left top of Fg. 3, smlarly. By ombnng ompatble trees, only three trees, out of sx, reman as shown n Fg. 3.

10 3.4 Fne-graned Lokng JavaMOP uses one global lok throughout all the operatons for handlng an event, whh may nvolve multple GWRT aesses and ndexng tree lookups. Ths an sgnfantly hnder onurrent exeuton n the presene of multple spefatons beause t s lkely that more events our smultaneously. To redue ths hndrane, we remove the global lok, and nstead use fne-graned lokng gven that eah of the GWRTs and ndexng trees s ndependently aessed. Frst, we synhronze eah GWRT separately beause GWRTs do not nterfere wth eah other. Ths enables multple threads to run onurrently, unless they handle the same type of parameters reall that one GWRT s reated for eah parameter type. Seond, we synhronze eah level of an ndexng tree separately. For example, onsder a getc event n Fg. 1, whh brngs two parameters: m and. When lookng up the left tree n Fg. 3 for retrevng montor nstanes orrespondng to the provded parameters, we frst aqure a lok orrespondng to the frst level. On retrevng the node at the seond level aordng to the objet bound to m, we mmedately release the lok. Ths way, another request on the frst level of ths tree an be served wth relatvely short delay. In order to promote onurrent exeuton further, we moved to thread-loal storage (TLS) two ahes: the ahe n eah ndexng tree and the GWRTCahe n eah GWRT (Seton 3.2). Ths s based on the observaton that most objets bound to parameters are solely used n a sngle thread. Wth ths hange, f a request s served by the ahe, no synhronzaton s performed, at the ost of addng a few ahe entres to eah GWRT and eah level of ndexng trees, per thread. 4 Implementaton and Evaluaton In ths seton we desrbe our mplementaton of RV-Montor and evaluate the orretness of all our formal spefatons as well as the performane of RV-Montor. 4.1 Implementaton We have developed RV-Montor n suh a way that t an be used as a unversal platform for buldng varous montorng systems. To aheve ths, we re-mplemented JavaMOP and separated event montorng funtonalty from event frng funtonalty. RV-Montor mplements only the ommon and ndspensable montorng funtonalty-suh as lstenng to events and trggerng handlers when a pattern mathes. Usng RV-Montor, one an generate a montorng lbrary that exposes an nterfae more spefally, a set of methods, eah of whh orresponds to an event defnton for lstenng to events. One an then montor a program by notfyng the lbrary of events;.e., nsertng nvoatons of suh methods ether manually, or systematally usng nstrumentaton tools. Havng ths separate system enables one to wrte a spefaton n terms of vrtually any event, from method nvoaton to montor (n the ontext of lokng) wat, usng dverse means, suh as byteode nstrumentaton or JVMTI. No matter how events are fred, montorng an be effent beause t s entrely delegated to the ode generated by RV-Montor. We also re-mplemented the event frng funtonalty of JavaMOP as an AspetJ front end of RV-Montor. All the orgnal JavaMOP spefatons an be translated nto

11 smple AspetJ ode. Insde those AspetJ ode, RV-Montor event montorng lbrary methods are now alled. The AspetJ front end of RV-Montor enables us to easly ompare the performane of RV-Montor wth JavaMOP. 4.2 Corretness of Formal Spefatons The orretness of spefatons s vtal, beause norret spefatons would mss bugs or gve out false alarms. However there s no slver bullet to guarantee the orretness of spefatons other than areful nspeton and thorough testng. All of our spefatons were revewed by at least two authors who are knowledgeable about Java. In addton to peer revew, we wrote small defetve Java programs and tested f the formal spefaton an reveal volatons. We have wrtten more than 100 (publly avalable) small programs n total, and all the tests revealed the nserted defets, whh brngs evdene that these spefatons are apable of detetng errors. All our spefatons and tests an be found at [3]. 4.3 Runtme Overhead Comparson wth JavaMOP To measure the effeny of RV-Montor, we ompare the overhead of RV-Montor (wth the AspetJ front end) wth that of the latest verson of JavaMOP (v2.3). We hose JavaMOP beause, at the tme of wrtng ths paper, t was the most effent system, to the best of our knowledge. All our experments were onduted on Sun Java SE 6 (buld 1.6.0_35) under a system wth a 3.5 GHz Intel Core 7 and 32GB of memory runnng Lnux 3.2. We olleted the exeuton tme for eah benhmark, as shown n Table 1, under a steady state usng DaCapo 9.12 s -onverge opton. We took the average tme of the last fve runs. The Orgnal olumn shows the unmontored runs of benhmarks, whereas the other olumns show the montored runs wth all the spefatons. The overhead olumns ndate the perentage overhead. In average, the perentage overhead of RV-Montor s less than half of that of JavaMOP. In partular, mprovement was sgnfant for four benhmarks that aused exessve overheads under JavaMOP; avrora, lusearh, pmd and xalan under JavaMOP respetvely showed 150%, 654%, 797% and 2,968% overheads, but they aused only 22%, 420%, 261% and 97% overheads under RV-Montor. On one benhmark, fop, RV- Montor was noteably worse. Ths s beause fop s sngle-threaded, and RV-Montor has been geared toward nreasng onurreny, suh as fne-graned lokng and threadloal storage (TLS) ahes (Seton 3.4), whh are benefal to mult-threaded programs but may end up addng extra omputaton when onurreny s not needed. Note that n xalan, RV-Montor performs muh better than JavaMOP. The reason s that the program tself s omputaton ntensve. Many montor nstanes are reated and used only one, nurrng a large memory overhead wth JavaMOP. Therefore, RV-Montor s able to greatly redue the overhead by mergng ndexng trees and usng global ahes. If we exlude ths program, the average overhead of RV-Montor wll be 143%, stll better than JavaMOP wth 163%. We beleve the DaCapo benhmark has been oneved as a whole, to test varous aspets of the language, and establshed tself as the most sutable benhmark for runtme verfaton. Therefore, we thnk the huge overhead reduton ase shows a bg mprovement of our tehnque, not an solated ase.

12 Benhmark Orgnal JavaMOP RV-Montor MOPBox tme (s) tme (s) overhead tme (s) overhead tmes (s) overhead GCs avrora % % > batk % % > elpse % % > fop % % > h % % > jython % % > lundex % % % 199 lusearh % % > pmd % % > sunflow % % > tomat % % % 18 tradebeans % % % 44 tradesoap % % % 87 xalan ,968% % > Average 364% 140% - Table 1. Exeuton tme and the number of alloatons durng exeuton. It should be noted that the runtme overhead may look hgh, for some benhmarks, but those ases are extreme: here we montored programs that ntensvely use the Java API aganst lots of spefatons, and ths s ndeed a hallengng task. For example, most benhmarks emtted mllons of events; n partular, avrora, h2 and pmd emtted 32,804,400, 65,647,663 and 48,866,293 events, respetvely. In usual ases, however, one an expet moderate overhead. Comparson wth MOPBox MOPBox [21] s the most smlar RV system to RV- Montor. MOPBox [21] requres one to onstrut an FSM by settng alphabets, states, and transtons usng ts API and t does not support other formalsms. Beause of that, we tested only the most heavly used spefaton n DaCapo, Colleton_UnsafeIterator, whh warns f a olleton s modfed whle an terator s beng used. Table 1 shows the exeuton tme and the number of garbage olletons. Even though only one spefaton was montored, most benhmarks barely fnshed the frst teraton n 10 mnutes, exept three benhmarks that fre relatvely few events: tomat, tradebeans and tradesoap. One of reasons for suh exessve overhead was memory onsumpton: we noted that an exeuton under MOPBox trggered garbage olletons frequently as shown n Table 1, whereas that under RV-Montor trggered merely at most 5 garbage olletons, for eah teraton, n the worst ase. Overall, the result shows that the overhead of MOPBox wth one spefaton s by far more than that of RV-Montor wth 179 spefatons. There ould be many possble reasons for the bg dfferene. Frst, RV-Montor enables the montored program to dretly pass parameters to events (through generated AspetJ ode), whle MOPBox requres a VarableBndng objet to be reated for eah parameter. Ths may ause extra runtme overhead. Also, MOPBox uses the C + X algorthm [8], whh s less

13 Pakage o lang net utl Severty err. warn. sugg. err. warn. sugg. err. warn. sugg. err. warn. sugg. # of spes # of volated spes # of volatons Table 2. The number of spefatons, volated spefatons and volatons. effent than D X [8], used by RV-Montor. Moreover, MOPBox does not use all the effent data strutures for handlng montor nstanes, suh as ndexng trees. We beleve that our omparson wth MOPBox, as well as that wth JavaMOP, shows that engneerng a hgh-performane runtme verfaton system for parametr spefatons s a hghly hallengng task. 4.4 Bug Fndng Runtme montorng s apable of fndng bugs and bad prates even from wdely used benhmarks. Compared wth JavaMOP, RV-Montor s able to montor all the propertes smultaneously on large programs, makes t pratal for fndng bugs. Table 2 summarzes the number of spefatons, the number of volated spefatons, and the number of volatons for eah severty level from all the benhmarks of DaCapo When we ounted the number of volatons, we ounted all volatons aused by the same all ste as one. Here we fous on the volatons of spefatons that are marked as error and warnng among thousands of volatons. Reader_ManpulateAfterClose, whh warns f a read operaton s performed after a Reader objet has been losed, was volated by 13 out of 14 benhmarks of DaCapo. In fat, read() faled and the reader was mmedately losed, but read() was nvoked agan on that losed reader. The latter read() all was reahed beause the exepton rased at the frst falure was dsarded by a method and t returned as f there were no errors. Sne the Reader mplementaton rases IOExepton anyway and t s properly handled, ths volaton does not result n a notable falure. Nevertheless, we beleve t s a bad prate to rely on an exepton even when a volaton s predtable. ShutdownHook_LateRegster, whh warns f one regsters or unregsters a shutdown hook 4 after the Java Vrtual Mahne (JVM) s shutdown sequene has begun, was volated by h2 ths program attempted to unregster a shutdown hook. One may thnk that suh attempt would be safe as long as the resultng exepton s properly handled, but t s unsafe beause regstered hooks are started n unspefed order and, onsequently, the hook to be unregstered may have been already started. Colletons_SynhronzedColleton, whh warns f a synhronzed (thread-safe) olleton s aessed n an unsynhronzed manner, was volated by jython. Ths program reated a synhronzed olleton, usng Colletons.synhronzedLst(), but terated over the olleton wthout synhronzng on t, whh may result n nondetermnst behavor, aordng to the API spefaton. Besdes these volatons that mply notable problems, we also found many mnor yet nformatve volatons that stat analyss mght not be able to detet. One suh 4 Aordng to the API spefaton, a shutdown hook s an ntalzed but unstarted thread.

14 example s a volaton of Math_ContendedRandom, whh reommends one to reate a separate pseudorandom-number generator per thread for better performane f multple threads nvoke Math.random(). Another example s StrngBuffer_SngleThreadUsage, whh heks f a StrngBuffer objet s solely used by a sngle thread. To detet suh volatons wthout false postves, t s neessary to aurately ount how many threads aess an objet or a method, whh s mpossble for stat hekers n full generalty. 4.5 Ineffetual Approahes In ths seton, we dsuss some neffetual approahes that we have tred whle mprovng the salablty of parametr montorng. Although they turn out to be neffetual for parametr montorng, some of them mght be useful n dfferent settngs or they mght nspre new effetual deas. Combnng Indexng Trees between Spefatons As mentoned n Seton 3.3, we ombne ndexng trees only wthn eah spefaton. If we ombne ndexng trees for dfferent spefatons, as well, we an redue the number of ndexng trees even more. However, there s a lot of wasted spae n the ombned ndexng tree. For example, an ndexng tree A maps p 1 to m 1 and p 2 to m 2, and another ndexng tree B maps p 2 to m 3 and p 3 to m 4. The ombned ndexng tree of A and B wll map p 1 to (m 1, ), p 2 to (m 2, m 3 ), and p 3 to (, m 4 ). All empty spaes ndated by wll be wasted whle the ndexng trees A and B do not have empty spae. More memory overhead from wasted spae trggers more garbage olleton, slowng down the montorng. Enhaned Indexng Cahe The ndexng ahe provdes faster retreval of montors from the ndexng tree. There are several deas to mprove ts ht rato. We an apply a mult-entry ahe from Seton 3.2. Also, we an ahe not only montors but also lak thereof to save searhng the ndexng tree for nothng. However, sne the ndexng ahe already has a hgh ht rato, these enhanements do not mprove the rato enough to justfy ther overhead. Indexng Tree Cleanng by GWRT Sne we an manage all weak referenes for eah parameter type n one plae, the GWRT, we an let the GWRT lean up the ndexng trees. In ths way, we an remove garbage olleted parameter objets from all ndexng trees at one, elmnatng the need for partal leanups. Note that partal leanups ould our even when there s no garbage olleted parameter objet. We an also have a bt map n the weak referene to ndate to whh ndexng trees the referent belongs so that we need hek only the ndexng trees that atually ontan t. However, ths approah only moves leanup osts from ndexng trees to the GWRT, showng no mprovement. The leanup by the GWRT s more effetve beause t knows whh weak referenes should be removed. However, leanng up from outsde of the ndexng tree osts more beause we must loate the entry before we an remove t. Statsts-Based Indexng Tree Cleanng As mentoned prevously, partal leanups at ndexng trees an our even when there are no garbage olleted parameter objets. Sne we have the GWRT, we an keep statsts about garbage olleted parameter

15 objets and use t for dedng whether to trgger a partal leanup. However, n most ases, there are garbage olleted parameter objets. Savng a relatvely small number of partal leanups does not ompensate the overhead neessary. 5 Related Work Produng Spefatons Many automated spefaton mnng approahes [2, 12, 11, 17] has been proposed to redue the onsderable human effort n wrtng spefatons. Unfortunately, they tend to have a few problems n general. Frst, many of them requre or assume non-trval nputs to yeld a meanngful spefaton [2]. Seond, many approahes support only very partular propertes [12, 11]. Last but not least, most of them do not guarantee orretness of ther results [17]. Montorng Spefatons Many approahes have been proposed for runtme verfaton [10, 4, 19, 1, 6, 7, 13]. Most of them do not fous on redung the overhead of montorng multple propertes smultaneously. Reently Purandare et al. [22] presents the frst study of overhead arsng durng the smultaneous montorng of multple fnte-state mahne (FSM) spefatons. Ther approah redues runtme overhead by mergng FSM montors nto bgger montors, resultng n sgnfant runtme overhead reduton ( 50%) over JavaMOP v2.1; t s unknown how that ompares wth the more reent JavaMOP v2.3, whh added several optmzatons over v2.1 that mnmze the number of reated montors. Ther montor-mergng dea s orthogonal and omplementary to ours tehnques of mergng ndexng trees and usng global ahes, as our tehnques do not take nto aount the nternal struture of eah montor. Exept for that work, most exstng montorng systems are not apable of handlng multple spefatons smultaneously, or have rudmentary support, onsderng eah spefaton separately. Sne eah spefaton s ndvdually handled, the runtme overhead for runnng them smultaneously s lkely to be at least the summaton of the overheads of runnng eah n solaton. In fat, our prelmnary experment wth JavaMOP (v2.3) showed that the overhead of montorng several spefatons smultaneously s hgher than the summaton of the overheads of montorng eah n solaton, probably beause more memory pressure and less ahe ht. 6 Conluson Runtme verfaton has not been wdely adopted by developers and users, manly beause of lak of usable spefatons and large runtme montorng overhead. In ths paper we have presented a set of 179 formal spefatons overng most ommonly used Java pakages. We have also employed a seres of tehnques to derease runtme overhead of montorng multple propertes smultaneously, whh resulted n a new montorng system, RV-Montor. Results showed that: 1) our spefatons help fndng bugs or bad prate odng n real world benhmarks; 2) our tehnques make RV- Montor runs sgnfantly faster than other state-of-the-art RV systems. Aknowledgement: the work presented n ths paper was supported n part by the Boeng grant on Formal Analyss Tools for Cyber Seurty , the NSF grants CCF , CCF and CCF , and the DARPA grant under agreement number FA C-0284.

16 Referenes [1] C. Allan, P. Avgustnov, A. S. Chrstensen, L. J. Hendren, S. Kuzns, O. Lhoták, O. de Moor, D. Seren, G. Sttampalam, and J. Tbble. Addng trae mathng wth free varables to AspetJ. In OOPSLA, [2] G. Ammons, R. Bodík, and J. R. Larus. Mnng spefatons. In POPL, [3] Annotated Java API Spefatons. annotated-java-ap/. [4] H. Barrnger, D. Rydeheard, and K. Havelund. Rule systems for run-tme montorng: from EAGLE to RULER. J. Log Computaton, November [5] S. M. Blakburn, R. Garner, C. Hoffman, A. M. Khan, K. S. MKnley, R. Bentzur, A. Dwan, D. Fenberg, D. Frampton, S. Z. Guyer, M. Hrzel, A. Hoskng, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovć, T. VanDrunen, D. von Dnklage, and B. Wedermann. The DaCapo benhmarks: Java benhmarkng development and analyss. In OOPSLA, [6] E. Bodden. J-LO, a tool for runtme-hekng temporal assertons. Master s thess, RWTH Aahen Unversty, [7] S. Chaudhur and R. Alur. Instumentng C programs wth nested word montors. In SPIN, [8] F. Chen, P. Meredth, D. Jn, and G. Rosu. Effent formalsm-ndependent montorng of parametr propertes. In ASE, [9] M. d Amorm and K. Havelund. Event-based runtme verfaton of Java programs. SIGSOFT Softw. Eng. Notes, [10] D. Drusnsky. The Temporal Rover and the ATG Rover. In SPIN, [11] M. D. Ernst, A. Czesler, W. G. Grswold, and D. Notkn. Qukly detetng relevant program nvarants. In ICSE, [12] M. Gabel and Z. Su. Symbol mnng of temporal spefatons. In ICSE, [13] S. Goldsmth, R. O Callahan, and A. Aken. Relatonal queres over program traes. In OOPSLA, [14] K. Havelund and G. Roşu. Montorng Java programs wth Java PathExplorer. In RV, [15] D. Jn, P. O. Meredth, D. Grffth, and G. Roşu. Garbage olleton for montorng parametr propertes. In PLDI, [16] D. Jn, P. O. Meredth, and G. Roşu. Salable parametr runtme montorng. Tehnal Report Department of Computer Sene, Unversty of Illnos at Urbana-Champagn, [17] C. Lee, F. Chen, and G. Roşu. Mnng parametr spefatons. In ICSE, [18] C. Lee, D. Jn, P. O. Meredth, and G. Roşu. Towards ategorzng and formalzng the JDK API. Tehnal Report Department of Computer Sene, Unversty of Illnos at Urbana-Champagn, [19] M. Martn, V. B. Lvshts, and M. S. Lam. Fndng applaton errors and seurty flaws usng PQL: a program query language. In OOPSLA. ACM, [20] P. O. Meredth, D. Jn, D. Grffth, F. Chen, and G. Roşu. An overvew of the MOP runtme verfaton framework. STTT, [21] MOPBox. [22] R. Purandare, M. B. Dwyer, and S. G. Elbaum. Optmzng montorng of fnte state propertes through montor ompaton. In ISSTA, 2013.