diagnosing memory leaks in large java applications

Transcription

1 diagnosing memory leaks in large java applications Nick Mitchell, Gary Sevitsky IBM J Watson Research Center Hawthorne, New York, USA July 24, 2003

2 our two challenges big applications framework-heavy: J2EE web transaction processing object-noisy: myriad language-level memory managers high-volume: problem resolution occurs in (or near) production small tools automated: small budget for expertise and diagnosis time lightweight: tool must slow down production by <5%

3 knowing the common datatypes doesn't help much live instances java/lang/string com/ibm/servlet/util/hashtableentry java/util/hashtable$entry org/apache/xerces/dom/extimpl org/apache/xerces/dom/attrimpl org/apache/xalan/xpath/xml/stringostringable org/apache/xerces/dom/documentimpl 52 (in addition allocation sites are expensive to know, and, by themselves, don't add too much)

4 factoring out objects via heap differencing is insufficient new live instances java/lang/string 9444 org/apache/xerces/dom/extimpl 6810 org/apache/xerces/dom/attrimpl 5290 java/util/hashtable$entry 3244 org/apache/xerces/dom/namednodemapimpl 2713 org/apache/xerces/dom/elementimpl org/apache/xerces/dom/documentimpl 27

5 an atom of a leak (every leaking operation leaks lots of these objects) you're leaking Strings String

6 a bowl leaks (every leaking operation leaks one of these data structures) extimpl (an XML document) arrays ElementImpl AttrImpl DocumentImpl a whole leaking bowl of spaghetti NamedNodeMapImpl Vector AttrImpl String

7 group into data structures? # constituents com/ibm/servlet/dynamicclassloader com/ibm/servlet/dynamicclassloader com/.../xslransform com/.../propertiesfactory elements of Finalizer queue org/apache/xalan/xslt/emplatelist owned by native code if we wait long enough, then the leaking data structure will float to the top; otherwise, noise effects dominate

8 anatomy of a leak diagnose memory problems by understanding how regions within the reference graph evolve over time you're leaking Strings there is a region of the XSL rendering engine that is leaking XML docs leaking into a Vector

9 categories of region evolution XML documents

10 categories of region evolution XML documents

11 categories of region evolution growing without bound (a leak)

12 categories of region evolution constituency doesn't change (a pool) constituency is in flux (hot cache) growing without bound (a leak) transient (transaction-local) shrinking (cold cache)

13 summary of approach 1. reconstitute regions from reference graph use dominance, reachability, age, datatype 2. construct short-list of candidate leaking regions use mixture model to rank regions e.g. from 1M live objects, identify handful of highly-ranked regions 3. online probing validates, categorizes candidates introduce region change detectors, short traversals of a reference graph to detect three kinds of region evolution achieves perturbation of <2%

14 what defines a region? XSLransform XSLEngineImpl Vector [Object

15 what defines a region? oops, a leak! DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object

16 what defines a region? DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object oops, a leak! DocumentImpl ElementImpl

17 what defines a region? DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object DocumentImpl ElementImpl the set of objects that is historically on the fringe between old and new objects

18 what distinguishes regions? two regions, distinguished by datatype on the historic fringe DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object DocumentImpl ElementImpl Keytable

19 what distinguishes regions? two regions, distinguished by owning container DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object DocumentImpl ElementImpl Stack [Object DocumentImpl

20 what distinguishes regions? two regions, distinguished by root data structure DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object DocumentImpl ElementImpl emplatecache Hashtable [Hashtable$Entry DocumentImpl

21 region keys L: leak root DocumentImpl ElementImpl XSLransform XSLEngineImpl Vector [Object O: owner proxy DocumentImpl ElementImpl C: change proxy for each object o, associate region key (L,O,C) set of unique keys is initial set of regions

22 finding L, good leak roots Define leak root metric, LRM=B M G, such that each leaking region has one o with LRM(o)>0, and few o's have LRM(o)>. B: eight binary rules to rule out impossible (be Sherleak Holmes!) narrow from a million to a hundred M: mixture model to rank the remaining narrow from a hundred to tens G: global fixpoint to ensure uniqueness narrow from tens to a handful of highly-ranked leak roots

23 C. ibid for objects not at the head of a single-entry region B: ruling out the impossible (using structural information) fraction of objects remaining # objects -A -A-B -A-B-C -A-B-C-D phone company 267, IDE 350, brokerage1 838, brokerage2 1,015, credit bureau 1,320, D D A. objects pointing to nothing aren't very interesting B. arrays themselves don't leak (but their dominating containers might) C D B A A D. objects which don't uniquely own anything also aren't interesting

24 B: ruling out the impossible (using temporal information) # objects remaining # objects -structural -E -E-F -E-F-G all told phone company 267,956 16, IDE 350,136 25, brokerage1 838,912 26, brokerage2 1,015,112 12, credit bureau 1,320, , E. ignore structures that contain only old or only new objects (e.g. an already-primed pool) F. structures that contain only new arrays are borinng (theres nothing new in those arrays) G. ignore structures with no overlap in datatypes over time H. structures that contain no objects on the fringe are safe to ignore

25 M: the mixture model no single property is entirely indicative instead, use gated mixure of them all instances newer on-stack on-fringe type overlap EventNotifier 377,276 34% % hreaddiscriminator 274,433 2% % FormProperties % % XslemplateCache 32 90% % VerifySignonScenario 18 11% % this application had two leaks e.g. gating function

26 LRM=B M G, on brokerage2 (before applying the bug fixes) # constituents size rank LRM(o) com/.../eventnotifier com/.../formproperties com/.../xslemplatecollection com/.../verifysignonscenario of the highest-ranked candidate roots, the top two indeed leak (from 1,015,112 live objects)

27 LRM=B M G, on brokerage2 (after applying the bug fixes) # constituents size rank LRM(o) com/websphere/alarmhread com/.../contextmodel com/websphere/poolmanager com/websphere/poolepm after fixing the leak, there are no stand-out candidates (from 779,540 live objects)

28 detecting evolution cheaply a region evolves when elements are added to removed from relinked within track evolution with region change detectors

29 detecting evolution cheaply a region evolves when elements are added to removed from relinked within track evolution with region change detectors a detector is a tuple (R,H,,B,P,M) R: region to detect changes in H,: the head and tail of a short, bounded-size traversal B: a sample bias P: a match precondition M: a mutator, updates the set of existing detectors

30 XML documents a region evolves

31 XML documents a region evolves

32 a region evolves growing without bound (a leak)

33 a region evolves along the historic fringe thus, the traversals are the short path which just spans the fringe H

34 a region evolves along the historic fringe H H H H thus, the traversals are the short path which just spans the fringe H

35 preconditions instanceof DocumentImpl for additions, the precondition P is the datatype of the change proxy

36 a precondition is matched

37 once a detector fires, we mutate the set of detectors M: for additions, we remove the firing addition detector increase B of detectors with same R add removal detector add new addition detectors

38 leakbot and its loops snapshot the heap sleep recontruct regions dotimes(n) { d=next detector if(d.b > rand()) { traverse from d.h to d. if(d.p(d.)) { d.m() } } sleep } rank regions generate region change detectors

39 diagnosing memory leaks in large java applications Nick Mitchell, Gary Sevitsky IBM J Watson Research Center Hawthorne, New York, USA July 24, 2003

40 checking data structure properties at runtime, cheaply and automatically Nick Mitchell, Gary Sevitsky IBM J Watson Research Center Hawthorne, New York, USA July 24, 2003

41 leakbot in action just after initial analysis over time... about one minute later a non-leaking region and another few minutes... is downgraded

42 Strategies for Dissecting histogram by datatype Leaks (and some problems with each) Strings are in every data structure histogram by allocation site Strings are allocated everywhere expensive (c.f. HPROF's 5-10x slowdown) visualize reference graph an application doesn't just leak objects, it leaks entire (and entirely ugly) data structures c.f. Jinsight, JProbe, Purify

43 Summary of the LeakBot echnique structure live objects into Co-evolving Regions portions of data structures which change in similar ways rank regions according to likelihood of problem only present to user those regions likely to leak, the suspects e.g. of Schwab's 1M live objects, leakbot identifies three suspects offline mode track evolution of regions as program runs treat structuring and ranking as initial estimates e.g. we might have caught a pool being populated it'll eventually plateau from them, derive a scheme for very lightweight probing verify whether initial estimates correct, and update ranking online mode

44 anatomy of a leak? String