PVS. Empirical Study of Usage and Performance of Java Collections. Diego Costa, 1 Artur Andrzejak, 1 Janos Seboek, 2 David Lo

Transcription

1 Empirical Study of Usage and Performance of Java Collections Diego Costa, Artur Andrzejak, Janos Seboek, 2 David Lo Heidelberg University, 2 Singapore Management University PVS

2 Empirical Study of Usage and Performance of Java Collections Diego Costa, Artur Andrzejak, Janos Seboek, 2 David Lo Heidelberg University, 2 Singapore Management University Published as: Diego Costa, Artur Andrzejak, Janos Seboek, and David Lo Empirical Study of Usage and Performance of Java Collections. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE '7), L'Aquila, Italy April 22-26, 207 Paper/slides available at: 2

3 Collections Collections are objects that groups multiple elements into a single unit. Use its metadata to track, access and manipulate its elements 2

4 Collections Collections are objects that groups multiple elements into a single unit. Use its metadata to track, access and manipulate its elements ArrayList<Integer> Object[] Integer 2

5 Collections Collections are objects that groups multiple elements into a single unit. Use its metadata to track, access and manipulate its elements ArrayList<Integer> Object[] Collection Overhead Elements Footprint Integer 2

6 Motivation Numerous studies have identified the inefficient use of collections as the main cause of runtime bloat Execution Time +7% Improv. Configuration of one HashMap instance [Liu et al. 2009] Memory Usage +54% Improv. Use of ArrayMaps instead of HashMaps [Ohad et al. 2009] Energy Consumption +00% Improv. Use of ArrayList instead of LinkedList [Jung et al. 206]

7 Collection Frameworks The Java Collection Framework offers a Standard implementation of the major collection abstractions Stable and reliable framework Easy to use However, there exist alternative libraries that provide a myriad of different implementations: Primitive Collections (IntArrayList) Immutable Collections Multimaps (Map<K, Collection>) Multisets (Map<K, int>) 4

8 Collection Frameworks The Java Collection Framework offers a Standard implementation of the major collection abstractions Stable and reliable framework Easy to use However, there exist alternative libraries that provide a myriad of different implementations: Primitive Collections (IntArrayList) Immutable Collections Multimaps (Map<K, Collection>) Multisets (Map<K, int>) Unsupported features 4

9 Collection Frameworks The Java Collection Framework offers a Standard implementation of the major collection abstractions Stable and reliable framework Easy to use However, there exist alternative libraries that provide a myriad of different implementations: Primitive Collections (IntArrayList) Immutable Collections Multimaps (Map<K, Collection>) Multisets (Map<K, int>) Unsupported features Simplified API 4

10 Analysis of Performance Impact of Alternative Collections Goal: Can we find alternatives to the Standard collection types which improve performance on time/memory?. Study on Collections Usage How often do programmers use alternative implementations? 2. Experimental Evaluation of popular Java Collection Libraries Are there better alternatives to the most commonly used Collections regarding performance? 5

11 Study on Usage of Collections Dataset We analyze the GitHub Java Corpus 0K projects 268 MLOC Static Analysis Use of Java Parser to extract variable declaration and allocation sites of Types with suffix: {List, Map, Set, Queue, Vector} Manually removed false-positives 6

12 Developers Rarely Use non-standard Collections HashMap 22% Maps 28% Sets 5% Lists 57% ArrayList 48% HashSet 0% 7

13 Developers Rarely Use non-standard Collections LinkedHashMap 2% HashMap 22% LinkedHashSet % Maps 28% Sets 5% Lists 57% ArrayList 48% HashSet 0% LinkedList 5% 8

14 Developers Rarely Use non-standard Collections TreeMap % LinkedHashMap 2% Other Maps % Top 4 represent 86% of all declared instantiations HashMap 22% Other Sets % LinkedHashSet TreeSet % HashSet 0% Maps 28% Sets 5% Other Lists 4% Lists 57% ArrayList 48% LinkedList 5% Non-Standard collections are declared <4% Evaluate alternatives to ArrayList, HashMap, HashSet and LinkedList 9

15 Commonly Used Element Data Types 00% 80% 60% 40% 20% 0% Top50 Lists Sets Map Keys Map Values Lists Sets Map Keys Map Values String Numeric Collections Other From the categorized data types: Strings are the most commonly held data type, followed by Numeric 0

16 Commonly Used Element Data Types 00% 80% 60% 40% 20% 0% Top50 Lists Sets Map Keys Map Values Lists Sets Map Keys Map Values String Numeric Collections Other From the categorized data types: Strings are the most commonly held data type, followed by Numeric Evaluate collections holding Strings, Integer and Long 0

17 Initial Capacity is Rarely Specified 00% Top50 80% 60% 40% 20% Specified Default Copy 0% ArrayList HashMap HashSet

18 Initial Capacity is Rarely Specified 00% Top50 80% 60% 40% 20% Specified Default Copy Evaluate collections with default Initial Capacity 0% ArrayList HashMap HashSet

19 Superior Alternatives Superior Alternative: a Non-Standard implementation that can outperform a Standard counterpart in terms of execution time and/or memory consumption. Can we find a superior alternative to the most commonly used collection types? 2

20 Experimental Study on Java Collections We selected 6 alternative libraries: Repository Popularity (GitHub) Appearance in previous partial benchmarks Libraries Version JCF Compatible Available at Trove.0. yes trove.starlight-systems.com Guava 8.0 yes /google/guava GSCollections yes /goldmansachs/gs-collections HPPC 0.7. no github /carrotsearch/hppc Fastutil yes /vigna/fastutil Koloboke yes /leventov/koloboke

21 Experimental Study on Java Collections Seven typical scenarios evaluated populate, iterate, contains, get, add, remove, copy Collections holding from 00 to million elements Alternatives to the most commonly used collections JDK.8.0_65 ArrayList, HashMap, HashSet and LinkedList Object collection alternatives Primitive collection alternatives 4

22 CollectionsBench Suite We create a benchmark suite: CollectionsBench Open Java Microbenchmark public void setup() { fulllist = this.createnewlist(); fulllist.addall(values); // Randomly } public void iterate() { for (T element : fulllist) { blackhole.consume(element); // Blackholes avoid dead-code // optimization } } 5

23 CollectionsBench Suite We create a benchmark suite: public void setup() { fulllist = this.createnewlist(); fulllist.addall(values); // Randomly } public void iterate() { for (T element : fulllist) { blackhole.consume(element); // Blackholes avoid dead-code // optimization } Only the instantiation is needed for each collection type* return new ArrayList<T>(); return new LinkedList<T>(); return new FastList<T>(); Here we measure: Execution time (ns) Collection Overhead (allocation) GC Profiler } 6

24 Experimental Planning To accomplish a steady state performance public void iterate() 5s Iteration 7

25 Experimental Planning To accomplish a steady state performance evaluation: public void iterate() Warmup 0x Iteration 5s Measured 0x Iteration Iteration 7

26 Experimental Planning To accomplish a steady state performance evaluation: Fork Fork public void iterate() Warmup 0x Iteration 0x Iteration 5s Measured 0x Iteration 0x Iteration Iteration 7

27 Experimental Planning To accomplish a steady state performance evaluation: Fork Fork public void iterate() Warmup 0x Iteration 0x Iteration 5s Measured 0x Iteration 0x Iteration Iteration AVG Time ± CI (99%) AVG Memory allocated ± CI (99%) 7

28 Reporting Speedup/Slowdown We present the results of alternatives normalized to the Standard implementation performances. Means with overlapping CI are set to zero We use the following speedup/slowdown definitions: S = T std T alt, if T std > T alt T alt T std, otherwise 8

29 2 2 Reporting Speedup/Slowdown We present the results of alternatives normalized to the Standard implementation performances. Means with overlapping CI are set to zero We use the following speedup/slowdown definitions: S = T std T alt, if T std > T alt T alt T std, otherwise S = {0,.5,.9, 2, 7.8} S = {0,.5,.9, 2, 7.8} Heat map 7 7 Overlapping Confidence Intervals 9

30 Reporting Memory Overhead For the memory comparison we present the collection overhead reduction per element (with compressed object pointers) 20

31 Reporting Memory Overhead For the memory comparison we present the collection overhead reduction per element (with compressed object pointers) For instance Collection X: 00 bytes per element Collection Y: 0 bytes per element Evaluated on the copy scenario 20

32 Reporting Memory Overhead For the memory comparison we present the collection overhead reduction per element (with compressed object pointers) For instance Collection X: 00 bytes per element Collection Y: 0 bytes per element Evaluated on the copy scenario 90% Reduction of Collection Overhead 20

33 * * * * Superior Alternatives: LinkedList LinkedList was outperformed by every ArrayList alternative JCF LinkedList$Entry consumes 24 bytes Standard JCF FU 2 Fastutil 2 GSCollec. 2 HPPC 2 2 contains add get remove copy * * * * * * * * * * * * * * * * 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 84% Reduction of Collection Overhead 2

34 * * * * Superior Alternatives: LinkedList LinkedList was outperformed by every ArrayList alternative JCF LinkedList$Entry consumes 24 bytes Standard JCF FU 2 Fastutil 2 GSCollec. 2 HPPC 2 2 contains add get remove copy * * * * * * * * * * * * * * * * 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M Asymptotic disadvantage 84% Reduction of Collection Overhead 2

35 * * * * Superior Alternatives: LinkedList LinkedList was outperformed by every ArrayList alternative JCF LinkedList$Entry consumes 24 bytes Standard JCF FU 2 Fastutil 2 GSCollec. 2 HPPC 2 2 contains add get remove copy * * * * * * * * * * * * * * * * 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M Asymptotic disadvantage Asymptotic advantage public boolean remove(object o) 84% Reduction of Collection Overhead 2

36 Superior Alternatives: ArrayList GSCollections provides a superior alternative Faster when populating the list (no time penalty) Fastutil FU GSCollec. HPPC populate iterate copy K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M No memory difference 22

37 Superior Alternatives: ArrayList GSCollections provides a superior alternative Faster when populating the list (no time penalty) Fastutil FU GSCollec. HPPC populate iterate copy K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M No memory difference Distinct Array copy calls Std: Arrays.copyOf(); Alt: System.arraycopy(); 22

38 Superior Alternatives: ArrayList GSCollections provides a superior alternative Faster when populating the list (no time penalty) Fastutil FU GSCollec. HPPC populate iterate copy K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M No memory difference Distinct Array copy calls Std: Arrays.copyOf(); Alt: System.arraycopy(); HPPC adds each element instead of copying the array 22

39 Superior Alternatives: HashSet Koloboke provides a superior alternative Fastutil is a solid 2 nd option Std HashSet$Node object consumes 2 bytes Fastutil FU 0 GSCollec. - Guava 0 HPPC Koloboke 0 Trove HP -2 Tr -2 populate iterate contains add remove copy 00 K 0K 00K M 00 K 0K 00 K M 00 K 0K 00 K M 00 K 0K 00 K M 00 K 0K 00 K M 00 K 0K 00 K M 75% Reduction of Collection Overhead 2

40 Superior Alternatives: HashSet Koloboke provides a superior alternative Fastutil is a solid 2 nd option Std HashSet$Node object consumes 2 bytes Fastutil FU 0 GSCollec. - Guava 0 HPPC Koloboke 0 Trove HP -2 Tr -2 populate iterate contains add remove copy 00 K 0K 00K M 00 K 0K 00 K M 00 K 0K 00 K M 00 K 0K 00 K M 00 K 0K 00 K M 00 K 0K 00 K M 75% Reduction of Collection Overhead Impact of memory efficiency on time Koloboke performs a memory copy of its HashTable 2

41 Superior Alternatives: HashMap Standard HashMap is a solid implementation No superior alternatives on time populate iterate contains remove copy Fastutil FU 0 GSCollec. - Guava HPPC Koloboke Trove HP -2 Ko K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M -2 24

42 Superior Alternatives: HashMap Standard HashMap is a solid implementation No superior alternatives on time Fastutil provides a superior alternative on memory consumption populate iterate contains remove copy Fastutil FU 0 GSCollec. - Guava HPPC Koloboke Trove HP -2 Ko K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M -2 55% Reduction of Collection Overhead Std. HashMap$Node object consumes 2 bytes 24

43 Object vs Primitive Collections Reducing collection footprint: overhead + element footprint Object collection ArrayList<> Primitive collection IntArrayList Object[] int[] 25

44 Object vs Primitive Collections Reducing collection footprint: overhead + element footprint Object collection ArrayList<> Smaller Collection Overhead Primitive collection IntArrayList Object[] int[] 25

45 Object vs Primitive Collections Reducing collection footprint: overhead + element footprint Object collection ArrayList<> Smaller Collection Overhead Primitive collection IntArrayList Object[] int[] Integer (6 bytes) int (4 bytes) 25

46 Object vs Primitive Collections Reducing collection footprint: overhead + element footprint Object collection ArrayList<> Smaller Collection Overhead Primitive collection IntArrayList Smaller Element Footprint Object[] int[] Integer (6 bytes) int (4 bytes) 25

47 Object vs Primitive Collections Reducing collection footprint: overhead + element footprint Object collection ArrayList<> Smaller Collection Overhead Primitive collection IntArrayList Smaller Element Footprint Object[] int[] Integer (6 bytes) int (4 bytes) 25

48 Object vs Primitive Collections Reducing collection footprint: overhead + element footprint Object collection ArrayList<> Primitive collection IntArrayList Smaller Collection Overhead Collection Footprint Reduction Smaller Element Footprint 80% Object[] int[] Integer (6 bytes) int (4 bytes) 25

49 Superior Alternatives: Primitive Collections We found superior alternatives to all three abstraction types Data Type: Integer to int Performance varies considerably from distinct libraries for multiple reasons For instance, ArrayList primitive implementations Fastutil GSCollec. HPPC Trove Libs populate iterate contains copy FU GS 4 HP Tr K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 26

50 Superior Alternatives: Primitive Collections We found superior alternatives to all three abstraction types Data Type: Integer to int Performance varies considerably from distinct libraries for multiple reasons For instance, ArrayList primitive implementations Fastutil GSCollec. HPPC Trove Libs populate iterate contains copy FU GS 4 HP Tr K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M for(object..) 5x slower than for(intprocedure) commonly implemented 26

51 Superior Alternatives: Primitive Collections We found superior alternatives to all three abstraction types Data Type: Integer to int Performance varies considerably from distinct libraries for multiple reasons For instance, ArrayList primitive implementations Fastutil GSCollec. HPPC Trove Libs populate iterate contains copy FU GS 4 HP Tr K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M 00 K 0K 00K M for(object..) 5x slower than for(intprocedure) commonly implemented 0% more branch misses for (int = offset; i-- > 0; ) 26

52 Superior Alternative: Primitive Collections GSCollections, Koloboke and Fastutil provide solid superior alternatives ArrayList HashMap HashSet Footprint Reduction Integer int 80% 76% 84% Time impact (M elements) +2x populate +4x contains 2x +2x remove +2x populate -2x iterate 2x +6x/+4x copy +4x populate +4x/+x iterate 2x +x/+7x copy 27

53 Summary There are performance opportunities on Alternative Collection Frameworks Time/memory improvement with moderate refactor effort We provide a Guideline to assist developers on: Identifying superior alternative implementations Which scenarios an alternative could lead to a substantial performance improvement CollectionsBench is open-source and available at GitLab 28

54 Thank you for your time Questions? 29