Is It Possible to Code an Efficient Software Switch?

Transcription

1 Is It Possible to Code an Efficient Software Switch? Linearizing the Heap, and the Pervasive Use of Hardware Accelerators! Nick Mitchell along with Ioana Baldini, Peter Sweeney IBM T.J. Watson Research Center!! July 31, 2014

2 What Nick Does: Work with IBM customers and developers to make their applications run well the data herein is backed by thousands of real apps

3 Summer School Means Nick Learns At Least As Much From You, As You From Him

4 0. The Apps

5 Most of Our Data Comes from Somewhere Else Your Service Request My Service MongoDB Response Hadoop FPGA, GPU

6 Each Source Has its Own Wire Protocol Thrift Your Service Request HTTP BSON MongoDB My Service Response HTTP RPC Hadoop streaming! binary FPGA, GPU

7 Each Service Has its Operational Form Your Service Request My Service MongoDB Response Hadoop FPGA, GPU

8 Solutions Denied: (but worth learning from)! keep everything in memory code in a systems language

9 Summary Programmers are plumbers who write software switches ECOOP Languages offer poor support for efficient software switches, because of the distance between operational and wire forms Amdahl s Law strikes again, severely limiting the use of specialized computational circuitry What are our options for doing better?

10 1. Initial Thought Exercises

11 Implement an Efficient Map Entry HashMap a conventional chained hash map Key Value ~3 pointers per entry ~3-5 cache misses per GET copying required for RPC c.f. Entry Key Value

12 Implement This Without Copying Any Bits Into the Heap Entry HashMap HashMap Key Value Java Heap Entry Key Value Entry Key Value Entry Key Value update this value

13 Implement an Efficient String String 4 8 bytes of data bytes of non-data 8 33% efficiency characters

14 2. Programmers as Plumbers

15 For Example (a JAX-RS RESTful Response addtocart(@cookieparam( session ) Cookie item ) int quantity ) int quantity) {! ShoppingCart cart = new ShoppingCart(session); cart.add(itemcode, quantity); return Response.ok(cart.status()).cookie(cart.toCookie()).build(); }

16 Dispatch Response addtocart(@cookieparam( session ) Cookie item ) int quantity ) int quantity) {! ShoppingCart cart = new ShoppingCart(session); cart.add(itemcode, quantity); return Response.ok(cart.status()).cookie(cart.toCookie()).build(); }

17 Response addtocart(@cookieparam( session ) Cookie item ) int quantity ) int quantity) {! ShoppingCart cart = new ShoppingCart(session); cart.add(itemcode, quantity); return Response.ok(cart.status()).cookie(cart.toCookie()).build(); }

18 Deserialization into Application Response addtocart(@cookieparam( session ) Cookie item ) int quantity ) int quantity) {! ShoppingCart cart = new ShoppingCart(session); cart.add(itemcode, quantity); return Response.ok(cart.status()).cookie(cart.toCookie()).build(); }

19 Response addtocart(@cookieparam( session ) Cookie item ) int quantity ) int quantity) {! ShoppingCart cart = new ShoppingCart(session); cart.add(itemcode, quantity); return Response.ok(cart.status()).cookie(cart.toCookie()).build(); } (a lonely increment operation)

20 Serialize Application Data Structures Back into Response addtocart(@cookieparam( session ) Cookie item ) int quantity ) int quantity) {! ShoppingCart cart = new ShoppingCart(session); cart.add(itemcode, quantity); return Response.ok(cart.status()).cookie(cart.toCookie()).build(); }

21 Marshalling and Data Formats operational form wire form serialize transfer deserialize

22 3. The Profitability Threshold

23 Trade-offs At what point does externalizing a computation become more pain than it s worth? granularity of kernel cost of externalization Amdahl s Law accelerator speedup

24 Amdahl s Law Tuned Kernel Grows Smaller Offloading Computation Hurts Offloading Computation Helps Externalization More Expensive

25 Amdahl s Law make kernel 6.25% faster Externalization More Expensive free externalization, kernel is 100% of overall computation

26 Amdahl s Law make kernel 6.25% faster Externalization More Expensive externalization equivalent to 1% of overall computation, kernel is 100% of overall computation

27 Amdahl s Law make kernel 12.5% faster

28 Amdahl s Law make kernel 25% faster

29 Amdahl s Law make kernel 50% faster

30 Amdahl s Law make kernel 2x faster

31 Amdahl s Law make kernel 4x faster

32 Amdahl s Law make kernel 8x faster

33 Amdahl s Law make kernel 16x faster

34 Amdahl s Law make kernel 32x faster

35 4. The Costs of Externalization

36 Rough Measurements Object Churn Fractal Webs of Invocations 3000 temps to ingest one document 70 temps to turn a SOAP date (as bytes) into a Java Calendar 200k calls to ingest that document 2000 calls to turn an XML timecard into a Java data structure 6 temps to turn month into int (Sevitsky 2006)

37 Spending CPU Cycles 43% 32% 21% Serialization String Operations Intra-object Copying Application Logic Caching Encryption Reflection Data-driven Dispatching Connection Management

38 Three Kinds of Expenses 43% 32% 21% Serialization String Operations Intra-object Copying Optimizable Data Motion Reflection Data-driven Dispatching Connection Management Amortizable Application Logic Caching Encryption Vital

39 Does the Story Vary? All Data Sets 50% Optimizable Data Motion 31% Amortizable 19% Vital Analytics Apps Web Apps SPECjbb % 53% Vital SPECjEnterprise 49% Optimizable 34% Amortizable 13% trade 54% Amortizable BENCHMARKS

40 What the JIT Doesn t Catch (numbers relative to original w/o JIT optimizations) Copies Comparisons ALU Loads/Stores 0% 25% 50% 75% 100% original with JIT optimizations handtuned w/o JIT optimizations (Xu 2009)

41 Allocate-Use Separation Thread.run How much do we have to inline to have a chance of removing temps and copies?

42 Method Inlining % temps eliminated Date field base J9 inliner JOLT inliner Eclipse 0.4% 1.9% JPetStore on Spring 0.7% 2.5% TPCW on JBoss 0% 4.3% DaCapo 3.4% 13.3% Inlining is hard! Small benchmarks don t reflect difficulties (Shankar 2008)

43 Spending Memory COBOL vs Java 01 Student occurs Name PIC X(10) 02 Score PIC 999V99 class Student { String name; BigDecimal score; } Student[] data = new Student[40]; bytes 4492 bytes (Suganuma 2008)

44 Memory Overhead 0-10% 10-20% 20-30% 30-40% Overhead 40-50% 50-60% 60-70% 70-80% 80-90% % Number of Heap Snapshots

45 5. Alternative Marshalling Schemes

46 Options operational form operational form operational form optimize the translation wire form wire form transmit less wire form transmit less

47 Options operational form operational form operational form optimize the translation Google protobuf, Apache Thrift, Apache Avro, Scala Pickling???? wire form wire form wire form transmit less transmit less

48 protobuf, avro, thrift declaratively specify schema of data automatically generate marshallers struct Work { 1: i32 num1 = 0, 2: i32 num2, 3: Operation op, 4: optional string comment, } operational form optimize the translation wire form operational form wire form transmit less

49 Does it Matter? java-manual protostuff-manual protostuff kryo protobuf thrift-compact thrift avro hessian java-built-in scala/java-built-in relative time to serialize and deserialize an object (

50 Does it Matter? Java Kryo v1 Kryo v2 Scala Pickling Pickler Combinators Unsafe Pickler Combinators Time [ms] e+06 Number of Elements (Miller 2013)

51 Experiment: specjbb % Fraction of CPU Spent in Serialization 40% 30% 20% 10% original implementation change ~100 lines of code in the marshaller 0% Stack Sample Frequency (seconds)

52 6. Alternative Storage Schemes

53 Value Types and Structs struct Student { char[] name; int score; } pay the header cost once per record

54 Ref Poisoning struct Student { String name; int score; }. unless we use a reference type in which case, we re back to where we started

55 Arrays of Records struct Student { chart[] name; int score; } Student[] data = new Student[10]; pay the header cost once per array

56 Marshalling Arrays of Records struct Student { string name; decimal score; } Student[] data = new Student[10]; RPC pay the header cost once per array data[3].name.charat(5) operational form and wire form

57 c.f. Cobol both are fine until we need to store non-scalar data, i.e. variable-length data

58 Column Stores class Student { String name; BigDecimal score; } namestart nameend scorestart scoreend scorepool Student # 1 Student # 2 PROS maintains easily serializability allows for mix-and-match use of attributes Student # CONS added overhead (start and end pointers) unspeakably horrible code

59 7. Alternative Compilation Schemes and Optimizable Language Kernels

60 High-level Languages for Targeting FPGAs LegUp (U. Toronto) Kiwi (Microsoft Research) Bluespec (MIT) make it easier to write computational kernels Lime (IBM Research)

61 asm.js! C/C++ Javascript LLVM subset that is easier to optimize Browser JIT with asm.js support

62 RPython (c.f. Truffle)! Python Human codes interpreter for language X subset that is easier to optimize JIT for language X

63 Models vs Storage class Student { String name; BigDecimal score; } code is hard to maintain, or impossible to express in the language, but performance is great code is easy to maintain, but performance sucks

64 Lowering class Student { String name; BigDecimal score; } Can we lower from one to the other?

65 Lowering class Student { String name; BigDecimal score; } What is lowering other than a partial evaluation of serialization?

66 Partial Marshal class Student { String name; BigDecimal score; } transformer?? student.getname(); transformer?? class StudentTable { CharTable names; int[] namestart; int[] nameend; IntTable scores; int[] scorestart; int[] scoreend; } students.getname(i); names.splice(students.namestart[i], students.nameend([i]);

67 Partial Marshal class Student { String name; BigDecimal score; } data model class StudentTable CharTable names; int[] namestart; int[] nameend; IntTable scores; int[] scorestart; int[] scoreend; } student.getname(); data access students.getname(i); names.splice(students.namestart[i], students.nameend([i]);

68 Attic

69 All Data Sets 50% Optimizable Data Motion 31% Amortizable 19% Vital Batch A Batch B 77% 55% Amortizable Framework T Framework U Framework Z Framework Y Framework X Framework W 59% 43% Amortizable J2EE Provider A jboss J2EE Provider B Analytics D Analytics C Analytics A Analytics F Analytics E SPECjEnterprise tradesoap SPECjbb2013 tradebeans 49% Optimizable 46% 35% 11% 65% Vital