WAY MORE THAN YOU EVER WANTED TO KNOW ABOUT C++ TEMPLATES AND C++ TEMPLATE METAPROGRAMMING DYLAN KNUTSON, UCSD 13-17

Transcription

1 WAY MORE THAN YOU EVER WANTED TO KNOW ABOUT C++ TEMPLATES AND C++ TEMPLATE METAPROGRAMMING DYLAN KNUTSON, UCSD Creative Commons Attribution + Noncommercial

2 WHO AM I Facebook Seattle s Core Systems/Infra Returning full time Thrift team Small team: Less than 10 engineers Worked on: Thrift s compiler and serialization layer C++ Template Metaprogramming (TMP) libraries

3 THE AGENDA Working on the Thrift team Responsibilities and differences/similarities to academia How Thrift implements serialization Why Apache Thrift s approach doesn t scale all that well How a TMP based approach might help Templates Templates Templates

4 WORKING AS AN INTERN Lots of autonomy Interns are full time employees There s value in projects that might not work out Easy to talk to other teams & get input Lots of world-class engineers to pick the brains of

5 THE COMPANY FB has ~17k employees Thousands of independent projects Nearly every service uses Thrift in one way or another

6

7 This meant: Writing really solid code It was never good enough the first time around Lots of testing

8 SOME BACKGROUND C++ BNF GRAMMAR Wot in compilation 1.2k lines Undecidable grammar

9 THE SOLUTION? JUS T GENERATE LESS CODE Don t generate the code that does serialization at all! Use templated C++ code to do this in the compiler [1] Parse a generic library that implements marshaling, and the Thrift IDL embedded in C++ templates Separate what to serialize, from how to serialize it [1]

10 WHAT THIS BUYS YOU Much less code generated Metadata code is declarative Code is easier to debug Compiler is another tool that can be improved Specifics on how this was done? Another talk

11 SOME BACKGROUND WHAT DOES SERIALIZATION LOOK LIKE? Thrift IDL Constructors Member variables Setters (update `isset`) Equality operators (==, <,!=) Read/write methods Generated C++

12 SOME BACKGROUND WHAT DOES SERIALIZATION LOOK LIKE? start: - Write header start struct for every field: - Write header start field - Write the value and type of the field - Is this an aggregate? Recurse - Write footer end field end: - Write footer end struct

13 SOME BACKGROUND WHAT DOES SERIALIZATION LOOK LIKE? start: - Read header start struct for every field: - Read header start field - Read the value and type of the field (and check it s right) - Is this an aggregate? Recurse - Read footer end field end condition: - Read footer end struct

14 SOME BACKGROUND WHAT DOES SERIALIZATION LOOK LIKE?

15 SOME BACKGROUND WHAT DOES SERIALIZATION LOOK LIKE? json protocol: { "1":{"str":"John Doe"}, "2":{"i32":24}, "3":{"lst":["str",2,"coding","hiking"]} } binary protocol: \v\x00\x01\x00\x00\x00\bjohn Doe\b\x00\x02\x00\x00\x00\x18\x0F\x00\x03\v\x00\x 00\x00\x02\x00\x00\x00\x06coding\x00\x00\x00\x06h iking\x00 compact protocol: \x18\bjohn Doe\x150\x19(\x06coding\x06hiking\x00

16 SOME BACKGROUND WHAT DOES SERIALIZATION LOOK LIKE? This generated code gets really big, really fast ~10 to 50 lines per field map<map<i32, string>> list<list<list<i32>>> Parsing C++ is hard Minutes of time spent just parsing, every time compilation occurs

17 WHAT ARE C++ TEMPLATES? Way to make your C++ code generic Write type-specific versions of functions Turing complete!

18 WHAT ARE C++ TEMPLATES?

19 WHAT ARE C++ TEMPLATES?

20 TEMPLATE SYNTAX

21 TEMPLATE SYNTAX

22 PERFORMING COMPUTATION

23 VARIADIC TEMPLATES

24 VARIADIC TEMPLATES

25 VARIADIC TEMPLATES

26 DESTRUCTURING TEMPLATES

27 DATA STRUCTURES IN TEMPLATES Let s make a list! Implement indexing Test that it works

28 DATA STRUCTURES IN TEMPLATES ARE T WO T YPES THE SAME HELPER

29 DATA STRUCTURES IN TEMPLATES THE ACTUAL T YPE LIS T

30 DATA STRUCTURES IN TEMPLATES THE INDEX FUNCTION

31 DATA STRUCTURES IN TEMPLATES PUTTING IT ALL TOGETHER

32 DATA STRUCTURES IN TEMPLATES OTHER METHODS Sorting Slicing Reversing Binary searching Maps Sets

33 TRANSLATING THRIFT TO C++ TEMPLATE

34 WRITING A SERIALIZER

35 WRITING A SERIALIZER

36 WRITING A SERIALIZER

37 WRITING A SERIALIZER

38 WRITING A SERIALIZER

39 WRITING A SERIALIZER

40 RESULTS ~50% less code in the average case Compared to generated code, much easier to debug Easier to extend pdimov.com/cpp2/simple_cxx11_metaprogramming.html github.com/facebook/fatal ericniebler.com/2014/11/13/tiny-metaprogramminglibrary/