Parsing All of C by Taming the Preprocessor. Robert Grimm, New York University Joint Work with Paul Gazzillo

Transcription

1 Parsing All of C by Taming the Preprocessor Robert Grimm, New York University Joint Work with Paul Gazzillo

2 Our Research Goal Better Tools for C 2

3 The Good Operating systems ios, OS X, Linux, Free/Open/Net BSD Databases SQLite, Berkeley DB, IBM DB2 Internet servers Apache, Bind, Sendmail 3

4 The Bad > emacs main.c > gcc main.c >./a.out > gdb a.out 4

5 We Need Better Tools Source browsing 7,500+ compilation units, 5.5 million lines [x86 Linux ] 5

6 We Need Better Tools Source browsing 7,500+ compilation units, 5.5 million lines [x86 Linux ] Bug finding 1,000+ bugs in single static analysis study [Chou et al., SOSP 01] 6

7 We Need Better Tools Source browsing 7,500+ compilation units, 5.5 million lines [x86 Linux ] Bug finding 1,000+ bugs in single static analysis study [Chou et al., SOSP 01] Automated refactoring 150+ bugs due to manual API evolution [Padioleau et al., EuroSys 08] 7

8 We Need Better Tools Source browsing Cxref, LXR Bug finding Astrée, Coverity Automated refactoring Apple s Xcode, Eclipse CDT 8

9 All These Tools Need to Parse Code First... 9

10 But Code Is Written in Two Languages! 10

11 The Fall From Grace Preprocessor enriches C Static conditionals (#if, #ifdef, ) Capture variability Macros (#define) Enable code transformation File includes (#include) Provide rudimentary modularity 11

12 The Fall From Grace Preprocessor obfuscates C Static conditionals (#if, #ifdef, ) Code has many meanings Macros (#define) Code doesn t mean what s written File includes (#include) Code is incomplete 12

13 The Fall From Grace Preprocessor obfuscates C Static conditionals (#if, #ifdef, ) Code has many meanings Macros (#define) Code doesn t mean what s written File includes (#include) Code is incomplete Preprocessor only works on tokens, not C constructs 13

14 Back to Eden? One configuration at a time (preprocess first) Causes exponential explosion Incomplete heuristic parsing algorithm Works, if you don t write the wrong code Plug-in architecture for further CPP idioms Creates arms race: tool builders vs developers 14

15 More of the Same Adams et al., AOSD 09 Akers et al., WCRE 05 Badros and Notkin, SP&E 00 Baxter and Mehlich, WCRE 01 Favre, IWPC 97 Garrido and Johnson, ICSM 05 McCloskey and Brewer, ESEC 05 Padioleau, CC 09 Spinellis, TSE 03 Vittek, CSMR 03 15

16 Not Quite the Same MAPR by Platoff et al., ICSM 91 TypeChef by Kästner et al., OOPSLA 11 16

17 This Talk Introduction Problem and Solution Approach The Configuration-Preserving Preprocessor The Configuration-Preserving Parser Our Tool and Evaluation Conclusion 17

18 #include major.h static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == MISC_MAJOR) i = 31; else i = iminor(inode) - 32; } 18

19 #define MISC_MAJOR 10 #include major.h static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == MISC_MAJOR) i = 31; else i = iminor(inode) - 32; } 19

20 #define MISC_MAJOR 10 static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == MISC_MAJOR) i = 31; else i = iminor(inode) - 32; } 20

21 #define MISC_MAJOR 10 static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == MISC_MAJOR) i = 31; else i = iminor(inode) - 32; } 20

22 #define MISC_MAJOR 10 static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == 10) i = 31; else i = iminor(inode) - 32; } 21

23 static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == 10) i = 31; else i = iminor(inode) - 32; } 22

24 Fork subparsers on conditional static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == 10) i = 31; else i = iminor(inode) - 32; } 22

25 Fork subparsers on conditional Parse entire static int if-then-else mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == 10) i = 31; else i = iminor(inode) - 32; } 22

26 Fork subparsers on conditional Parse entire static int if-then-else mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == 10) else i = 31; Parse just assignment i = iminor(inode) - 32; } 22

27 static int mousedev_open( ) { #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX if (imajor(inode) == 10) else i = 31; Merge and create static choice node i = iminor(inode) - 32; } Static Choice INPUT PSAUX If-Then-Else!INPUT PSAUX Assignment 23

28 Solution Approach 1. Lex to produce tokens (as before) 2. Preprocess while preserving conditionals 3. Parse across conditionals 24

29 Language Construct Lexer Layout Preprocessor Macro (Un)Definition Implementation Annotate tokens Surrounded by Contain Contain Multiply- Conditionals Conditionals Defined Macros Other Use conditional Add multiple entries Do not expand Trim infeasible entries macro table to macro table until invocation on redefinition Object-Like Expand all Ignore infeasible Expand nested Get ground truth for Macro Invocations definitions definitions macros built-ins from compiler Function-Like Expand all Ignore infeasible Hoist conditionals Expand nested Support di ering argument Macro Invocations definitions definitions around invocations macros numbers and variadics Token Pasting & Stringification File Includes Static Conditionals Conditional Expressions Error Directives Line, Warning, & Pragma Directives Parser Apply pasting & stringification Hoist conditionals around token pasting & stringification Include and Preprocess under Hoist conditionals Reinclude when guard preprocess files presence conditions around includes macro is not false Preprocess all branches Conjoin presence conditions Ignore infeasible definitions Evaluate presence Hoist conditionals Preserve order for nonconditions around expressions boolean expressions Ignore erroneous branches Treat as layout C Constructs Use FMLR Parser Fork and merge subparsers Typedef Names Use conditional Add multiple entries Fork subparsers on symbol table to symbol table ambiguous names 25

30 Three Key Surrounded by Techniques Contain Contain Multiply- Language Construct Implementation Other Conditionals Conditionals Defined Macros Lexer Layout Annotate tokens Preprocessor Macro (Un)Definition Use conditional Add multiple entries Do not expand Trim infeasible entries macro table to macro table until invocation on redefinition Track presence conditions Object-Like Expand all Ignore infeasible Expand nested Get ground truth for Macro Invocations definitions definitions macros built-ins from compiler Function-Like Expand all Ignore infeasible Hoist conditionals Expand nested Support di ering argument Macro Invocations definitions definitions around invocations macros numbers and variadics In preprocessor and parser Token Pasting & Hoist conditionals around Apply pasting & stringification Stringification token pasting & stringification File Includes Hoist conditionals Static Conditionals Include and Preprocess under Hoist conditionals Reinclude when guard preprocess files presence conditions around includes macro is not false Preprocess all branches Conjoin presence conditions Ignore infeasible definitions Evaluate presence Hoist conditionals Preserve order for nonconditions around expressions boolean expressions Conditional Expressions In preprocessor Error Directives Ignore erroneous branches Line, Warning, & Treat as Pragma Directives layout Parser Contain number of forked subparsers C Constructs Use FMLR Parser Fork and merge subparsers Typedef Names In parser Use conditional Add multiple entries Fork subparsers on symbol table to symbol table ambiguous names 25

31 This Talk Introduction Problem and Solution Approach The Configuration-Preserving Preprocessor The Configuration-Preserving Parser Our Tool and Evaluation Conclusion 26

32 Track Presence Conditions Do not evaluate conditional expressions Rather translate to boolean functions (BDDs) Enables reasoning about configurations Including equivalent and invalid branches Four base cases Constants, free macros, defined operator Straight-forward case analysis Arithmetic expressions Treat as opaque: no known efficient algorithm 27

33 Track Presence Conditions Maintain conditional and ordered macro table Contains all definitions of a macro Invocation is an implicit conditional #ifdef CONFIG_64BIT #define BITS_PER_LONG 64 #else #define BITS_PER_LONG 32 28

34 Track Presence Conditions Maintain conditional and ordered macro table Contains all definitions of a macro Invocation is an implicit conditional #ifdef CONFIG_64BIT #define BITS_PER_LONG 64 #else #define BITS_PER_LONG 32 BITS_PER _LONG 64BIT!64BIT

35 Hoist Conditionals Preprocessor invocations do not compose Directives are exactly one source line Operators only work on tokens le ## BITS_PER_LONG 29

36 Hoist Conditionals Preprocessor invocations do not compose Directives are exactly one source line Operators only work on tokens le ## BITS_PER_LONG Macro expands to conditional 29

37 Hoist Conditionals Preprocessor invocations do not compose Directives are exactly one source line Operators only work on tokens le ## BITS_PER_LONG le ## #ifdef CONFIG_64BIT 64 #else 32 29

38 Hoist Conditionals le ## BITS_PER_LONG le ## #ifdef CONFIG_64BIT 64 #else 32 30

39 Hoist Conditionals One operator, two operations le ## BITS_PER_LONG le ## #ifdef CONFIG_64BIT 64 #else 32 30

40 Hoist Conditionals le ## BITS_PER_LONG Hoist conditional around pasting le ## #ifdef CONFIG_64BIT 64 #else 32 #ifdef CONFIG_64BIT le ## 64 #else le ## 32 30

41 Algorithm 1 Hoisting Conditionals 1: procedure Hoist(c, ) 2: B Initialize a new conditional with an empty branch. 3: C [(c, )] 4: for all a 2 do 5: if a is a language token then 6: B Append a to all branches in C. 7: C [(c i, i a) (c i, i ) 2 C ] 8: else B a is a conditional. 9: B Recursively hoist conditionals in each branch. 10: B [ b b 2 Hoist(c i, i ) and (c i, i ) 2 a ] 11: B Combine with already hoisted conditionals. 12: C C B 13: end if 14: end for 15: return C 16: end procedure 31

42 Algorithm 1 Hoisting Conditionals The Power of Hoisting 1: procedure Hoist(c, ) 2: B Initialize a new conditional with an empty branch. 3: C [(c, )] 4: for all a 2 do 5: if a is a language token then 6: B Append a to all branches in C. 7: C [(c i, i a) (c i, i ) 2 C ] 8: else B a is a conditional. 9: B Recursively hoist conditionals in each branch. 10: B [ b b 2 Hoist(c i, i ) and (c i, i ) 2 a ] 11: B Combine with already hoisted conditionals. 12: C C B 13: end if 14: end for 15: return C 16: end procedure Iterates over conditional branches Recurses into nested conditionals Duplicates tokens across innermost branches Works for token pasting, stringification, includes, conditional expressions, macros 31

43 This Talk Introduction Problem and Solution Approach The Configuration-Preserving Preprocessor The Configuration-Preserving Parser Our Tool and Evaluation Conclusion 32

44 Review: LR Parser p Finite u Control S S u p e r C Input I Stack 33

45 Review: LR Parser p Finite u Control S u p e r C Input Shift token from input to stack S I Stack 33

46 Reduce top n elements to nonterminal Review: LR Parser p Finite u Control S u p e r C Input Shift token from input to stack S I Stack 33

47 Use table for actions and transitions Reduce top n elements to nonterminal Review: LR Parser p Finite u Control S u p e r C Input Shift token from input to stack S I Stack 33

48 Why Build on LR? Has explicit state and stack Straight-forward to fork and merge Is table-driven Can make it fast Supports left-recursion But shift-reduce and reduce-reduce conflicts 34

49 Fork-Merge LR (FMLR) Input: Preprocessed source code Ordinary tokens from C language Main data structure: Subparser Head, i.e., next token LR stack 35

50 Fork-Merge LR (FMLR) Input: Preprocessed source code Ordinary tokens from C language Entire conditionals Branches contain tokens and conditionals Main data structure: Subparser Presence condition Head, i.e., next token or conditional LR stack 36

51 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

52 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

53 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

54 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

55 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

56 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

57 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

58 Algorithm 2 Fork-Merge LR Parsing 1: procedure Parse(a 0 ) 2: Q.init((true, a 0, s 0 )) B The initial subparser for a 0. 3: while Q, ; do 4: p Q.pull() B Step the next subparser. 5: T Follow(p.c, p.a) 6: if T = 1 then 7: B Do an LR action and reschedule the subparser. 8: Q.insert(LR(T(1), p)) 9: else B The follow-set contains several tokens. 10: B Fork subparsers and reschedule them. 11: Q.insertAll(Fork(T, p)) 12: end if 13: Q Merge(Q) 14: end while 15: end procedure 37

59 Fork-Merge LR Highlights Fork subparsers from other subparsers Have different heads, presence conditions But reuse LR stack frames in a DAG 38

60 Fork-Merge LR Highlights Fork subparsers from other subparsers Have different heads, presence conditions But reuse LR stack frames in a DAG Merge subparsers if same heads and stacks Disjoin presence conditions Priority queue ensures merging happens ASAP 39

61 Fork-Merge LR Highlights Fork subparsers from other subparsers Have different heads, presence conditions But reuse LR stack frames in a DAG Merge subparsers if same heads and stacks Disjoin presence conditions Priority queue ensures merging happens ASAP Perform ordinary LR on ordinary tokens Reuse grammars and table generators 40

62 The Merge Criterion Merge only if same heads and stacks Implies same derivation of nonterminals Which also ensures a well-formed AST But may parse some code more than once #ifdef CONFIG PSAUX if ( ) i = 31; else i = ; 41

63 The Merge Criterion Merge only if same heads and stacks Implies same derivation of nonterminals Which also ensures a well-formed AST But may parse some code more than once #ifdef CONFIG PSAUX if ( ) i = 31; else i = ; Parse twice and merge thereafter 41

64 The Merge Criterion Merge only if same heads and stacks Implies same derivation of nonterminals Which also ensures a well-formed AST But may parse some code more than once #ifdef CONFIG PSAUX if ( ) i = 31; else i = ; Static Choice INPUT PSAUX!INPUT PSAUX If-Then-Else Assignment 41

65 The Serpent Is in the Details Which subparsers to fork? 42

66 Which Subparsers to Fork? Naively, subparser for each conditional branch But conditionals may Be directly nested in each other Directly follow each other Creates many unnecessary subparsers! 43

67 Which Subparsers to Fork? Insight: Real (LR) work performed on tokens Therefore, create subparser for each token Directly reachable from current position Across all configurations 44

68 Which Subparsers to Fork? Insight: Real (LR) work performed on tokens Therefore, create subparser for each token Directly reachable from current position Across all configurations Encoded in token follow-set 45

69 struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 46

70 6 subparsers struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 46

71 6 subparsers 3 subparsers struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 46

72 struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 46

73 struct rq { SCHED_HRTICK && SMP #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 46

74 struct rq { SCHED_HRTICK && SMP #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif! (SCHED_HRTICK && SMP) && SCHEDSTATS #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 46

75 struct rq { SCHED_HRTICK && SMP #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif! (SCHED_HRTICK && SMP) && SCHEDSTATS #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; };! (SCHED_HRTICK && SMP) &! SCHEDSTATS 46

76 Algorithm 3 The Token Follow-Set 1: procedure Follow(c, a) 2: T ; B Initialize the follow-set. 3: procedure First(c, a) 4: loop 5: if a is a language token then 6: T T [ {(c, a)} 7: return false 8: else B a is a conditional. 9: c r false B Initialize remaining condition. 10: for all (c i, i ) 2 a do 11: if i = then 12: c r c r _ c ^ c i 13: else 14: c r c r _ First(c ^ c i, i (1)) 15: end if 16: end for 17: if c r = false or a is last element in branch then 18: return c r 19: end if 20: c c r 21: a next token or conditional after a 22: end if 23: end loop 24: end procedure 25: loop 26: c First(c, a) 27: if c = false then return T end if B Done. 28: a next token or conditional after a 29: end loop 30: end procedure 47

77 struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 48

78 Find first token of each branch struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 48

79 Find first token of each branch struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; # endif Recurse into nested conditionals #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 48

80 Find first token of each branch struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; Recurse into nested conditionals # endif Continue after empty branches #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; 48

81 Find first token of each branch struct rq { #ifdef CONFIG_SCHED_HRTICK # ifdef CONFIG_SMP int hrtick_csd_pending; Recurse into nested conditionals # endif Continue after empty branches #ifdef CONFIG_SCHEDSTATS struct sched_info rq_sched_info; }; Stop after all reachable tokens found 48

82 The Serpent Is in the Details There is more! 49

83 static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, // And so on, 16 more times NULL }; 50

84 Encodes binary number! static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, // And so on, 16 more times NULL }; 50

85 static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, // And so on, 16 more times NULL }; 50

86 Shared Reduces: Reduce same stack for several heads static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, // And so on, 16 more times NULL }; 50

87 static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, // And so on, 16 more times NULL }; Shared Reduces: Reduce same stack for several heads Lazy Shifts: Delay forking for several heads 50

88 static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, }; // And so on, 16 more times NULL Shared Reduces: Reduce same stack for several heads Lazy Shifts: Delay forking for several heads Early Reduces: Pick reducing before shifting subparser 50

89 static int (*check_part[])(struct *) = { #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ics, #ifdef CONFIG_ACORN_PARTITION_POWERTEC adfspart_check_powertec, // And so on, 16 more times NULL }; Uses only two subparsers! 50

90 Shared Reduces & Lazy Shifts Require multi-headed subparser Several tokens and their presence conditions But only one LR stack Prioritized by first head Fork by removing one head, presence condition Merge by disjoining presence conditions For same heads, LR stack 51

91 This Talk Introduction Problem and Solution Approach The Configuration-Preserving Preprocessor The Configuration-Preserving Parser Our Tool and Evaluation Conclusion 52

92 SuperC Language Java Preprocessor Built from scratch Parser formalism Grammar Forking & Merging FMLR Modified Roskind s, ran through Bison Automatic AST Grammar annotations Presence conditions BDDs Complete Yes 53

93 SuperC TypeChef Language Java Java & Scala Preprocessor Built from scratch Modified existing Parser formalism FMLR LL Grammar Modified Roskind s, ran through Bison Rewritten using combinator library Forking & Merging Automatic 7 merge combinators AST Grammar annotations Semantic actions Presence conditions BDDs SAT solver Complete Yes Still missing features 54

94 And MAPR? Preprocessor is not documented Parser boils down to naive FMLR Fork on every conditional branch No other optimizations Reimplemented in SuperC as option 55

95 Who Is Our Serpent? 56

96 Who Is Our Serpent? Linux! 56

97 The Linux OS Critical to government and business Jarring flexibility and performance requirements From smart phones to cloud computing farms Large and complex Many developers with varying styles, skills 57

98 Evaluation Plan Focus on x86 Linux kernel How complex is preprocessor usage? Compare to MAPR along the way How well does SuperC perform? Compare to TypeChef along the way 58

99 Linux Preprocessor Usage Breadth of conditionals Force forking of parser states Incidence of incomplete C constructs Prevent merging of parser states 59

100 Linux Preprocessor Usage Breadth of conditionals Force forking of parser states Incidence of incomplete C constructs Prevent merging of parser states Count subparsers per FMLR loop iteration! 60

101 Cumulative Distribution of Subparser # per Iteration Fraction % 100% Shared All Follow-Set MAPR: >16,000 on 98% of C files Number of Subparsers 61

102 Cumulative Distribution of Latency per C File Max: 10.4 Max: 931 Fraction 0.50 SuperC TypeChef Latency in Seconds 62

103 Sidebar Totals, averages, and medians are deceiving We really need the entire distribution! 63

104 Latency vs File Size 12 Parse, Preprocess, Lex Latency in Seconds Size in Millions of Bytes 64

105 Future Work Fully characterize variability in Linux For all tools (Kconfig, Kbuild, Preprocessor) Over history of kernel Analyze names and types across conditionals Foundation for all language processing tools Annotate AST with layout, macros, includes Refactorings need to print source code again Modulo intended changes 65

106 Conclusions Preprocessor makes it hard to parse all of C No viable solution for 40 years! We performed a systematic analysis of problem Identified solution strategies Notably: Conditional hoisting, FMLR parsing And implemented a real-world tool, SuperC On x86 Linux kernel, SuperC is fast and scales well 66

107 SuperC 67

108 SuperC 68

109 69

110 SuperC Context Management C syntax is context-sensitive: T * p; If T is a type name, declare p as a pointer Otherwise, calculate the product of T and p Worse, C with conditionals can be ambiguous Same name may be both a type and an object Under different presence conditions 70

111 SuperC Context Management C syntax is context-sensitive: T * p; If T is a type name, declare p as a pointer Otherwise, calculate the product of T and p Worse, C with conditionals can be ambiguous Same name may be both a type and an object Under different presence conditions Fork and merge subparsers based on context! 71

112 SuperC Context Management Provide context object for each subparser Customize forking and merging Reclassify Fork context May merge Merge contexts 72

113 SuperC Context Management Provide context object for each subparser Conditional, scoped symbol table Customize forking and merging Reclassify: change into or add type name token Fork context: duplicate symbol table reference May merge: check number of nested scopes Merge contexts: combine symbol table entries 73