Program Synthesis for Direct Manipulation Interfaces. Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob Albers

Transcription

1 Program Synthesis for Direct Manipulation Interfaces Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob Albers

2 Motivating Application The original Chicago Ferris Wheel, built for the 1893 World's Columbian Exposition [Wikipedia] 2

3 Imagine Using a Direct Manipulation System 3

4 Imagine Using a Direct Manipulation System 4

5 Imagine Using a Direct Manipulation System 5

6 Imagine Using a Direct Manipulation System 6

7 Imagine Using a Direct Manipulation System 7

8 Imagine Using a Direct Manipulation System 8

9 Imagine Using a Direct Manipulation System 9

10 Imagine Using a Direct Manipulation System 10

11 Imagine Using a Direct Manipulation System 11

12 Imagine Using a Direct Manipulation System 12

13 Imagine Using a Direct Manipulation System 13

14 Imagine Using a Direct Manipulation System 14

15 Imagine Using a Direct Manipulation System 15

16 Imagine Using a Direct Manipulation System 16

17 Imagine Using a Programmatic System function ferriswheel ( numspokes, spokelen, rotangle, sizecar, radiuscenter, cx, cy ) {... } ferriswheel(8,...); 17

18 Imagine Using a Programmatic System function ferriswheel ( numspokes, spokelen, rotangle, sizecar, radiuscenter, cx, cy ) {... } ferriswheel(8,...); 18

19 Imagine Using a Programmatic System function ferriswheel ( numspokes, spokelen, rotangle, sizecar, radiuscenter, cx, cy ) {... } ferriswheel(8,...); 19

20 Imagine Using a Programmatic System function ferriswheel ( numspokes, spokelen, rotangle, sizecar, radiuscenter, cx, cy ) {... } ferriswheel(8,...); 20

21 Imagine Using a Programmatic System function ferriswheel ( numspokes, spokelen, rotangle, sizecar, radiuscenter, cx, cy ) {... } ferriswheel(16,...); 21

22 GOAL: Programmatic Manipulation GOAL: Direct Manipulation VISION and LONG-TERM GOAL Immediate Feedback Rapid Prototyping Abstraction Capabilities 22

23 Program Output 23

24 Live Synchronization Constrain Search Space? Predictable Updates? Efficient Search? Resolve Ambiguities? Trace-Based Synthesis Small Updates Simple Solvers Heuristics Updated Program GOAL: Infer Update to Program While User Changes the Output Updated Output 24

25 Live Synchronization Constrain Search Space? Predictable Updates? Efficient Search? Resolve Ambiguities? General Challenges Trace-Based Synthesis Small Updates Simple Solvers Heuristics General Solutions Sketch-n-Sketch Scalable Vector Graphics (SVG) 25

26 Sketch-n-Sketch let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi [ ] 26

27 Sketch-n-Sketch let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 7) 27

28 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) 28

29 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2 =

30 GOAL: Infer Program Update s.t. xi = 155 when i = 2 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2 = 110 x2' =

31 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

32 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = if (i=2) 155 (x0 + i * sep) in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

33 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

34 let (x0, y0, w, h, sep) = (95, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

35 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

36 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 52.5) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

37 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

38 let (x0, y0, w, h, sep) = (55, 120, 20, 90, 20 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

39 let (x0, y0, w, h, sep) = (75, 120, 20, 90, 10 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

40 let (x0, y0, w, h, sep) = (85, 120, 20, 90, 5 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x2' =

41 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) Constrain Search Space? Predictable Updates? Efficient Search? Resolve Ambiguities? x2' =

42 Key Idea: Value-Trace Equations e' t 'v' 42

43 Key Idea: Value-Trace Equations e' t 'v' e' t 'v' Constraint on Original e: v' = t Constraint on Updated e': v' = t 43

44 Key Idea: Value-Trace Equations e' t 'v' e' t 'v' Constraint on Original e: v' = t Constraint on Updated e': v' = t 44

45 Dataflow-Only Traces e' t 'v' 45

46 Dataflow-Only Traces let a = 1 let b = 2 in a a 1' a+b a+b 3' a+b*b a+b*b 5' (\x.x) a a 1' true? a : b a 1' 46

47 Constraint on Updated e': 155 = x0 + i * sep x2' =

48 Constraint on Updated e': 155 = x0 + 2 i * sep xi if (i=2) 155 else x0 + i * sep x0 55, sep 20 x0 75, sep 10 x0, sep x0 95 sep 52.5 x2' =

49 xi if (i=2) 155 else x0 + i * sep x0 55, sep 20 x0 75, sep 10 x0, sep x0 95 sep 52.5 Change Only Constants in Original Program 49

50 Small Updates xi if (i=2) 155 else x0 + i * sep x0 55, sep 20 x0 75, sep 10 x0, sep x0 95 Change Only Constants in Original Program sep 52.5 Change Only One Constant per Updated Attribute 50

51 Small Updates How to Break Ambiguity? Asking User Live Synchronization Use Simple Heuristics! x0 95 sep 52.5 x2 = 110 x2' =

52 "Fair" Heuristic x0... sep... 52

53 "Fair" Heuristic + Other Simple Heuristics Choices Made Before User Action + Values Updated During User Action = Live Synchronization 53

54 Implementation ~6,500 LOC in Elm + JavaScript Little: Untyped Functional Language + SVG A Simple, Syntax-Directed Equation Solver No GUI Tools for Drawing New Shapes Evaluation Examples (2,000+ LOC in Little) 2. User Study: Effectiveness of Heuristics 54

55 Examples (Demo) n- sketch/releases/v0.4/ 55

56 Freezing Constants let sep = 30! in

57 Range Annotations and Sliders zeroto 3{1-15}

58 Programmatic Abstractions Botanic Garden Chicago Flag ActiveTrans

59 Programmatic Abstractions Botanic Garden Chicago Flag ActiveTrans

60 User-Defined Widgets Custom Sliders Rounded Rectangles Mask Tokens

61 User Study Dealing with Ambiguity in Sketch-n-Sketch (A) Sliders (No Heuristics) (B) Heuristics (No Sliders) 61

62 User Study Dealing with Ambiguity in Sketch-n-Sketch (A) Sliders (No Heuristics) (B) Heuristics (No Sliders) Hypotheses A Sometimes Better Than B B Sometimes Better Than A 62

63 Before After For Each Before/After Task, Two Videos: Mode (A): Sliders Mode (B): Heuristics 25 Users Rated Preferences: A B 63

64 Before After Survey Results A B A B A B 64

65 Survey Results Hypotheses A Sometimes Better Than B B Sometimes Better Than A A A B B A B 65

66 RECAP: Live Synchronization Constrain Search Space? Predictable Updates? Efficient Search? Resolve Ambiguities? General Challenges Trace-Based Synthesis Small Updates Simple Solvers Heuristics General Solutions Sketch-n-Sketch Scalable Vector Graphics (SVG) 66

67 FUTURE: Live Synchronization Smarter Solvers Smarter Heuristics Other Domains HTML+CSS Presentations Visualizations Spreadsheets Trace-Based Synthesis Small Updates Simple Solvers Heuristics Sketch-n-Sketch General Solutions Scalable Vector Graphics (SVG) 67

68 GOAL: GOAL: VISION: Programmatic Manipulation Direct Manipulation "Prodirect" Manipulation Immediate Feedback Rapid Prototyping Abstraction Capabilities 68

69 VISION: Prodirect Manipulation GOAL A: Live Synchronization Infer "Small" Program Update During Direct Manipulation Program GOAL B: Infer Program Output GOAL C: Ad Hoc Synchronization Infer "Large" Program Update After Direct Manipulation 69

70 Thanks! Prodirect Manipulation = Programmatic + Direct Manipulation Sketch-n-Sketch = Prodirect Manipulation for SVG ravichugh.github.io/sketch- n- sketch

71 EXTRA SLIDES 71

72 Related Work [Use + Draw] Bret Victor / Apparatus / PBM [Code] Processing, Slideshow [Draw + Code] SVG-edit / PaintCode [Draw + Code] SketchPad / ThingLab 72

73 Related Work Program Synthesis by Sketching Little programs are sketches, where Holes are constants, and Specification is set of trace-value equations Programming by Example (PBE) Bidirectional Programming 73

74 Examples (Demo) Freezing Constants let sep = 30! in Range Annotations and Sliders zeroto 3{1-15} Programmatic Abstractions Botanic Garden Chicago Flag ActiveTrans User-Defined Widgets Custom Sliders Rounded Rectangles Mask Tokens 74

75 User Study Ways to Deal with Ambiguity: (A) Sliders (No Heuristics) (B) Heuristics (No Sliders) (C) Neither Programming Only Unique to Sketch-n-Sketch 75

76 User Study Ways to Deal with Ambiguity: (A) Sliders (No Heuristics) (B) Heuristics (No Sliders) Unique to Sketch-n-Sketch (C) Neither Programming Only Hypotheses: #1: A or B Usually Better Than C #2: B Sometimes Better Than A 76

77 Before After For Each Before/After Task, Two Videos: Mode (A) and Mode (B) 25 Users Rated Preferences Between Each Pair of Modes: (A) vs (B) 77

78 Before After Survey Results 78

79 Survey Results Hypothesis #1: A or B Usually Better Than C 79

80 Survey Results Hypothesis #2: B Sometimes Better Than A 80

81 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) Small Update (Change x0) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) x0 x0' 81

82 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) Small Update (Change x0) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) Small Update (Change sep) x1 x1' 82

83 let (x0, y0, w, h, sep) = (50, 120, 20, 90, 30 ) Small Update (Change x0) let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h map boxi (zeroto 3) Small Update (Change sep) x2 x2' 83

84 "Fair" Heuristic Small Update (Change x0) Small Update (Change sep) 84