The Rise and Rise of Dataflow in the JavaVerse

Transcription

1 The Rise and Rise of Dataflow in the 1

2 The Plan for the Session Stuff. More stuff. Even more stuff possibly. Summary and conclusions maybe. Q & A Notice the flow going on here. 2

3 Some Definitions Dataflow moving data from one transformation to another. Javaverse the Java Platform: the JVM the Java compiler, and the standard library; other languages such as Kotlin, Ceylon, Groovy, Scala, Clojure, etc; other libraries from Bintray, Maven Central, etc. 3

4 More on the Term Dataflow Wikipedia: Dataflow can also be called stream processing or reactive programming. Dataflow programming is about pipelines. data flow programming promotes high-level functional style Streams, pipelines imply one-dimensional. 4

5 Some Libraries of Interest GPars Akka Quasar RxJava Apache Spark 5

6 What is a Program? n m 6

7 Example 7

8 Let s be a bit abstract for the moment. 8

9 Let Us Assume Java and public static Integer f1(final Integer i) { return i * 2; } public static Integer f2(final Integer i) { return i * 3; } public static Integer f3(final Integer i) { return i * 4; } All this in a class. 9

10 Compute a Value Given an Input public static Integer statementsequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3; } Relatively declarative style. Quite dataflow oriented. 10

11 Diagrammatically 11

12 Compute a Value Given an Input public static Integer statementsequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3; } Relatively declarative style. Quite dataflow oriented. 12

13 Compute a Value Given an Input public static Integer statementsequence(final Integer i) { Integer x = f1(i); x = f2(x); x = f3(x); return x; } Very traditional Imperative style. 13

14 State change vs. Data flow 14

15 Compute a Value Given an Input public static Integer functionapplication(final Integer i) { return f3(f2(f1(i))); } A more functional and declarative approach. 15

16 Diagrammatically 16

17 Compilers deal in dataflow analysis, let us not forget this view of a program. 17

18 That is all very Java, how about doing it with Frege? 18

19 The Frege Transforms f1 = (* 2) f2 = (* 3) f3 = (* 4) Pointfree definition Frege has top-level functions. of functions using partial evaluation. 19

20 Compute a Value Given an Input bindingsequencescalar i = f3 i3 where i2 = f1 i i3 = f2 i2 An Imperative style functional approach. 20

21 Diagrammatically 21

22 Compute a Value Given an Input functionapplicationscalarexplicit i = f3 (f2 (f1 i)) A more functional and declarative approach. 22

23 Compute a Value Given an Input functionapplicationscalar i = f3 $ f2 $ f1 $ i A more functional and declarative approach. 23

24 Compute a Value Given an Input functioncompositionscalar i = f3. f2. f1 $ i A more functional and declarative approach. 24

25 Diagrammatically 25

26 Let s do the maths 26

27 Function Composition f 3 (f 2 (f 1 ( x))) = (f 3 f 2 f 1 )( x) It s only a bit of maths, do not be afraid. 27

28 Compute a Value Given an Input functionsavedcompositionscalar i = f i where f = f3. f2. f1 A more functional and declarative approach. 28

29 This example is seriously unrealistic. 29

30 Make it a wee bit more realistic by switching to a potentially infinite dataset 30

31 let s stage this by doing a finite sequence first 31

32 let s go (statically compiled) Groovy Do not have to have classes, top-level functions are allowed. 32

33 The fs but Groovy Integer f1(final Integer i) { i * 2 } Integer f2(final Integer i) { i * 3 } Integer f3(final Integer i) { i * 4 } 33

34 Compute a Value Given an Input List<Integer> statementsequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { final i1 = f1(i) final i2 = f2(i1) final i3 = f3(i2) result.add(i3) } result } 34

35 Compute a Value Given an Input List<Integer> statementsequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { def x = f1(i) x = f2(x) x = f3(x) result.add(x) } result } 35

36 Compute a Value Given an Input List<Integer> functionapplication(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i : l) { result.add(f3(f2(f1(i)))) } result } 36

37 But this is all about state, no real dataflow. So 37

38 Compute a Value Given an Input List<Integer> usingstream(final List<Integer> l) { l.stream().map(this.&f1).map(this.&f2).map(this.&f3).collect(collectors.tolist()) } This is using Streams from the Java Platform library. 38

39 Diagrammatically 39

40 Diagrammatically 40

41 Compute a Value Given an Input List<Integer> usingstream(final List<Integer> l) { l.stream().map(this.&f1) Intermediate.map(this.&f2).map(this.&f3) Terminal.collect(Collectors.toList()) } This is using Streams from the Java Platform library. 41

42 Do this with explicit composition? 42

43 Frege functioncompositionsequence l = map (f3. f2. f1) l 43

44 Let s introduce a new language: Kotlin Do not have to have classes, top-level functions are allowed. 44

45 The Three Functions fun f1(i:int):int = i * 2 fun f2(i:int):int = i * 3 fun f3(i:int):int = i * 4 We can already tell that Kotlin will be lots of fun. 45

46 Using Streams fun usingstream(l:list<int>):list<int> = l.stream().map(::f1).map(::f2).map(::f3).collect(collectors.tolist<int>()) 46

47 Kotlin version fun usingmap(l:list<int>):list<int> = l.map(::f1).map(::f2).map(::f3) 47

48 Using composition fun usingcomposedmap(l:list<int>):list<int > = l.map(::f1 compose ::f2 compose ::f3) 48

49 Kotlin Compose infix fun<v, T, R> Function1<T, R>.compose(before: (V) -> T): (V) -> R { return { v: V -> this(before(v)) } } t b (i) = t (b(i)) 49

50 But this is still finite, what about potentially infinite? 50

51 Cannot do collect. Map is another name for collect in most circumstances. 51

52 Infinite Data With an infinite data sequence you can: Do some form of reduction Or windowing. Perform a side-effect, e.g. output. 52

53 New (more realistic) problem 53

54 Cumulative mean and Standard deviation. 54

55 Equation warning: please do not be afraid. 55

56 1 n x = i=0 x i n n 1 2 s= (x x ) i i=0 n 1 56

57 1 n x = i=0 x i n n s= (( i=0 x i ) n x ) n 1 57

58 58

59 I only do equations after being fed 59

60 Code? 60

61 A little architecture first. 61

62 62

63 Code. 63

64 What s the Message? Small, single threaded, communicating processes are easy to program. (Communicating Sequential Processes, CSP) Threadpools and processpools make parallelism easy to realize, without manual locks. Most calculations and dataset are now very big, hence Big Data. 64

65 Parallelism is mandatory For Big Data. 65

66 Dataflow not state is required for parallelism. 66

67 The Rise and Rise of Dataflow in the 67

68 But before I go 68

69 69

70 Q & A 70

71 The Rise and Rise of Dataflow in the 71