Redesigning Apache Flink s Distributed Architecture. Till

Transcription

1 Redesigning Apache Flink s Distributed Architecture Till Rohrmann

2 2

3 1001 Deployment Scenarios Many different deployment scenarios Yarn Mesos Docker/Kubernetes Standalone Etc. 3

4 Different Usage Patterns Few long running vs. many short running jobs Overhead of starting a Flink cluster Job isolation vs. sharing resources Allowing to define per job credentials & secrets Efficient resource utilization by sharing 4

5 Job & Session Mode Job mode Dedicated cluster for a single job Session mode Shared cluster for multiple jobs Resources can be shared across jobs 5

6 Flink s Current State 6

7 As-Is State (Standalone) Client (2) Submit Job (1) Register (3) Deploy Tasks Standalone Flink Cluster 7

8 As-Is State (YARN) Client (1) Submit YARN App. (FLINK) YARN ResourceManager (6) All started (3) Poll status Application Master (2) Spawn Application Master (4) Start s (7) Submit Job (5) Register (8) Deploy Tasks YARN Cluster 8

9 Problems No clear separation of concerns No dynamic resource allocation No heterogeneous resources Not well suited for containerized execution 9

10 Flink s New Distributed Architecture 10

11 Flink Improvement Proposal 6 Introduce generic building blocks Compose blocks for different scenarios Mainly driven by: Flip-6 design document: pageid=

12 The Building Blocks ResourceManager Dispatcher ClusterManager-specific May live across jobs Manages available Containers/s Used to acquire / release resources Lives across jobs Touch-point for job submissions Spawns s May spawn ResourceManager Single job only, started per job Thinks in terms of "task slots" Deploys and monitors job/task execution Registers at ResourceManager Gets tasks from one or more s 12

13 The Building Blocks Dispatcher ResourceManager (4) Start (2) Start (1) Submit Job (5) Register (3) Request slots (6) Offer slots Client (7) Deploy Tasks 13

14 Building Flink-on-YARN YARN Cluster Client (1) Submit YARN App. (JobGraph / JARs) YARN ResourceManager (2) Spawn Application Master Application Master Flink-YARN ResourceManager (3) Request slots (4) Start s (5) Register (6) Deploy Tasks YARN Cluster 14

15 Differences to old YARN mode JARs in classpath of all components Dynamic resources allocation No two phase job submission 15

16 Building Flink-on-Mesos Mesos Cluster Client (1) HTTP POST JobGraph/Jars Flink Mesos Dispatcher (2) Allocate container for Flink master Mesos Master (3) Start Process (and supervise) Flink Master Process Flink Mesos ResourceManager (4) Request slots (5) Start s (6) Register (7) Deploy Tasks Mesos Cluster 16

17 Building Flink-on-Docker/K8S Master Container Flink Master Process Flink-Container ResourceManager (2) Run & Start Program Runner (3) Register (4) Deploy Tasks Worker Container Worker Container Worker Container (1) Container framework starts Master & Worker Containers 17

18 Containerized Execution Single dedicated Resource- and container and multiple containers Generalization Start N containers Use leader election to determine role; remainder role Enabling auto-scaling groups by rescaling job to fill all available slots 18

19 Multi Job Sessions 19

20 Building Standalone Standby Master Process Flink Master Process Standalone ResourceManager (4) Request slots Standby Master Process (1) Register Flink Cluster Client (2) Submit JobGraph/Jars (3) Start Dispatcher (5) Deploy Tasks Standalone Cluster 20

21 YARN Session (1) Submit YARN App. (FLINK session) YARN ResourceManager (2) Spawn Application Master Client (3) Submit Job A ApplicationMaster Flink-YARN ResourceManager (5) Request slots (11) Request slots (6) Start s (7) Register (9) Submit Job B (A) (B) (8, 12) Deploy Tasks (4) Start JobMngr (10) Start JobMngr Dispatcher YARN Cluster 21

22 Multi Job Sessions Dispatcher spawns for each job a dedicated Jobs run under session user credentials ResourceManager holds on to resources Reuse of allocated resources Quicker response for successive jobs 22

23 Miscellaneous Resource profiles Specify CPU & memory requirements for individual operators ResourceManager allocates containers according to resource profiles New RPC abstraction similar to Akka s typed actors Properly defined interface eases development No longer locked in on Akka 23

24 Conclusion 24

25 Conclusion Different cluster environments have different deployment paradigms Support for Job as well as Session mode in various environments necessary Flip-6 architecture provides necessary flexibility to achieve both 25

26 @dataartisans 26

27 We are hiring! data-artisans.com/careers