Qing Zheng Kai Ren, Garth Gibson, Bradley W. Settlemyer, Gary Grider Carnegie Mellon University Los Alamos National Laboratory

Transcription

1 What s Beyond IndexFS & BatchFS Envisioning a Parallel File System without Dedicated Metadata Servers Qing Zheng Kai Ren, Garth Gibson, Bradley W. Settlemyer, Gary Grider Carnegie Mellon University Los Alamos National Laboratory

2 Throughput (Kop/s) Scaling needs decoupling NASD [asplos98] o decoupling data from metadata o Lustre, Google FS, etc IndexFS [sc14] o dynamically partitioned metadata middleware o orders of magnitude faster than Lustre in metadata 5, Parallel Data Lab - PDSW 2015 Page IndexFS_Lustre (32 clients run IndexFS) Lustre (single server, 32 clients) 100x 30x empty file creation file lookup file deletion Exa- scaling demands ever more decoupling 300x

3 File Creates (Kop/s) Compute-side server code 25,000 BatchFS [pdsw14] o decoupling clients from servers o temporarily scale beyond the total number of servers o very fast for a while and eventually clients communicate with servers to merge updates 20,000 15,000 10,000 5,000 Parallel Data Lab - PDSW 2015 Page servers, 64 clients 618 IndexFS 19,692 BatchFS How much further can we delay & decouple merging? 30X

4 FS Goal Want the peak Tput BatchFS demonstrated Compel freedom from server synchronization o by eliminating all server machines o by dealing with issues rising from the absence of metadata servers o by not assuming an underlying PFS Scale beyond BatchFS Parallel Data Lab - PDSW 2015 Page 4

5 Agenda DeltaFS design Why no dedicated servers is not a problem Parallel Data Lab - PDSW 2015 Page 5

6 Middleware Design FS is middleware spawned by each parallel app App App App P1 P2 P3 FS Pn App App object store storing data/metadata Parallel Data Lab - PDSW 2015 Page 6

7 FS Overview FS defined by a set of snapshots stored as sets of metadata logs and data objects Logical View Object Storage Note: data objects not shown here a list of metadata ops Log rmdir /c Log Log Log FS snapshot Parallel Data Lab - PDSW 2015 Page 7 b / e b / c d b / c e rename /d->/e

8 System Model Reads input dataset from an existing FS snapshot Creates a new snapshot with output data inserted input snapshot produced by a previous app a new snapshot ready to be used by future apps Logical View Object Storage input App create Log Log Log Parallel Data Lab - PDSW 2015 Page 8

9 Key take-away NO global namespace Each namespace is defined by the app and the logs loaded by it NO false sharing Apps don t access logs not needed by them NO dedicated metadata servers App directly communicates with the storage to load/dump metadata logs Parallel Data Lab - PDSW 2015 Page 9

10 How logs are implemented? TableFS [atc13] o namespace = a large dir entry table + embedded inodes implemented as LSM-Tree (a collection of ordered B-Trees) Each log object is a differential B-Tree (diff) o representing a set of recent updates (e.g. newly inserted/modified inodes) Log k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v k/v Parallel Data Lab - PDSW 2015 Page 10

11 Why LSM-Tree is a good idea? Logs are 1 st class data No need to replay logs to recover namespaces Near-zero cost of merging namespaces Each log is self-indexed Scanning/reading within a single log is fast: O(logN) Scanning/reading a series of non-overlapping logs is as fast as a single log Parallel Data Lab - PDSW 2015 Page 11

12 Agenda DeltaFS design Why no dedicated servers is not a problem Parallel Data Lab - PDSW 2015 Page 12

13 P1: Do my apps need the FS to communicate/synchronize? Parallel Data Lab - PDSW 2015 Page 13

14 Unrelated Apps Work on different datasets and don t communicate. / ocean climate App 2 App 1 pacific atlantic Don t need the FS to communicate Parallel Data Lab - PDSW 2015 Page 14

15 Self-Coordinating Apps Use middleware to share faster & more efficiently Parallel Scientific App MPI MPI P1 P2 P3 File Don t need the FS to communicate Parallel Data Lab - PDSW 2015 Page 15

16 Workflow Apps Externally coordinated by job schedulers / Reducer login_log movie_profile user_profile Mapper job scheduler workflow engine Iter3 Iter4 Don t need the FS to communicate Parallel Data Lab - PDSW 2015 Page 16

17 Anonymous Synchronization e.g. Two app instances competing for mastership App 1 App 2 App 1 App 2 Lustre.LOCK.LOCK Zookeeper (ZAB), Paxos, Raft Turn to a mechanism outside the FS to coordinate Parallel Data Lab - PDSW 2015 Page 17

18 Anonymous Synchronization e.g. Two app instances competing for mastership App 1 App 2 App 1 App 2 Lustre.LOCK.LOCK Zookeeper (ZAB), Paxos, Raft Turn to a mechanism outside the FS to coordinate Parallel Data Lab - PDSW 2015 Page 18

19 P2: But I often use different programs to access data concurrently! Parallel Data Lab - PDSW 2015 Page 19

20 User requested concurrent sharing Mon FS attach Viz FS attach P1 P2 App P3 FS Pn Link to FS middleware and attach to the primary parallel app Parallel Data Lab - PDSW 2015 Page 20

21 P3: Which snapshots to use? Parallel Data Lab - PDSW 2015 Page 21

22 Which snapshots to use? Option 1: rely on job schedulers to automate namespace propagation App App App App_1 input= output= job scheduler workflow engine input= output= App_2 Parallel Data Lab - PDSW 2015 Page 22

23 Which snapshots to use? Option 2: ask external registries using search predicates App 1 publish snapshot registry App App_1 2 search 3 collect App App_2 Parallel Data Lab - PDSW 2015 Page 23

24 Finding snapshots is like searching a page using Google Possible search predicates o find latest stable science code for my science o find latest recommended mesh model and cleaned input data o find latest vendor recommended HW libraries Also, there can be multiple snapshot registries Allows programmable namespace composition Parallel Data Lab - PDSW 2015 Page 24

25 P4: What about potential conflicts among different snapshots? Parallel Data Lab - PDSW 2015 Page 25

26 Unrelated Apps Work on different portions of the namespace / ocean climate App 2 App 1 pacific atlantic Won t generate any conflicts Parallel Data Lab - PDSW 2015 Page 26

27 Workflow Apps Access the same dataset at different time / Reducer login_log movie_profile user_profile Mapper job scheduler workflow engine Iter3 Iter4 Won t generate any conflicts Parallel Data Lab - PDSW 2015 Page 27

28 Self-Coordinating Apps Coded to be conflict-free Parallel Scientific App MPI MPI P1 P2 P3 File Won t generate any conflicts Parallel Data Lab - PDSW 2015 Page 28

29 Namespace composition is fast if there is no conflict Recall: near-zero cost of merging logs o better if those logs do not overlap with each other What if there are conflicts? Parallel Data Lab - PDSW 2015 Page 29

30 Use domain knowledge Conflicts resolved per app s own reconciliation policy file_1 /deltafs file_2 /deltafs /deltafs /deltafs file_1 file_2 file_1 file_2 file_1(a) file_2(b) file_1(b) file_2(a) input snapshot input snapshot possible resolution outcome Parallel Data Lab - PDSW 2015 Page 30

31 Use curators to remember conflict resolution results So no duplicated resolutions by different apps a curator inherits a preresolved namespace from an app App a namespace curator another namespace curator Parallel Data Lab - PDSW 2015 Page 31 App App an app directly takes namespaces from 2 curators

32 Conclusion Strong scalability needs strong decoupling o exiting clients synch too often with servers o removing servers force us to rethink on what is necessary o need to try radically different model for shared storage Parallel Data Lab - PDSW 2015 Page 32