EFF-IO M7.5 Demo. Semantic Migration of Multi-dimensional Arrays

Transcription

1 EFF-IO M7.5 Demo Semantic Migration of Multi-dimensional Arrays John Bent, Sorin Faibish, Xuezhao Liu, Harriet Qui, Haiying Tang, Jerry Tirrell, Jingwang Zhang, Kelly Zhang, Zhenhua Zhang NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY EMC UNDER INTEL S SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL SECURITY, LLC WHO IS THE OPERATOR AND MANAGER OF LAWRENCE LIVERMORE NATIONAL LABORATORY UNDER CONTRACT NO. DE-AC52-07NA27344 WITH THE U.S. DEPARTMENT OF ENERGY. THE UNITED STATES GOVERNMENT RETAINS AND THE PUBLISHER, BY ACCEPTING THE ARTICLE OF PUBLICATION, ACKNOWLEDGES THAT THE UNITED STATES GOVERNMENT RETAINS A NON-EXCLUSIVE, PAID-UP, IRREVOCABLE, WORLD-WIDE LICENSE TO PUBLISH OR REPRODUCE THE PUBLISHED FORM OF THIS MANUSCRIPT, OR ALLOW OTHERS TO DO SO, FOR UNITED STATES GOVERNMENT PURPOSES. THE VIEWS AND OPINIONS OF AUTHORS EXPRESSED HEREIN DO NOT NECESSARILY REFLECT THOSE OF THE UNITED STATES GOVERNMENT OR LAWRENCE LIVERMORE NATIONAL SECURITY, LLC. 1

2 Demo System Configuration IONS: Four IONs lola-[12-15].hpdd.intel.com 8 SSD burst buffers on each DAOS: Global /mnt/daos mount on each DAOS-lustre MDS(lola-2);OSTS(lola-[16-19]/8xSSD) BB s: Local /mnt/lustre[1-4] on each ION All IONs have lustre cross-mounts to all BB s CN s: Four CNs available via Mercury lola-[20,25-27] Our demo runs directly on ION s though without HDF MPI job making calls directly to IOD API 2

3 Demo System Configuration CN1 lola-20 CN2 lola-25 CN3 lola-26 CN4 lola-27 M7.5 App is not using any App CN s. Running App test suite as App MPI HDF5 HDF5 HDF5 FShip-c job FShip-c across the IONs. FShip-c HDF5 FShip-c FShip-server FShip-server FShip-server FShip-server IOD-1 IOD-2 IOD-3 IOD-4 lola-12 lola-13 lola-14 lola-15 /lustre1 8xSSD /lustre2 8xSSD /lustre3 8xSSD /lustre4 8xSSD /DAOS /DAOS /DAOS /DAOS /lustre2,/lustre3,/lustre4 /lustre1,/lustre3,/lustre4 /lustre1,/lustre2,/lustre4 /lustre1,/lustre2,/lustre3 /lustre1/.plfs cannonical /lustre1/.plfs shadow /lustre1/.plfs cannonical /lustre2/.plfs shadow /lustre1/.plfs cannonical /lustre3/.plfs shadow /lustre1/.plfs cannonical /lustre4/.plfs shadow /DAOS Lustre/zfs MDS(lola-2);OSTS(lola-[16-19]/8xSSD) 3

4 Demo scope Semantic migration of arrays Persist (IOD -> DAOS) Fetch (DAOS -> IOD) Dimensional sequence transformation Changing the access dimension Reading from a persisted / purge array i.e. Reading directly from DAOS Query layout / sharding 4

5 Initialization 1/2 Script started on Fri 21 Mar :21:30 AM PDT M7.5_demo]#./M7.5_demo1.sh Cleaning the BB's and DAOS to return to initial state. M7.5 demonstration 1 -- Array Migration. will run M7.5_case1_array_migration_ops on 4 IODS: lola-12 lola-13 lola-14 lola-15 > mpirun -np 4 -hosts lola-12,lola-13,lola-14,lola-15 -c demo_config -t M7.5_case1_array_migration_ops Fri Mar 21 09:21:41 PDT 2014 ********* Test Configuration ********* inputfile = M7.5_case1_array_migration_ops nproc = 4 all_async = True 5

6 Initialization 2/2 ========================================================================= # Demonstration: Array Migration, Data Reorganization and Layout Discovery # # This demonstration will cover these elements: # Migration of array objects using structural descriptions between DAOS and IODs # - Will be shown in both directions # Reorganization of data layout during migration for array objects # - E.g. Migrate using a layout which specifies a different dimension sequence # Layout query of array objects # - E.g. Returning structure set to place data when migrating to DAOS # # Out of scope demonstration elements: # - Similar functionality for KV objects and blob objects will not be # demonstrated at this time # ========================================================================= 6

7 Step 1: Create and open container Running command: * contopen /containera create Call iod_container_open(), path: /containera, open_mode: 0x83... Finish to open container - passed (rc = 0) 7

8 Step : Skip Transaction 0 Running command: 0 transstart 0 write Call iod_trans_start(), tid: 0x0, num_ranks: 0, mode: 0x2... Finish to start transaction - passed (rc = 0) # ######## Step finish TID0 for write # Command format: tid transfinish flag # flag only meaningful for writing transaction. 'normal' for commonly finish > Press enter to continue... Running command: 0 transfinish normal Call iod_trans_finish(), tid: 0x0, num_ranks: 0, flag: 0 Finish to finish transaction - passed (rc = 0) 8

9 Step : Start TID=1, create array Running command: 1 transstart 0 write Call iod_trans_start(), tid: 0x1, num_ranks: 0, mode: 0x2... Finish to start transaction - passed (rc = 0) # ######## Step in TID1, create ARRAY object: cell_size=32 num_dims=3 (16 X 32 X 64) > Press enter to continue... Leader mode Running command: 1 create array array_obj ,32,64 16 Call iod_obj_create(), tid: 0x1, type: 0x3... Array struct: cell_size: 512, num_dims: 3, firstdim_max: 16, current_dims: 16,32,64 Finish to create object - passed (rc = 0) 16MB array (512x16x32x64) 9

10 16 x 32 x 64 Multi-dimensional Array

11 Step 2.5: Write into array # Call iod_array_write() to write the ARRAY object # start is [0,0,0], stride is [8,8,8], count is [2,4,8], block is [8,8,8] # > Press enter to continue... Running command: 1 write array_obj 0,0,0 8,8,8 2,4,8 8,8,8 Rank 0 ID 0: start[ 0,0,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 1 ID 0: start[ 0,8,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 2 ID 0: start[ 0,16,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 3 ID 0: start[ 0,24,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Call iod_array_write(), oh: 123, tid: 0x1 Rank 0 ID 0: iod_array_write() returns 0 Rank 1 ID 0: iod_array_write() returns 0 Rank 3 ID 0: iod_array_write() returns 0 Rank 2 ID 0: iod_array_write() returns 0 Finish to write array - passed (rc = 0) 11

12 Write Pattern Rank 3 Rank 2 Rank 1 Rank 0 12

13 Step 2.6: Prepare semantic persist # location "DAOS" (i.e. IOD_LOC_CENTRAL) # type currently only support stripped layout (i.e.iod_layout_striped). # target_start -- the offset in DAOS shard list, currently IOD ignores this value (chooses one internally for load balance). # target_num ---- number of storage targets (i.e. DAOS shards). If 0, it defaults to all shards. # stripe_size --- resharding granularity. The number of cells for contiguous layout array. # dims_seq For contigous layout, it is an array with num_dims size (i.e. a 4D array may have # [3][2][1][0] as its logical dimension sequence) Running command: 1 setlayout array_obj DAOS,striped,0,2,128,<1,2,0> Call iod_obj_set_layout(), tid: 0x1 Layout to set: location type target_start target_num stripe_size dims_seq DAOS striped , 2, 0 Finish to set object layout - passed (rc = 0) 13

14 Step 2.7: Commit TID=1 Running command: 1 transfinish normal Call iod_trans_finish(), tid: 0x1, num_ranks: 0, flag: 0 Finish to finish transaction - passed (rc = 0) 14

15 Step 3.2: Query Eventual Persist Layout Running command: 1 getlayout array_obj At tid 0x1: dump IOD returned layout: location type target_start target_num stripe_size dims_seq DAOS striped , 2, 0 Rank 0 ID 0: get_layout verification succeed Finish to get object layout - passed (rc = 0) 15

16 Step 3.3: Query Semantic Shards on IONs Running command: 1 querymap array_obj Call iod_obj_query_map(), tid: 0x1 Rank 0 in tid 0x1: dump IOD returned map: obj_id type n_range b6056d ARRAY 64 range: 0, start: 0, 0, 0, end: 0, 7, 63, n_cell: 512, loc: /mnt/lustre1/.plfs shadow range: 1, start: 0, 8, 0, end: 0, 15, 63, n_cell: 512, loc: /mnt/lustre2/.plfs shadow range: 2, start: 0, 16, 0, end: 0, 23, 63, n_cell: 512, loc: /mnt/lustre3/.plfs shadow range: 3, start: 0, 24, 0, end: 0, 31, 63, n_cell: 512, loc: /mnt/lustre4/.plfs shadow range: 4, start: 1, 0, 0, end: 1, 7, 63, n_cell: 512, loc: /mnt/lustre1/.plfs shadow range: 5, start: 1, 8, 0, end: 1, 15, 63, n_cell: 512, loc: /mnt/lustre2/.plfs shadow range: 6, start: 1, 16, 0, end: 1, 23, 63, n_cell: 512, loc: /mnt/lustre3/.plfs shadow range: 7, start: 1, 24, 0, end: 1, 31, 63, n_cell: 512, loc: /mnt/lustre4/.plfs shadow <snip> range: 56, start: 14, 0, 0, end: 14, 7, 63, n_cell: 512, loc: /mnt/lustre1/.plfs shadow range: 57, start: 14, 8, 0, end: 14, 15, 63, n_cell: 512, loc: /mnt/lustre2/.plfs shadow range: 58, start: 14, 16, 0, end: 14, 23, 63, n_cell: 512, loc: /mnt/lustre3/.plfs shadow range: 59, start: 14, 24, 0, end: 14, 31, 63, n_cell: 512, loc: /mnt/lustre4/.plfs shadow range: 60, start: 15, 0, 0, end: 15, 7, 63, n_cell: 512, loc: /mnt/lustre1/.plfs shadow range: 61, start: 15, 8, 0, end: 15, 15, 63, n_cell: 512, loc: /mnt/lustre2/.plfs shadow range: 62, start: 15, 16, 0, end: 15, 23, 63, n_cell: 512, loc: /mnt/lustre3/.plfs shadow range: 63, start: 15, 24, 0, end: 15, 31, 63, n_cell: 512, loc: /mnt/lustre4/.plfs shadow Result of write pattern 16

17 Semantic Sharding on IONs ION 4 ION 3 ION 2 ION 1 17

18 Step 3.3: Query Semantic Shards on IONs # Verify data in BB s: [root@lola-3 scripts]# plfs_query /tmp/iod_plfs/containera/ /1 \ grep dropping.data xargs du -h 4.0M /mnt/lustre7/.plfs shadow///containera/ /1/hostdir.1/droppin g.data lola-7.lola.whamcloud.com M /mnt/lustre6/.plfs shadow///containera/ /1/hostdir.2/droppin g.data lola-6.lola.whamcloud.com M /mnt/lustre5/.plfs shadow///containera/ /1/hostdir.3/droppin g.data lola-5.lola.whamcloud.com M /mnt/lustre3/.plfs shadow///containera/ /1/hostdir.5/droppin g.data lola-3.lola.whamcloud.com

19 Step 4.1: Persist container at TID=1 Running command: 1 persist Call iod_trans_persist(), coh: 25, tid : 0x1... Finish to persist container - passed (rc = 0) # Verify proper scatter-gather migration: > [root@lola-3 scripts]# egrep 'reqmsg ACK' /tmp/iod.debug.log head 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO handling reqmsg, type: 1, msg_len: 63.(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO ENTER: _container_ops_trans_start_reqmsg_handler (rank 0)(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO EXIT : _container_ops_trans_start_reqmsg_handler, rc: 0.(rank 0)(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO handling reqmsg, type: 1, msg_len: 63.(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO ENTER: _container_ops_trans_start_reqmsg_handler (rank 0)(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO EXIT : _container_ops_trans_start_reqmsg_handler, rc: 0.(rank 0)(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO sending ACK req_id 2, result: 0, piggyback: ffffffffffffffff from 0 to 0... (rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO handling reqmsg, type: 2, msg_len: 32.(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO ACK arrived, from IOD rank 0, req_id: 2, result: 0.(rank 0) 2014/03/23-16:52:33.07 lola-3 plfs[107367] IOD INFO ENTER: iod_reqmsg_ack_handler (rank 0)(rank 0) 19

20 Step 4.1: Persist container at TID=1 # Verify data on DAOS: > [root@lola-3 scripts]#./daos_query.sh /mnt/daos/containera [root@lola-3 scripts]# daos_query.sh /mnt/daos/containera /scratch/iod/tests/output/daos_query.sh/mar ### ls -l /mnt/daos/containera -rw-r--r-- 1 root root Mar 23 19:55 /mnt/daos/containera ### du -h /mnt/daos/containera 27M /mnt/daos/containera #### daos_ctl run -c /mnt/daos/containera Cor,Cq,Eq,Cc: 1 HCE is 1 #### daos_ctl run -c /mnt/daos/containera Cor,Cq,Eq,Cc: 8 8 shards #### daos_ctl run -c /mnt/daos/containera Cor,Sq1:0,Sl1:0,Cc: 141, Shard 0 has 141, objects and uses (15 MB) Objects in shard 0: [0: ] [0: ] [0: ] [0: ] [0: ] #### daos_ctl run -c /mnt/daos/containera Cor,Sq1:1,Sl1:1,Cc: Shard 1 has 135, objects and uses (5 MB) Objects in shard 1: [1: ] <snip> #### daos_ctl run -c /mnt/daos/containera Cor,Sq1:7,Sl1:7,Cc: Shard 7 has 135, objects and uses (5 MB) Objects in shard 7: [7: ] 20

21 Semantic Sharding on DAOS KEY Shard0 Shard1 Shard2 Shard3 Shard4 Shard5 Shard6 Shard7 21

22 Step 4.2: Read array (from IONs) Running command: 1 read array_obj 0,0,0 8,8,8 2,4,8 8,8,8 Rank 0 ID 0: start[ 0,8,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 1 ID 0: start[ 0,16,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 2 ID 0: start[ 0,24,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 3 ID 0: start[ 0,0,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Call iod_array_read(), oh: 234, tid: 0x1 Rank 2 ID 0: iod_array_read() returns 0 Rank 1 ID 0: iod_array_read() returns 0 Rank 0 ID 0: iod_array_read() returns 0 Finish to read array - passed (rc = 0) Rank 3 ID 0: iod_array_read() returns 0 22

23 Reading Array (from IONs) Rank 2 Rank 1 Rank 0 Rank 3 ION 4 ION 3 ION 2 ION 1 23

24 Step 4.3/4.4: Purge Array / Query map Running command: 1 purge array_obj Call iod_obj_purge(), oh: 242, tid: 0x1... Finish to purge object - passed (rc = 0) Running command: 1 querymap array_obj Call iod_obj_query_map(), tid: 0x1 Rank 0 in tid 0x1: dump IOD returned map: obj_id type n_range b6056d ARRAY 0 Finish to query object map - passed (rc = 0) No data is IOD resident # Verify data purged: [root@lola-3 scripts]# plfs_query /tmp/iod_plfs/containera/ /1 \ grep dropping.data [root@lola-3 scripts]# 24

25 Step 4.5: Read Array (from DAOS) Running command: 1 read array_obj 0,0,0 8,8,8 2,4,8 8,8,8 Rank 0 ID 0: start[ 0,8,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 1 ID 0: start[ 0,16,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 2 ID 0: start[ 0,24,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 3 ID 0: start[ 0,0,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Call iod_array_read(), oh: 258, tid: 0x1 Rank 0 ID 0: iod_array_read() returns 0 Finish to read array - passed (rc = 0) Rank 3 ID 0: iod_array_read() returns 0 Rank 1 ID 0: iod_array_read() returns 0 Rank 2 ID 0: iod_array_read() returns 0 25

26 Step 4.5: Read Array (from DAOS) 26

27 Step 5.1/5.2: Fetch Array into 2 replicas # layout contains: location,type,target_start,target_num,strip_size,<dims_seq> # location "BB" (i.e. IOD_LOC_BB) # type only support stripped layout(i.e.iod_layout_striped) # target_start --- the beginning IOD # target_num the number of IODs. If 0, it defaults to all IODs # stripe_size ---- resharding granularity. The number of cells for contiguous layout array # dims_seq For contiguous layout, it is an array with num_dims size # # ######## Step prefetch object 1st: using one IOD and default dims_seq > Press enter to continue... Running command: 1 fetch array_obj tag1 BB,striped,0,1,64,<> Call iod_obj_fetch(), tid: 0x1... Rank 0: check all information saved for fetch: fake_tag tag location type target_start target_num stripe_size dims_seq tag1 0x BB striped (nil) Finish to fetch object - passed (rc = 0) Running command: 1 fetch array_obj tag2 BB,striped,0,0,1024,<2,0,1> Call iod_obj_fetch(), tid: 0x1... Rank 0: check all information saved for fetch: fake_tag tag location type target_start target_num stripe_size dims_seq tag1 0x BB striped (nil) tag2 0x BB striped , 0, 1 Finish to fetch object - passed (rc = 0) 27

28 Step 5.3: Query Replica 1 Shards on IONs Running command: * querymap array_obj tag1 Call iod_obj_query_map(), tag: 0x Rank 0 in tid 0x : dump IOD returned map: obj_id type n_range b6056d ARRAY 1 range: 0, start: 0, 0, 0, end: 15, 31, 63, n_cell: 32768, loc: /mnt/lustre3/.plfs shadow Finish to query object map - passed (rc = 0) 28

29 Step 5.4: Query Replica 2 Shards on IONs # ######## Step Query object's map: 2nd prefetched copy (should striped on all IODs) > Press enter to continue... Running command: * querymap array_obj tag2 Call iod_obj_query_map(), tag: 0x Rank 0 in tid 0x : dump IOD returned map: obj_id type n_range b6056d ARRAY 32 range: 0, start: 0, 0, 0, end: 15, 31, 1, n_cell: 1024, loc: /mnt/lustre1/.plfs shadow range: 1, start: 0, 0, 2, end: 15, 31, 3, n_cell: 1024, loc: /mnt/lustre2/.plfs shadow range: 2, start: 0, 0, 4, end: 15, 31, 5, n_cell: 1024, loc: /mnt/lustre3/.plfs shadow range: 3, start: 0, 0, 6, end: 15, 31, 7, n_cell: 1024, loc: /mnt/lustre4/.plfs shadow range: 4, start: 0, 0, 8, end: 15, 31, 9, n_cell: 1024, loc: /mnt/lustre1/.plfs shadow range: 5, start: 0, 0, 10, end: 15, 31, 11, n_cell: 1024, loc: /mnt/lustre2/.plfs shadow range: 6, start: 0, 0, 12, end: 15, 31, 13, n_cell: 1024, loc: /mnt/lustre3/.plfs shadow range: 7, start: 0, 0, 14, end: 15, 31, 15, n_cell: 1024, loc: /mnt/lustre4/.plfs shadow <snip> range: 28, start: 0, 0, 56, end: 15, 31, 57, n_cell: 1024, loc: /mnt/lustre1/.plfs shadow range: 29, start: 0, 0, 58, end: 15, 31, 59, n_cell: 1024, loc: /mnt/lustre2/.plfs shadow range: 30, start: 0, 0, 60, end: 15, 31, 61, n_cell: 1024, loc: /mnt/lustre3/.plfs shadow range: 31, start: 0, 0, 62, end: 15, 31, 63, n_cell: 1024, loc: /mnt/lustre4/.plfs shadow Finish to query object map - passed (rc = 0) 29

30 Replica 1 Replica 2 KEY ION 1 ION 2 ION 3 ION 4 30

31 Step 5.5: Read Replica 1 # ######## Step 5.5 read data of the 1st prefetched copy with its tag Check prefetched copy of object array_obj(0x b6056d) at TAG = 0x Rank 0 ID 0: start[ 0,8,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 1 ID 0: start[ 0,16,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 2 ID 0: start[ 0,24,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 3 ID 0: start[ 0,0,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Call iod_array_read(), oh: 330, tid: 0x Rank 0 ID 0: iod_array_read() returns 0 Rank 0 ID 0: array IO data verification succeed Finish to read array - passed (rc = 0) Finish to readprefetched copy - passed (rc = 0) Rank 2 ID 0: iod_array_read() returns 0 Rank 1 ID 0: iod_array_read() returns 0 Rank 3 ID 0: iod_array_read() returns 0 31

32 Step 5.6: Read Replica 2 # ######## Step 5.6 read data of the 2nd prefetched copy with its tag Check prefetched copy of object array_obj(0x b6056d) at TAG = 0x Rank 0 ID 0: start[ 0,8,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 1 ID 0: start[ 0,16,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 2 ID 0: start[ 0,24,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Rank 3 ID 0: start[ 0,0,0 ], stride[ 8,8,8 ], count[ 2,1,8 ], block[ 8,8,8 ] Call iod_array_read(), oh: 338, tid: 0x Rank 1 ID 0: iod_array_read() returns 0 Rank 2 ID 0: iod_array_read() returns 0 Rank 0 ID 0: iod_array_read() returns 0 Finish to read array - passed (rc = 0) Finish to readprefetched copy - passed (rc = 0) 32

33 Reading Replica 1 KEY ION 1 ION 2 ION 3 ION 4 33

34 Reading Replica 2 KEY ION 1 ION 2 ION 3 ION 4 34

35 Step 5.7/6: Cleanup / Fini # ######## Step finish TID1 for read > Press enter to continue... Running command: 1 transfinish normal Call iod_trans_finish(), tid: 0x1, num_ranks: 0, flag: 0 Finish to finish transaction - passed (rc = 0) # # ######## Step 6: close the container and quit. > Press enter to continue... Running command: * contclose Call iod_container_close(), handle: Finish to close container - passed (rc = 0) > Press enter to continue... Running command: * quit M7.5_case1_array_migration_ops: overall_result============= 0(Success) M7.5_case1_array_migration_ops: overall_result============= 0(Success) M7.5_case1_array_migration_ops: overall_result============= 0(Success) M7.5_case1_array_migration_ops: overall_result============= 0(Success) 35

36 Semantic Migration Summary Original DAOS Replica 1 Replica 2 36

37

38 EFF-IO M7.5 Demo Semantic Migration of Multi-dimensional Arrays John Bent, Sorin Faibish, Xuezhao Liu, Harriet Qui, Haiying Tang, Jerry Tirrell, Jingwang Zhang, Kelly Zhang, Zhenhua Zhang 38