Milestone 6.3: Basic Analysis Shipping Demonstration

Transcription

1 The HDF Group Milestone 6.3: Basic Analysis Shipping Demonstration Ruth Aydt, Mohamad Chaarawi, Ivo Jimenez, Quincey Koziol, Jerome Soumagne 12/17/2013 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL SECURITY, LLC WHO IS THE OPERATOR AND MANAGER OF LAWRENCE LIVERMORE NATIONAL LABORATORY UNDER CONTRACT NO. DE-AC52-07NA27344 WITH THE U.S. DEPARTMENT OF ENERGY. THE UNITED STATES GOVERNMENT RETAINS AND THE PUBLISHER, BY ACCEPTING THE ARTICLE OF PUBLICATION, ACKNOWLEDGES THAT THE UNITED STATES GOVERNMENT RETAINS A NON-EXCLUSIVE, PAID-UP, IRREVOCABLE, WORLD-WIDE LICENSE TO PUBLISH OR REPRODUCE THE PUBLISHED FORM OF THIS MANUSCRIPT, OR ALLOW OTHERS TO DO SO, FOR UNITED STATES GOVERNMENT PURPOSES. THE VIEWS AND OPINIONS OF AUTHORS EXPRESSED HEREIN DO NOT NECESSARILY REFLECT THOSE OF THE UNITED STATES GOVERNMENT OR LAWRENCE LIVERMORE NATIONAL SECURITY, LLC The HDF Group 1

2 Basic Analysis Shipping Objectives Locally enumerate/run analysis on objects that are local to the IO node or storage server node Perform operations as close as possible to the data to minimize data movement costs Easily modify analysis that locally runs on the node Gather results from local enumeration/analysis Main concepts Query: retrieves from an object specific elements / selection that match certain condition(s) Split: parallel and local operation on the data selected by the query Combine: gather data generated by the split operation 2

3 Basic Analysis Shipping Restrictions for this quarter: Queries only operate on dataset elements Ship analysis only to IO nodes (will do storage server nodes in next quarters) 3

4 Architecture Master Filename + Object name + Query + User-defined Python routines Analysis Shipping Client (Mercury client) Python Type "help", "copyright", "credits" or "license" for more information. >>> import h5py_ff >>> h5py_ff.h5asexecute(filename, obj_name, query, split, combine) 4

5 Architecture Master Retrieve storage layout Analysis Shipping Client (Mercury client) Python Type "help", "copyright", "credits" or "license" for more information. >>> import h5py_ff >>> h5py_ff.h5asexecute(filename, obj_name, query, split, combine) 5

6 Architecture Farm split to servers that own part of the object(s) Master Analysis Shipping Client (Mercury client) Python Type "help", "copyright", "credits" or "license" for more information. >>> import h5py_ff >>> h5py_ff.h5asexecute(filename, obj_name, query, split, combine) 6

7 Architecture Master Combine data Analysis Shipping Client (Mercury client) Python Type "help", "copyright", "credits" or "license" for more information. >>> import h5py_ff >>> h5py_ff.h5asexecute(filename, obj_name, query, split, combine) 7

8 Query API /* Query type */ typedef enum H5Q_type_t { H5Q_TYPE_DATA_ELEM, /* selects data elements */ H5Q_TYPE_ATTR_NAME, /* selects attributes */ H5Q_TYPE_LINK_NAME /* selects objects */ } H5Q_type_t; /* Query match conditions */ typedef enum H5Q_match_op_t { H5Q_MATCH_EQUAL, /* equal */ H5Q_MATCH_NOT_EQUAL, /* not equal */ H5Q_MATCH_LESS_THAN, /* less than */ H5Q_MATCH_GREATER_THAN /* greater than */ } H5Q_match_op_t; /* Query combine operators */ typedef enum H5Q_combine_op_t { H5Q_COMBINE_AND, H5Q_COMBINE_OR } H5Q_combine_op_t; 8

9 Query API /* Used on analysis shipping client */ H5_DLL hid_t H5Qcreate(H5Q_type_t query_type, H5Q_match_op_t match_op,...); H5_DLL herr_t H5Qclose(hid_t query_id); H5_DLL hid_t H5Qcombine(hid_t query_id1, H5Q_combine_op_t combine_op, hid_t query_id2); /* Used on server */ H5_DLL herr_t H5Qapply(hid_t query_id, hbool_t *result, hid_t type_id, const void *elem); /* Encode / decode */ H5_DLL herr_t H5Qencode(hid_t query_id, void *buf, size_t *nalloc); H5_DLL hid_t H5Qdecode(const void *buf); 9

10 Demo example - Lola Dataset D1 (512 x 3) CN 0 (lola-12) Process 1 ION 0 (lola-18) /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) /mnt/lustre/bb/lola19/.plfs shadow 10

11 Demo example - Lola D1 Write dataset using hyperslab selection filename: eff_analysis_file.h5 obj_name: D1 ION 0 (lola-18) CN 0 (lola-12) Process 1 /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) /mnt/lustre/bb/lola19/.plfs shadow 11

12 Demo example - Lola Send an analysis shipping request from filename: eff_analysis_file.h5 obj_name: D1 query: (39.1 < x < 42.1) OR (295 < x < 298) split: average combine: average CN 0 (lola-12) ION 0 (lola-18) /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) Process 1 /mnt/lustre/bb/lola19/.plfs shadow 12

13 Demo example - Lola Send an analysis shipping request from filename: eff_analysis_file.h5 obj_name: D1 query: (39.1 < x < 42.1) OR (295 < x < 298) split: average combine: average CN 0 (lola-12) ION 0 (lola-18) iod_obj_query_map() /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) Process 1 /mnt/lustre/bb/lola19/.plfs shadow 13

14 Demo example - Lola Send an analysis shipping request from filename: eff_analysis_file.h5 obj_name: D1 query: (39.1 < x < 42.1) OR (295 < x < 298) split: average combine: average CN 0 (lola-12) Element 40 matches query Farm work and Execute query ION 0 (lola-18) /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) Process 1 Element 296 matches query /mnt/lustre/bb/lola19/.plfs shadow 14

15 Demo example - Lola Send an analysis shipping request from filename: eff_analysis_file.h5 obj_name: D1 query: (39.1 < x < 42.1) OR (295 < x < 298) split: average combine: average CN 0 (lola-12) Split average: 41.0 HDF5->NumPy Execute split ION 0 (lola-18) /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) Process 1 Split average: /mnt/lustre/bb/lola19/.plfs shadow 15

16 Demo example - Lola Send an analysis shipping request from filename: eff_analysis_file.h5 obj_name: D1 query: (39.1 < x < 42.1) OR (295 < x < 298) split: average combine: average CN 0 (lola-12) Combined average: NumPy->HDF5 Gather Data HDF5->NumPy Execute combine ION 0 (lola-18) /mnt/lustre/bb/lola18/.plfs shadow ION 1 (lola-19) Process 1 /mnt/lustre/bb/lola19/.plfs shadow 16

17 Demo Analysis Shipping $ mpirun np 2./h5ff_client_analysis Analysis Execute on file eff_analysis_file.h5 Object D1 (0) Calling iod_container_open on eff_analysis_file.h5 (0) coh 281 objoh 378 objid rtid 2 (0) Calling iod_obj_query_map (0) range: 0, start: 0 0, end: 255 2, n_cell: 768, loc: /mnt/lustre/bb/lola19/.plfs shadow (0) range: 1, start: 256 0, end: 511 2, n_cell: 768, loc: /mnt/lustre/bb/lola18/.plfs shadow (0) Server 1 owns this object (0) Server 0 owns this object Dim 0: start 256 stride 256 block 256 count 1 Dim 1: start 0 stride 3 block 3 count 1 Dim 0: start 0 stride 256 block 256 count 1 Dim 1: start 0 stride 3 block 3 count 1 (0) Element 296 matches query (0) Element 297 matches query (0) Applying split on data (1) Element 40 matches query (1) Element 42 matches query (1) Applying split on data Split sum: 1779 Split average: Split sum: 369 Split average: (1) Transferring split data back to master (0) Applying combine on data Combined sum: Combined average:

18 Conclusion Future Work Add python layer with h5py to call H5ASexecute from python interpreter Cython to automatically generate Python/C wrappers Support queries on attributes and links Run on both IO and storage server nodes Add support inside Mercury for usage of multiple network transports at the same time (external workstation with TCP, internal with IB or other specific interconnect protocol) 18

19 Questions 19