Milestone 8.1: HDF5 Index Demonstration

Transcription

1 The HDF Group Milestone 8.1: HDF5 Index Demonstration Ruth Aydt, Mohamad Chaarawi, Quincey Koziol, Aleksandar Jelenak, Jerome Soumagne 06/30/2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY THE HDF GROUP UNDER INTEL S SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL SECURITY, LLC WHO IS THE OPERATOR AND MANAGER OF LAWRENCE LIVERMORE NATIONAL LABORATORY UNDER CONTRACT NO. DE-AC52-07NA27344 WITH THE U.S. DEPARTMENT OF ENERGY. THE UNITED STATES GOVERNMENT RETAINS AND THE PUBLISHER, BY ACCEPTING THE ARTICLE OF PUBLICATION, ACKNOWLEDGES THAT THE UNITED STATES GOVERNMENT RETAINS A NON-EXCLUSIVE, PAID-UP, IRREVOCABLE, WORLD-WIDE LICENSE TO PUBLISH OR REPRODUCE THE PUBLISHED FORM OF THIS MANUSCRIPT, OR ALLOW OTHERS TO DO SO, FOR UNITED STATES GOVERNMENT PURPOSES. THE VIEWS AND OPINIONS OF AUTHORS EXPRESSED HEREIN DO NOT NECESSARILY REFLECT THOSE OF THE UNITED STATES GOVERNMENT OR LAWRENCE LIVERMORE NATIONAL SECURITY, LLC The HDF Group 1

2 Background Analysis Shipping Queries sent to I/O node for enumeration of the data that is present locally on each node Queries operate on dataset elements Indexing support Add index to datasets Added indexing capabilities to H5Datasets, using a plugin mechanism Indexing packages may create, write and read index information anonymously within the H5Container, storing it outside of group hierarchy 2

3 Previous Indexing Demo What we demo ed (brute force index) A simple and not-to-smart index plugin that verifies access to H5Dataset information within an H5Container Index creation, access and updates within that container An application using a chosen plugin to create an index for an H5Dataset, querying its associated index, and verification that the index is used to help fulfill the query inside the view H5D Application H5Xplugin H5Xdummy H5 Container H5X 3

4 Indexing Demo Two indexing plugins added FastBit (bitmap index) ALACRITY (inverted index) Demo using these two plugins Creation of index / Answer of query Data update / Index update / Answer of query Current Limitation Incremental update of data is not supported by indexing packages Entire data has to be read to rebuild index 4

5 Acknowledgments FastBit team (LBNL) John Wu (many thanks for modifications to FastBit!) Suren Byna Bin Dong Prabhat ALACRITY team (NCSU) Nagiza Samatova David Boyuka Xiaocheng Zou 5

6 Index Plugin Class typedef struct { unsigned version; /* Version number of the index plugin class struct */ /* (Should always be set to H5X_CLASS_VERSION, which * may vary between releases of HDF5 library) */ unsigned id; /* Index ID (assigned by The HDF Group, for now) */ const char *idx_name; /* Index name (for debugging only, currently) */ H5X_type_t type; /* Type of data indexed by this plugin */ /* Callbacks */ void *(*create)(hid_t file_id, hid_t dataset_id, hid_t xcpl_id, hid_t xapl_id, size_t *metadata_size, void **metadata); herr_t (*remove)(hid_t file_id, hid_t dataset_id, size_t metadata_size, void *metadata); void *(*open)(hid_t file_id, hid_t dataset_id, hid_t xapl_id, size_t metadata_size, void *metadata); herr_t (*close)(void *idx_handle); herr_t (*pre_update)(void *idx_handle, hid_t dataspace_id, hid_t xxpl_id); herr_t (*post_update)(void *idx_handle, const void *buf, hid_t dataspace_id, hid_t xxpl_id); herr_t (*query)(void *idx_handle, hid_t query_id, hid_t xxpl_id, hid_t *dataspace_id); herr_t (*refresh)(void *idx_handle, size_t *metadata_size, void **metadata); } H5X_class_t; 6

7 Index API H5_DLL herr_t H5Xregister(const H5X_class_t *idx_class); H5_DLL herr_t H5Xunregister(unsigned plugin_id); H5_DLL herr_t H5Xcreate(hid_t file_id, unsigned plugin_id, hid_t scope_id, hid_t xcpl_id); H5_DLL herr_t H5Xcreate_ff(hid_t file_id, unsigned plugin_id, hid_t scope_id, hid_t xcpl_id, hid_t trans_id, hid_t estack_id); H5_DLL herr_t H5Xremove(hid_t file_id, unsigned plugin_id, hid_t scope_id); H5_DLL herr_t H5Xremove_ff(hid_t file_id, unsigned plugin_id, hid_t scope_id, hid_t trans_id, hid_t estack_id); H5_DLL herr_t H5Xget_count(hid_t scope_id, hsize_t *idx_count); H5_DLL herr_t H5Xget_count_ff(hid_t scope_id, hsize_t *idx_count, hid_t rcxt_id, hid_t estack_id); 7

8 FastBit Plugin Use FastBit in-memory API <iapi.h> Modifications to API to integrate into plugin API fastbit_iapi_build_index (build index) fastbit_iapi_decontruct_index (serialize index into key, offset and bitmap arrays) fastbit_iapi_attach_index (rebuild index from arrays) Create callback Build index from entire data Deconstruct index into arrays Create anonymous HDF5 datasets from arrays and store array dataset IDs into plugin metadata 8

9 FastBit Plugin Query callback: Reconstruct index if not reconstructed Create fastbit selection from query Evaluate selection Get coordinates from selection Return dataspace selection from coordinates Post-update callback: Gather/scatter data to merge new data with original data Update index and arrays used to store it (Refresh index metadata) 9

10 ALACRITY Plugin Use main ALACRITY API Create callback Build index from entire data using ALencode Store ALACRITY metadata and ALACRITY index into 2 anonymous datasets Store dataset IDs into plugin metadata Query callback: Read ALACRITY metadata if not present Translate HDF5 query into range Find bins that are touched by the query (all elements in the query range will fall into these bins) If bins are found, then read index from bin offsets and get coordinates, return dataspace selection from coordinates 10

11 ALACRITY Plugin Post-update callback: Gather/scatter data to merge new data with original data Update index and arrays used to store it (Refresh index metadata) 11

12 Demo example Dataset D0 (512 x 3) Create datasets using H5Dcreate/H5Dwrite etc filename: eff_index_file.h5 obj_name: D CNs ION Process 0 HDF5/IOD Server (Mercury Server) D

13 Demo example CNs Create index using H5Xcreate (call index create, create anonymous datasets for index information) filename: eff_index_file.h5 obj_name: D0 ION IDX0 Process 0 HDF5/IOD Server (Mercury Server) D0 set_index_info with index metadata created (stored on server using KVs for FF) 13

14 Demo example H5Dquery (call index query) filename: eff_index_file.h5 obj_name: D0 simple query: 38.8 < x < 42.1 Should return [39:0] [42:2] CNs ION IDX0 Process 0 HDF5/IOD Server (Mercury Server) D0 14

15 Demo example Dataset D0 (512 x 3) CNs Process 0 Process 1 Modify data and do incremental update (call post_update) / 1 write per transaction per process filename: eff_index_file.h5 obj_name: D0 ION HDF5/IOD Server (Mercury Server) D0 IDX0 15

16 Demo example H5Dquery (call index query) filename: eff_index_file.h5 obj_name: D0 simple query: 38.8 < x < 42.1 Should return [39:0] [511:2] CNs ION IDX0 Process 0 HDF5/IOD Server (Mercury Server) D0 16

17 Performance Number of Tuples , ,288 Brute Force (ms) FastBit (ms) ALACRITY (ms)

18 Time (ms) Performance Brute force FastBit ALACRITY k 8k 32k 128k 512k Number of tuples 18

19 Demo time 19

20 Conclusion Indexing capabilities integrated into analysis shipping Two Third-Party Index Package Plugins: FastBit ALACRITY Still room for optimization Pass minimal query range to index plugins Query optimizer Real incremental update Better use of pre_update/post_update calls 20

21 Questions 21