COSC 6374 Parallel Computation. Scientific Data Libraries. Edgar Gabriel Fall Motivation

Transcription

1 COSC 6374 Parallel Computation Scientific Data Libraries Edgar Gabriel Fall 2013 Motivation MPI I/O is good It knows about data types (=> data conversion) It can optimize various access patterns in applications MPI I/O is bad It does not store any information about the data type A file written as MPI_INT can be read as MPI_DOUBLE in another application No information is stored about data layout, e.g. two-dimensional data set, extent of each dimension etc. 1

2 Scientific data libraries Handle data on a higher level Provide additional information (Metadata) Size and type of of data structure Data format Name Units Two widely used libraries available NetCDF HDF-5 HDF-5 Hierarchical Data Format (HDF) developed since 1988 at NCSA (University of Illinois) Has gone through a long history of changes, the recent version HDF-5 available since 1999 HDF-5 supports Very large files Parallel I/O interface Fortran, C, Java bindings 2

3 HDF-5 dataset Multi-dimensional array of basic data elements A dataset consists of Header + data Header consists of Name Datatype : basic (e.g. HDF_NATIVE_FLOAT) or compound dataypes Dataspace: defines size and shape of a multidimensional array. Dimensions can be fixed or unlimited. Storage layout: defines how multidimensional arrays are stored in file. Can be contiguous or chunked. Example of an HDF-5 file HDF5 tempseries.h5 { GROUP / { GROUP tempseries { DATASET height { DATATYPE { H5_STD_I32BE } DATASPACE ( ARRAY (4) (4) } DATA { 0, 50, 100, 150 } ATTRIBUTES units { DATATYPE { undefined string } DATASPACE { ARRAY (0) (0) } DATA { unable to print } } } DATASET temperature { DATATYPE { H5T_IEEE_F32BE } DATASPACE{ ARRAY( 3,8,4 ) (H5S_UNLIMITED, 8, 4) } DATA { } 3

4 Storage layout: contiguous vs. chunked contiguous chunked Advantages and disadvantages of chunking Accessing rows and columns require the same number of accesses Data can be extended into all dimensions Efficient storage of sparse arrays Can improve caching HDF-5 API HDF-5 naming convention All API functions start with an H5 The next character identifies category of functions H5F: functions handling files H5G: functions handling groups H5D: functions handling datasets H5S: functions handling dataspaces H5A: functions handling attributes A HDF-5 group is a collection of data sets Comparable to a directory in a UNIX-like file system 4

5 Writing a sequential HDF-5 file 1. Create the file 2. Create a group (opt.) 3. Define a dataspace 4. Define datatype 5. Create dataset 6. Add attributes h5file = H5Fcreate( ) group = H5Gcreate (h5file, ) tspace = H5Screate_simple(ndims, dims, maxdims ); ttype = H5T_IEEE_F32BE; tset = H5Dcreate (group, testset, ttype, tspace, ); tattr = H5Acreate (tset, units, H5T_C_S1, ) H5Awrite (tattr, H5T_C_S1, meter ); H5Dwrite(tset,H5T_IEEE_F32BE,,data); 7. Write data 8. Close all objects Reading an HDF-5 file structure of the file known 1. Open the file 2. Open the group 3. Open dataset in the group 4. Look up dimensions 5. Read data 6. Read attributes 7. Read comments 8. Close all objects h5file = H5Fopen( ) group = H5Gopen(h5file, tempseries ) tset = H5Dopen(group, temperature ); tspace = H5Dget_space( tset ); H5Sget_simple_extent_dims (tspace, dims, ); H5Dread(tset,H5T_IEEE_F32BE, ttype, tspace,, buffer); tattr = H5Aopen_name(tset, units ); attrtype = H5Aget_type ( tattr ); H5Aread(tattr,attrtype,attr); 5

6 Compound Datatypes Abstraction for user structures Has a fixed size Each member has its own name, datatype, reference, and byte offset h5type = H5Tcreate( H5T_class class, size_t size); H5Tinsert ( h5type, const char *name, off_t offset, hid_t field_id); Hyperslab A hyperslab is a portion of a dataset H5Sselect_hyperslab (hid_t space_id, H5S_seloper_t operator, const hssize_t *start, const hsize_t *stride, const hsize_t *count, const hsize_t *block); Operator: H5S_SELECT_SET, H5S_SELECT_OR Start: array determining the starting coordinates of the hyperslab Stride: array indicating which elements along a dimension are to be selected Count: array determining how many points to use in each dimension Block: array determining the size of the element block by the datatype 6

7 Example using hyperslabs /* Define hyperslab in the dataset. */ offset[0] = 1; count[0] = NX_SUB; offset[1] = 2; count[1] = NY_SUB; status = H5Sselect_hyperslab (dataspace, H5S_SELECT_SET, offset, NULL, count, NULL); /*Read data from hyperslab in file into hyperslab in memory */ status = H5Dread (dataset, H5T_NATIVE_INT, memspace, dataspace, H5P_DEFAULT, data_out); offset[0] count[0] offset[1] count[1] Examples taken from HDF-5 webpage Process 0 More complex example using hyperslabs Memory offset[0] offset[1] on rank 0 1 File count[0] = 1; count[1] = dimsmem[1]; block[0] = dimsfile[0]; block[1] = 1; offset[0] = 0; offset[1] = mpi_rank; stride[0] = 1; stride[1] = 2; Process 1 dimsmem[0] dimsmem[1] block[0] For dimension x: you generate count[x] entries of block[x] elements starting from offset[x]. The distance between each element is stride[x]. Examples taken from HDF-5 webpage 7

8 Parallel I/O with HDF-5 Relies on MPI I/O Program has to use special properties (hints) indicating to use parallel I/O during File creation and file open Data access Properties are set through H5P functions /* example for using file properties */ fileprops = H5Pcreate (H5P_FILE_ACCESS); H5Pset_fapl_mpio ( fileprops, MPI_COMM_WORLD, MPI_INFO_NULL); h5file = H5Fcreate ( tempseries.h5,, fileprops); Parallel data access in HDF-5 Application has to define a set of interleaved file data spaces on the processes that will access the file Similar technique like setting the file-view in MPI I/O Usually based on defining hyperslabs Data transfer properties have to be set /* example for using file properties */ fileprops = H5Pcreate (H5P_DATASET_XFER); H5Pset_dxpl_mpio ( fileprops, H5FD_MPIO_COLLECTIVE); H5Dwrite ( tset,, fileprops); 8