Introduction to HDF5

Transcription

1 Introduction to parallel HDF Maison de la Simulation Saclay, 0-0 March 201, Parallel filesystems and parallel IO libraries

2 Evaluation form Please do not forget to fill the evaluation form at

3 Outline HDF in the context of Input/Output (IO) HDF Application Programming Interface (API) Playing with Dataspace Hands on session

4 Hardware/Software stack Application Data structures High level I/O library Standard library I/O library System Hardware Operating system File system Hard drive

5 High level I/O libraries The purpose of high level I/O libraries is to provide the developer a higher level of abstraction to manipulate computational modeling objects Meshes of various complexity (rectilinear, curvilinear, unstructured... ) Discretized functions on such meshes Materials... Until now, these libraries are mainly used in the context of visualization

6 Existing libraries Silo Wide range of objects Built on top of HDF Native format for VisIt Exodus Focused on unstructured meshes and finite element representations Built on top of NetCDF Famous/intensively used codes output format extensible Data Model and Format (XDMF)

7 I/O libraries Purpose of I/O libraries: Efficient I/O Portable binary files Higher level of abstraction for the developer Two main existing libraries: Hierarchical Data Format: HDF Network Common Data Form: NetCDF

8 HDF library HDF file: HDF group: a grouping structure containing instances of zero or more groups or datasets HDF dataset: a multidimensional array of data elements HDF dataset multidimensional array: Name Datatype (Atomic, NATIVE, Compound) Dataspace (rank, sizes, max sizes) Storage layout (contiguous, compact, chunked)

9 HDF High Level APIs Dimension Scale (HDS): Enables to attach dataset dimension to scales Lite (HLT): Enables to write simple dataset in one call Image (HIM): Enables to write images in one call Table (HTB): Hides the compound types needed for writing tables Packet Table (HPT): Almost HTB but without record insertion/deletion but supports variable length records...

10 HDF low level API HF: File manipulation routines HG: Group manipulation routines HS: Dataspace manipulation routines HD: Dataset manipulation routines... Just have a look at the outstanding on-line reference manual for HDF!

11 C order versus Fortran order /* C language */ #define NX #define NY 3 int x,y; int f[ny][nx];! Fortran language integer, parameter :: NX= integer, parameter :: NY=3 integer :: x,y integer, dimension(nx,ny) :: f for (y=0;y<ny;y++) for (x=0;x<nx;x++) f[y][x] = x+y; do y=1,ny do x=1,nx f(x,y) = (x-1) + (y-1) enddo enddo The memory mapping is identical, the language semantic is different!!

12 HDF first example #define NX #define NY # define RANK 2 i n t main ( void ) { h i d t f i l e, dataset, dataspace ; h s i z e t dimsf [ 2 ] ; h e r r t s t a t u s ; i n t data [NY ] [ NX ] ; i n i t ( data ) ; f i l e = HFcreate ( example. h, HF ACC TRUNC, HP DEFAULT, \ HP DEFAULT ) ; dimsf [ 0 ] = NY; dimsf [ 1 ] = NX;

13 HDF first example cont. dataspace = HScreate simple (RANK, dimsf, NULL ) ; dataset = HDcreate ( f i l e, I n t A r r a y, HT NATIVE INT, \ dataspace, HP DEFAULT, HP DEFAULT, HP DEFAULT ) ; s t a t u s = HDwrite ( dataset, HT NATIVE INT, HS ALL, \ HS ALL, HP DEFAULT, data ) ; HSclose ( dataspace ) ; HDclose ( dataset ) ; HFclose ( f i l e ) ; } return 0;

14 HDF high level example cont. s t a t u s = HLTmake dataset int ( f i l e, I n t A r r a y, RANK, dimsf, data ) ; HFclose ( f i l e ) ; } return 0;

15 Variable C type h i d t f i l e, dataset, dataspace ; h s i z e t dimsf [ 2 ] ; h e r r t s t a t u s ; hid t: handler for any HDF objects (file, groups, dataset, dataspace, datatypes... ) hsize t: C type used for number of elements of a dataset (in each dimension) herr t: C type used for getting error status of HDF functions

16 File creation f i l e = HFcreate ( example. h, HF ACC TRUNC, HP DEFAULT, \ HP DEFAULT ) ; example.h : file name HF ACC TRUNC: File creation and suppress it if it exists already HP DEFAULT: file creation property list HP DEFAULT: file access property list (needed for MPI-IO)

17 Dataspace creation dimsf [ 0 ] = NY; dimsf [ 1 ] = NX; dataspace = HScreate simple (RANK, dimsf, NULL ) ; RANK: dataset dimensionality dimsf: size of the dataspace in each dimension NULL: specify max size of the dataset being fixed to the size

18 Dataset creation dataset = HDcreate ( f i l e, I n t A r r a y, HT NATIVE INT, \ dataspace, HP DEFAULT, HP DEFAULT, HP DEFAULT ) ; file: HDF objects where to create the dataset. Should be a file or a group. IntArray : dataset name HT NATIVE INT: type of the data the dataset will contain dataspace: size of the dataset HP DEFAULT: default option for property list.

19 Datatype Predefined Datatypes: created by HDF. Derived Datatypes: created or derived from the predefined data types. There are two types of predefined datatypes: STANDARD: They defined standard ways of representing data. Ex: HT IEEE F32BE means IEEE representation of 32 bit floating point number in big endian. NATIVE: Alias to standard data types according to the platform where the program is compiled. Ex: on an Intel based PC, HT NATIVE INT is aliased to the standard predefined type, HT STD 32LE.

20 Datatype cont. A data type can be: ATOMIC: cannot be decomposed into smaller data type units at the API level. Ex: integer COMPOSITE: An aggregation of one or more data types. Ex: compound data type, array, enumeration

21 Dataset writing s t a t u s = HDwrite ( dataset, HT NATIVE INT, HS ALL, \ HS ALL, HP DEFAULT, data ) ; dataset: HDF objects representing the dataset to write HT NATIVE INT: Type of the data in memory HS ALL: dataspace specifying the portion of memory that needs be read (in order to be written) HS ALL: dataspace specifying the portion of the file dataset that needs to be written HP DEFAULT: default option for property list (needed for MPI-IO). data: buffer containing the data to write

22 Closing HDF objects HSclose ( dataspace ) ; HDclose ( dataset ) ; HFclose ( f i l e ) ; Opened/created HDF objects are closed.

23 Some comments s t a t u s = HLTmake dataset int ( f i l e, I n t A r r a y, RANK, dimsf, data ) ; HFclose ( f i l e ) ; } return 0; This example is almost a fwrite, but: The generated file is portable The generated file can be accessed with HDF tools Attributes can be added on datasets or groups The type of the data can be fixed The storage layout can be modified Portion of the dataset can be written...

24 Concept of start, stride, count block Considering a n-dimensional array, start, stride, count and block are arrays of size n that describe a subset of the original array start: Starting location for the hyperslab (default 0) stride: The number of elements to separate each element or block to be selected (default 1) count: The number of elements or blocks to select along each dimension block: The size of the block (default 1)

25 Conventions for the examples We consider: A 2D array f [N y ][N x ] with N x =, N y = Dimension x is the dimension contiguous in memory Graphically, the x dimension is represented horizontal Language C convention is used for indexing the dimensions Dimension y is index=0 Dimension x is index=1

26 Graphical representation Dimension x Dimension y Memory order i n t s t a r t [ 2 ], s t r i d e [ 2 ], count [ 2 ], block [ 2 ] ; s t a r t [ 0 ] = 0; s t a r t [ 1 ] = 0; s t r i d e [ 0 ] = 1; s t r i d e [ 1 ] = 1; block [ 0 ] = 1; block [ 1 ] = 1;

27 Illustration for count parameter Dimension x Dimension y y=0 y=1 y= count [ 0 ] = 3; count [ 1 ] = ;

28 Illustration for start parameter Dimension x Dimension y y=0 y=1 y=2 3 s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = 3; count [ 1 ] = ;

29 Illustration for stride parameter Dimension x Dimension y y=0 y=1 y=2 3 s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = 3; count [ 1 ] = ; s t r i d e [ 0 ] = 3; s t r i d e [ 1 ] = 1;

30 Illustration for stride parameter Dimension x Dimension y y=0 y=1 y=2 3 s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = 3; count [ 1 ] = 2; s t r i d e [ 0 ] = 3; s t r i d e [ 1 ] = 3;

31 Illustration for block parameter Dimension x Dimension y y=0 y=1 3 y=2 y=3 y= y= s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = 3; count [ 1 ] = 2; s t r i d e [ 0 ] = 3; s t r i d e [ 1 ] = 3; block [ 0 ] = 2; block [ 1 ] = 2;

32 Exercise 1 Please draw the elements selected by the start, stride, count, block set below Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = ;

33 Solution 1 Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = ;

34 Exercise 2 Please draw the elements selected by the start, stride, count, block set below Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = 1; count [ 1 ] = 1; block [ 0 ] = ; block [ 1 ] = ;

35 Solution 2 Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = 1; count [ 1 ] = 1; block [ 0 ] = ; block [ 1 ] = ;

36 Exercise 3 Please draw the elements selected by the start, stride, count, block set below Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = 3; count [ 1 ] = 2; s t r i d e [ 0 ] = 2; s t r i d e [ 1 ] = 2; block [ 0 ] = 2; block [ 1 ] = 2;

37 Solution 3 Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = 3; count [ 1 ] = 2; s t r i d e [ 0 ] = 2; s t r i d e [ 1 ] = 2; block [ 0 ] = 2; block [ 1 ] = 2;

38 What is a dataspace? Dataspace Objects Null dataspaces Scalar dataspaces Simple dataspaces rank or number of dimensions current size maximum size (can be unlimited) Dataspaces come into play: for performing partial IO to describe the shape of HDF dataset

39 What is a dataspace for? Figure : Access a sub-set of data with a hyperslab 1 Figure : Build complex regions with hyperslab unions 1 1 Figures taken from HDF website

40 What is a dataspace for? Figure : Use hyper-slabs to gather or scatter data 2 2 Figures taken from HDF website

41 How to play with dataspaces h i d t space id ; h s i z e t dims [ 2 ], s t a r t [ 2 ], count [ 2 ] ; h s i z e t s t r i d e =NULL, block=null ; dims [ 0 ] = ny ; dims [ 1 ] = nx ; s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = ; space id = HScreate simple ( 2, dims, NULL ) ; s t a t u s = HSselect hyperslab ( space id, HS SELECT SET, s t a r t, \ s t r i d e, count, block ) ;

42 How to play with dataspaces space id is modified by HSselect hyperslab, so it must exist start, stride, count, block arrays must be at least the same size as the rank of space id dataspace HS SELECT SET replaces the existing selection with the parameters from this call. Other operations : HS SELECT OR, AND, XOR, NOTB and NOTA stride, block arrays are considered as 1 if NULL is passed

43 Using dataspaces during a partial IO s t a t u s = HSselect hyperslab ( space id mem, HS SELECT SET, \ start mem, stride mem, count mem, block mem ) ; s t a t u s = HSselect hyperslab ( space id disk, HS SELECT SET, \ s t a r t d i s k, s t r i d e d i s k, count disk, b l o c k d i s k ) ; s t a t u s = HDwrite ( dataset, HT NATIVE INT, space id mem, \ space id disk, HP DEFAULT, data ) ; The two dataspace can describe non contiguous data and can be of different dimension But the number of elements must match

44 Parallel HDF implementation MPI rank 2 MPI rank 0 MPI rank 1 MPI rank 3

45 Parallel HDF implementation INTEGER(HSIZE_T) :: array_size(2), array_subsize(2), array_start(2) INTEGER(HID_T) :: plist_id1, plist_id2, file_id, filespace, dset_id, memspace array_size(1) = S array_size(2) = S array_subsize(1) = local_nx array_subsize(2) = local_ny array_start(1) = proc_x * array_subsize(1) array_start(2) = proc_y * array_subsize(2)!allocate and fill the tab array CALL hopen_f(ierr) CALL hpcreate_f(hp_file_access_f, plist_id1, ierr) CALL hpset_fapl_mpio_f(plist_id1, MPI_COMM_WORLD, MPI_INFO_NULL, ierr) CALL hfcreate_f('res.h', HF_ACC_TRUNC_F, file_id, ierr, access_prp = plist_id1)! Set collective call CALL hpcreate_f(hp_dataset_xfer_f, plist_id2, ierr) CALL hpset_dxpl_mpio_f(plist_id2, HFD_MPIO_COLLECTIVE_F, ierr) CALL hscreate_simple_f(2, array_size, filespace, ierr) CALL hscreate_simple_f(2, array_subsize, memspace, ierr) CALL hdcreate_f(file_id, 'pi_array', HT_NATIVE_REAL, filespace, dset_id, ierr) CALL hsselect_hyperslab_f (filespace, HS_SELECT_SET_F, array_start, array_subsize, ierr) CALL hdwrite_f(dset_id, HT_NATIVE_REAL, tab, array_subsize, ierr, memspace, filespace, plist_id2)! Close HDF objects

46 Hands on parallel HDF 1 tar xvfz /gpfslocal/pub/patc-io/hdf.tgz 2 cp -r HDF/phdf-1 phdf 3 cd phdf Examine the source code, compile and run it Examine the output file out*.h with hls and hdump command line tools Modify the program to write in a single file all the different datasets Modify the program to write in a single file all the data in a single dataset Modify the program to write the single dataset with MPI-IO

47 Hands on HDF 1 cp -r HDF/hdf-1 hdf 2 cd hdf 3 Examine the source code, compile and run it Examine the output file example.h with hls and hdump command line tools Modify the program to write an additional dataset containing a 3D data of size Nx.Ny.Nz Modify the program to write a single 2D slice of the 3D array instead of the whole array Modify the program to write the previous 2D slice as an extension of the original 2D dataset Implement parallel HDF output for the Jacobi solver instead of the MPI-IO ones

48 Parallel HDF for the Jacobi solver on tasks 1 Implement parallel HDF output for the Jacobi solver instead of the MPI-IO ones MPI rank 2 MPI rank 0 MPI rank 1 MPI rank 3 Single file