Introduction to serial HDF5

Transcription

1 Introduction to serial HDF Matthieu Haefele Saclay, - March 201, Parallel filesystems and parallel IO libraries PATC@MdS Matthieu Haefele

2 Training outline Day 1: AM: Serial HDF (M. Haefele) PM: Parallel IO and parallel HDF (M. Haefele) Day 2: AM 1: Luster file TGCC (T. Leibovici) AM 2 + PM: XIOS (M-H. Nguyen) Please do not forget to fill the evaluation form at

3 Outline Day 1 Morning: HDF in the context of Input/Output (IO) HDF Application Programming Interface (API) Playing with Dataspace Hands on session Afternoon: Parallel IO issues & concepts Basic concepts of MPI-IO Parallel HDF Hands on session

4 IO in a nustshell Doing Input / Output is about TRANSPORTING Data stored in memory to / from Data stored on disk

5 IO in a nustshell Three criteria / metrics to balance Code development / maintenance time Performance Post-processing requirement

6 Hardware/Software stack Application Data structures High level I/O library I/O library Standard library System Hardware Operating system File system Hard drive

7 High level I/O libraries The purpose of high level I/O libraries is to provide the developer a higher level of abstraction to manipulate computational modeling objects Meshes of various complexity (rectilinear, curvilinear, unstructured... ) Discretized functions on such meshes Materials... Until now, these libraries are mainly used in the context of visualization

8 Existing libraries Silo Wide range of objects Built on top of HDF Native format for VisIt Exodus Focused on unstructured meshes and finite element representations Built on top of NetCDF Famous/intensively used codes output format extensible Data Model and Format (XDMF) XIOS (XML IO Server)

9 I/O libraries Purpose of I/O libraries: Efficient I/O Portable binary files Higher level of abstraction for the developer Two main existing libraries: Hierarchical Data Format: HDF Network Common Data Form: NetCDF

10 HDF library HDF file: HDF group: a grouping structure containing instances of zero or more groups or datasets HDF dataset: a multidimensional array of data elements HDF dataset multidimensional array: Name Datatype (Atomic, NATIVE, Compound) Dataspace (rank, sizes, max sizes) Storage layout (contiguous, compact, chunked)

11 HDF High Level APIs Dimension Scale (HDS): Enables to attach dataset dimension to scales Lite (HLT): Enables to write simple dataset in one call Image (HIM): Enables to write images in one call Table (HTB): Hides the compound types needed for writing tables Packet Table (HPT): Almost HTB but without record insertion/deletion but supports variable length records...

12 HDF low level API HF: File manipulation routines HG: Group manipulation routines HS: Dataspace manipulation routines HD: Dataset manipulation routines... Just have a look at the outstanding on-line reference manual for HDF!

13 C order versus Fortran order /* C language */ #define NX #define NY int x,y; int f[ny][nx]; for (y=0;y<ny;y++) for (x=0;x<nx;x++) f[y][x] = x+y;! Fortran language integer, parameter :: NX= integer, parameter :: NY= integer :: x,y integer, dimension(nx,ny) :: f do y=1,ny do x=1,nx f(x,y) = (x-1) + (y-1) enddo enddo The memory mapping is identical, the language semantic is different!!

14 HDF first example #define NX #define NY # define RANK 2 i n t main ( void ) { h i d t f i l e, dataset, dataspace ; h s i z e t dimsf [ 2 ] ; h e r r t s t a t u s ; i n t data [NY ] [ NX ] ; i n i t ( data ) ; f i l e = HFcreate ( example. h, HF ACC TRUNC, HP DEFAULT, \ HP DEFAULT ) ; dimsf [ 0 ] = NY; dimsf [ 1 ] = NX;

15 HDF first example cont. dataspace = HScreate simple (RANK, dimsf, NULL ) ; dataset = HDcreate ( f i l e, I n t A r r a y, HT NATIVE INT, \ dataspace, HP DEFAULT, HP DEFAULT, HP DEFAULT ) ; s t a t u s = HDwrite ( dataset, HT NATIVE INT, HS ALL, \ HS ALL, HP DEFAULT, data ) ; HSclose ( dataspace ) ; HDclose ( dataset ) ; HFclose ( f i l e ) ; } return 0;

16 HDF high level example cont. s t a t u s = HLTmake dataset int ( f i l e, I n t A r r a y, RANK, dimsf, data ) ; HFclose ( f i l e ) ; return 0; }

17 Variable C type h i d t f i l e, dataset, dataspace ; h s i z e t dimsf [ 2 ] ; h e r r t s t a t u s ; hid t: handler for any HDF objects (file, groups, dataset, dataspace, datatypes... ) hsize t: C type used for number of elements of a dataset (in each dimension) herr t: C type used for getting error status of HDF functions

18 File creation f i l e = HFcreate ( example. h, HF ACC TRUNC, HP DEFAULT, \ HP DEFAULT ) ; example.h : file name HF ACC TRUNC: File creation and suppress it if it exists already HP DEFAULT: file creation property list HP DEFAULT: file access property list (needed for MPI-IO)

19 Dataspace creation dimsf [ 0 ] = NY; dimsf [ 1 ] = NX; dataspace = HScreate simple (RANK, dimsf, NULL ) ; RANK: dataset dimensionality dimsf: size of the dataspace in each dimension NULL: specify max size of the dataset being fixed to the size

20 Dataset creation dataset = HDcreate ( f i l e, I n t A r r a y, HT NATIVE INT, \ dataspace, HP DEFAULT, HP DEFAULT, HP DEFAULT ) ; file: HDF objects where to create the dataset. Should be a file or a group. IntArray : dataset name HT NATIVE INT: type of the data the dataset will contain dataspace: size of the dataset HP DEFAULT: default option for property list.

21 Datatype Predefined Datatypes: created by HDF. Derived Datatypes: created or derived from the predefined data types. There are two types of predefined datatypes: STANDARD: They defined standard ways of representing data. Ex: HT IEEE F2BE means IEEE representation of 2 bit floating point number in big endian. NATIVE: Alias to standard data types according to the platform where the program is compiled. Ex: on an Intel based PC, HT NATIVE INT is aliased to the standard predefined type, HT STD 2LE.

22 Datatype cont. A data type can be: ATOMIC: cannot be decomposed into smaller data type units at the API level. Ex: integer COMPOSITE: An aggregation of one or more data types. Ex: compound data type, array, enumeration

23 Dataset writing s t a t u s = HDwrite ( dataset, HT NATIVE INT, HS ALL, \ HS ALL, HP DEFAULT, data ) ; dataset: HDF objects representing the dataset to write HT NATIVE INT: Type of the data in memory HS ALL: dataspace specifying the portion of memory that needs be read (in order to be written) HS ALL: dataspace specifying the portion of the file dataset that needs to be written HP DEFAULT: default option for property list (needed for MPI-IO). data: buffer containing the data to write

24 Closing HDF objects HSclose ( dataspace ) ; HDclose ( dataset ) ; HFclose ( f i l e ) ; Opened/created HDF objects are closed.

25 Some comments s t a t u s = HLTmake dataset int ( f i l e, I n t A r r a y, RANK, dimsf, data ) ; HFclose ( f i l e ) ; } return 0; This example is almost a fwrite, but: The generated file is portable The generated file can be accessed with HDF tools Attributes can be added on datasets or groups The type of the data can be fixed The storage layout can be modified Portion of the dataset can be written... Matthieu Haefele

26 Concept of start, stride, count block Considering a n-dimensional array, start, stride, count and block are arrays of size n that describe a subset of the original array start: Starting location for the hyperslab (default 0) stride: The number of elements to separate each element or block to be selected (default 1) count: The number of elements or blocks to select along each dimension block: The size of the block (default 1)

27 Conventions for the examples We consider: A 2D array f [N y ][N x ] with N x =, N y = Dimension x is the dimension contiguous in memory Graphically, the x dimension is represented horizontal Language C convention is used for indexing the dimensions Dimension y is index=0 Dimension x is index=1

28 Graphical representation Dimension x Dimension y Memory order i n t s t a r t [ 2 ], s t r i d e [ 2 ], count [ 2 ], block [ 2 ] ; s t a r t [ 0 ] = 0; s t a r t [ 1 ] = 0; s t r i d e [ 0 ] = 1; s t r i d e [ 1 ] = 1; block [ 0 ] = 1; block [ 1 ] = 1; Matthieu Haefele

29 Illustration for count parameter Dimension y Dimension x y=0 y=1 y= count [ 0 ] = ; count [ 1 ] = ;

30 Illustration for start parameter Dimension y Dimension x y=0 y=1 y=2 s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = ; count [ 1 ] = ;

31 Illustration for stride parameter Dimension y Dimension x y=0 y=1 y=2 s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = ; count [ 1 ] = ; s t r i d e [ 0 ] = ; s t r i d e [ 1 ] = 1;

32 Illustration for stride parameter Dimension y Dimension x y=0 y=1 y=2 s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = ; count [ 1 ] = 2; s t r i d e [ 0 ] = ; s t r i d e [ 1 ] = ;

33 Illustration for block parameter Dimension x Dimension y y=0 y=1 y=2 y= y= y= s t a r t [ 0 ] = 1; s t a r t [ 1 ] = 2; count [ 0 ] = ; count [ 1 ] = 2; s t r i d e [ 0 ] = ; s t r i d e [ 1 ] = ; block [ 0 ] = 2; block [ 1 ] = 2; Matthieu Haefele

34 Exercise 1 Please draw the elements selected by the start, stride, count, block set below Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = ; Matthieu Haefele

35 Solution 1 Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = ;

36 Exercise 2 Please draw the elements selected by the start, stride, count, block set below Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = 1; count [ 1 ] = 1; block [ 0 ] = ; block [ 1 ] = ; Matthieu Haefele

37 Solution 2 Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = 1; count [ 1 ] = 1; block [ 0 ] = ; block [ 1 ] = ;

38 Exercise Please draw the elements selected by the start, stride, count, block set below Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = 2; s t r i d e [ 0 ] = 2; s t r i d e [ 1 ] = 2; Matthieu Haefele

39 Solution Dimension x Dimension y s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = 2; s t r i d e [ 0 ] = 2; s t r i d e [ 1 ] = 2; block [ 0 ] = 2; block [ 1 ] = 2; Matthieu Haefele

40 What is a dataspace? Dataspace Objects Null dataspaces Scalar dataspaces Simple dataspaces rank or number of dimensions current size maximum size (can be unlimited) Dataspaces come into play: for performing partial IO to describe the shape of HDF dataset

41 What is a dataspace for? Figure : Access a sub-set of data with a hyperslab 1 Figure : Build complex regions with hyperslab unions 1 1 Figures taken from HDF website Matthieu Haefele

42 What is a dataspace for? Figure : Use hyper-slabs to gather or scatter data 2 2 Figures taken from HDF website Matthieu Haefele

43 How to play with dataspaces h i d t space id ; h s i z e t dims [ 2 ], s t a r t [ 2 ], count [ 2 ] ; h s i z e t s t r i d e =NULL, block=null ; dims [ 0 ] = ny ; dims [ 1 ] = nx ; s t a r t [ 0 ] = 2; s t a r t [ 1 ] = 1; count [ 0 ] = ; count [ 1 ] = ; space id = HScreate simple ( 2, dims, NULL ) ; s t a t u s = HSselect hyperslab ( space id, HS SELECT SET, s t a r t, \ s t r i d e, count, block ) ;

44 How to play with dataspaces space id is modified by HSselect hyperslab, so it must exist start, stride, count, block arrays must be at least the same size as the rank of space id dataspace HS SELECT SET replaces the existing selection with the parameters from this call. Other operations : HS SELECT OR, AND, XOR, NOTB and NOTA stride, block arrays are considered as 1 if NULL is passed

45 Using dataspaces during a partial IO s t a t u s = HSselect hyperslab ( space id mem, HS SELECT SET, \ start mem, stride mem, count mem, block mem ) ; s t a t u s = HSselect hyperslab ( space id disk, HS SELECT SET, \ s t a r t d i s k, s t r i d e d i s k, count disk, b l o c k d i s k ) ; s t a t u s = HDwrite ( dataset, HT NATIVE INT, space id mem, \ space id disk, HP DEFAULT, data ) ; The two dataspace can describe non contiguous data and can be of different dimension But the number of elements must match

46 HDF command line tools HDF files are non ASCII files non human readable files Tools provided to manipulate and get information contained in HDF files Three main ones: hls, hdump, hdiff

47 Hands on HDF 1. git clone hands-on.git 2. cd HDF hands-on/hdf-1. Examine the source code, compile and run it. Examine the output file example.h with hls and hdump command line tools. Modify the program to write an additional dataset containing a D data of size Nx.Ny.Nz. Modify the program to write a single 2D slice of the D array instead of the whole array. Modify the program to write the previous 2D slice as an extension of the original 2D dataset