Using Chunked Extendible Array for Physical Storage of Scientific Datasets

Size: px

Start display at page:

Download "Using Chunked Extendible Array for Physical Storage of Scientific Datasets"

Angelina Ray
5 years ago
Views:

1 Using Chunked Extendible Array for Physical Storage of Scientific Datasets E.J Otoo, G. Nimako and D. Ohene-Kwofie School of Computer Science The University of the Witwatersrand Johannesburg, South Africa Sponsored By: CHPC 12, National Meeting

2 Introduction The conceptual model of a dataset as a multidimensional array has been around since the early days of relational database model. Records of multi-attribute files, perceived as points in multi-dimensional space, form the bases of many of the efficient data structures and algorithms. Examples: KD-Tree, Quad-Tree, Grid-File, multi-dimensional OLAP and data warehousing. Scientific data-sets are maintained predominantly as multi-dimensional arrays - HDF5, NetCDF, New SciDB, etc. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 2 / 26

3 Introduction The conceptual model of a dataset as a multidimensional array has been around since the early days of relational database model. Records of multi-attribute files, perceived as points in multi-dimensional space, form the bases of many of the efficient data structures and algorithms. Examples: KD-Tree, Quad-Tree, Grid-File, multi-dimensional OLAP and data warehousing. Scientific data-sets are maintained predominantly as multi-dimensional arrays - HDF5, NetCDF, New SciDB, etc. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 2 / 26

4 Motivation Processing massively large scientific data (Big Data), in the realm of Peta-scale and Exa-scale computing is with the Partitioned Global Address Space (PGAS) computational model. The PGAS computational model are exemplified by: Global Array Toolkit Unified Parallel C (UPC) OpenShmem The database consistent with this model of computation is SciDB It is an Array Data Model on shared nothing parallel architectures with unbounded non-uniform dimensions with holes, arbitrary nesting, sophisticated storage representations, including overlapping chunks, etc. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 3 / 26

5 Motivation: The Global Array Toolkit Distributed dense arrays that can be accessed through a shared memory-like style Single, shared data structure/global indexing Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 4 / 26

6 Motivation: The Global Array Toolkit Distributed dense arrays that can be accessed through a shared memory-like style Single, shared data structure/global indexing Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 4 / 26

7 Motivation: The SciDB Basic Architecture Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 5 / 26

8 Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 6 / 26

9 Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. The question then arises: How can we store arrays elements so that the index range of any dimension can be extended? Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 6 / 26

10 Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. The question then arises: How can we store arrays elements so that the index range of any dimension can be extended? The problem has been addressed in some earlier papers. Notably by : A. Rosenberg and L. Stockmeyer [1977]; Tsuji, M. Kuroda and K. Higuchi [2008]; E. Otoo and D. Rotem [2006]; K. M. Azharul Hasan, Tatsuo Tsuji [2011]; Extendible Arrays for Multidimensional Databases: Concept, Implementation and Evaluation Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 6 / 26

11 The Problem Statement Problem Definition Let A[U 0 ][U 1 ]... [U k 1 ], denote a k-dimensional array where U j, 0 j < k 1, is the bound on the indices of the dimension j that is allowed to extend arbitrary. Let L = {l 0, l 1..., l n 1,..., }, be a sequence of consecutive storage locations where n = k 1 j=0 U j. Define a computable access function F () that maps the k-dimensional indices one-to-one, onto the sequence of consecutive indices {0, 1,..., n 1,..., }, without reallocating previously allocated elements when a bound of the array is extended. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 7 / 26

12 Our Main Results and Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 8 / 26

13 Our Main Results and Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. Given j we can compute an inverse function F 1 (j) to derive the k-dimensional index i 0, i 1,..., i k 1 in time O(k + log N). Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 8 / 26

14 Our Main Results and Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. Given j we can compute an inverse function F 1 (j) to derive the k-dimensional index i 0, i 1,..., i k 1 in time O(k + log N). We introduce a data structure, the O 2 -Tree, a modified Red-Black-Tree, to store some vectors of values associated with each dimension. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 8 / 26

15 Our Main Results and Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. Given j we can compute an inverse function F 1 (j) to derive the k-dimensional index i 0, i 1,..., i k 1 in time O(k + log N). We introduce a data structure, the O 2 -Tree, a modified Red-Black-Tree, to store some vectors of values associated with each dimension. We chunk the array, to align with the paging scheme of a file-system, so that I/O units are in chunks. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 8 / 26

16 Road Map for the Rest of the Talk Conventional arrays of simple elements Extendible arrays of data Chunked extendible arrays Some experimental results Usage in a distributed parallel environment Summary Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 9 / 26

17 Conventional Array Using Row-Major Ordering Consider an 3-D array A[U 0 ][U 1 ][U 2 ] whose elements are allocated in row-major order. The location of an element A i 0, i 1, i 2, is given by loc(a 0, 0, 0 + q where q is computed as: q = i 0 C 0 + i 1 C 1 + i 2 C 2 and C 0 = U 1 U 2 C 1 = U 2 C 2 = 1 The values C 0, C 1, C 2 are held in the dope-vector. This reveals the strategy of our allocation scheme. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 10 / 26

18 Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 11 / 26

19 Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 11 / 26

20 Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 11 / 26

21 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 12 / 26

22 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 12 / 26

23 Linear Mapping for a Dense Extendible Array Suppose that in a k-dimensional extendible array A[U 0][U 1][U 2]...[U k 1 ], dimension l is extended by λ l, then the index range increases from U l to U l + λ l. Let the location A 0, 0,..., U l,..., 0 (i.e. the starting location of an allocated hyperslab ) be denoted as l Z 0 where Z 0 U U = k 1 l r=0 U r. l The Mapping Function k 1 q = F ( i 0, i 1, i 2,..., i k 1 )) = Z 0 U + (i l U l )C l + i j Cj l j=0 j l k 1 Cl = U j j=0 j l C j = k 1 r=j+1 r l U r Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 13 / 26

24 Linear Mapping for a Dense Extendible Array Suppose that in a k-dimensional extendible array A[U 0][U 1][U 2]...[U k 1 ], dimension l is extended by λ l, then the index range increases from U l to U l + λ l. Let the location A 0, 0,..., U l,..., 0 (i.e. the starting location of an allocated hyperslab ) be denoted as l Z 0 where Z 0 U U = k 1 l r=0 U r. l The Mapping Function k 1 q = F ( i 0, i 1, i 2,..., i k 1 )) = Z 0 U + (i l U l )C l + i j Cj l j=0 j l k 1 Cl = U j j=0 j l C j = k 1 r=j+1 r l U r Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 13 / 26

25 Chunking Extendible Dense Arrays The use of the list for axial-vectors can be expensive and depends in particular on the interruptible expansions. (worst case is when we have cubical extensions). Such interruptible expansions cause the addition of a new entry in the list of axial vectors. Chunking the array gives some additional advantages: It gives contiguous storage allocations for the elements of the chunks. When arrays are allocated onto secondary storage, I/O can be made in multiples of the chunk size. The allocation is done in chunks as opposed to the single elements. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 14 / 26

26 Chunking Extendible Dense Arrays The use of the list for axial-vectors can be expensive and depends in particular on the interruptible expansions. (worst case is when we have cubical extensions). Such interruptible expansions cause the addition of a new entry in the list of axial vectors. Chunking the array gives some additional advantages: It gives contiguous storage allocations for the elements of the chunks. When arrays are allocated onto secondary storage, I/O can be made in multiples of the chunk size. The allocation is done in chunks as opposed to the single elements. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 14 / 26

27 Chunking Extendible Dense Arrays, Cont. Given a chunked block Q[χ 0 ][χ 1 ][χ 2 ]...[χ k 1 ], the number of chunk indices, ρ i for a given dimension i, is given by: U ρ i = i χ i The allocation of chunks, denoted by A c, becomes A c [ρ 0 ][ρ 1 ][ρ 2 ]...[ρ k 1 ]. An entry is made to the requisite axial-vector only if: [U l + λ l] > [ρ l χ l ] The number of chunk indices ρ l to be allocated is given by: [U ρ l = l + λ l ] [ρ l χ l ] χ l Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 15 / 26

28 Chunking Extendible Dense Arrays, Cont. Given a chunked block Q[χ 0 ][χ 1 ][χ 2 ]...[χ k 1 ], the number of chunk indices, ρ i for a given dimension i, is given by: U ρ i = i χ i The allocation of chunks, denoted by A c, becomes A c [ρ 0 ][ρ 1 ][ρ 2 ]...[ρ k 1 ]. An entry is made to the requisite axial-vector only if: [U l + λ l] > [ρ l χ l ] The number of chunk indices ρ l to be allocated is given by: [U ρ l = l + λ l ] [ρ l χ l ] χ l Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 15 / 26

29 Illustration of Chunked Extendible Dense Arrays Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 16 / 26

30 Chunked Extendible Dense Array Mapping To alocate an array element A i 0, i 1, i 2,..., i k 1, the values of the k-dimensional index i 0, i 1, i 2,..., i k 1 are translated into values of a chunk index j 0, j 1, j 2,..., j k 1 where ii j i = The starting address, q c of the chunk containing A i 0, i 1, i 2,..., i k 1 can be found by: The Mapping Function for Chunked Extendible Array k 1 qc = F ( j 0, j 1, j 2,..., j k 1 )) = Z 0 ρ l + (j l ρ l )Cl + C l = C m = k 1 m=0 m l k 1 χ i ρ m r=m+1 r l ρ r m=0 m l j mc m Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 17 / 26

31 Chunked Extendible Dense Array Mapping To alocate an array element A i 0, i 1, i 2,..., i k 1, the values of the k-dimensional index i 0, i 1, i 2,..., i k 1 are translated into values of a chunk index j 0, j 1, j 2,..., j k 1 where ii j i = The starting address, q c of the chunk containing A i 0, i 1, i 2,..., i k 1 can be found by: The Mapping Function for Chunked Extendible Array k 1 qc = F ( j 0, j 1, j 2,..., j k 1 )) = Z 0 ρ l + (j l ρ l )Cl + C l = C m = k 1 m=0 m l k 1 χ i ρ m r=m+1 r l ρ r m=0 m l j mc m Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 17 / 26

32 Chunked Extendible Dense Array Mapping, Cont. To compute the address of A i 0, i 1, i 2,..., i k 1 within the local chunk, the input indices i 0, i 1, i 2,..., i k 1 needs to be translated to local chunk indices i c0, i c1, i c2,..., i c(k 1) by: i cm = (i m mod χ m ) The address of A i 0, i 1, i 2,..., i k 1 is only a displacement within the chunk. This can be done by using a row-major order or column-major order. If the chunk size is 2 n where n 2, then the Z-order sequence or Peano-Hilbert space filling curve can be used. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 18 / 26

33 Chunked Extendible Dense Array Mapping, Cont. To compute the address of A i 0, i 1, i 2,..., i k 1 within the local chunk, the input indices i 0, i 1, i 2,..., i k 1 needs to be translated to local chunk indices i c0, i c1, i c2,..., i c(k 1) by: i cm = (i m mod χ m ) The address of A i 0, i 1, i 2,..., i k 1 is only a displacement within the chunk. This can be done by using a row-major order or column-major order. If the chunk size is 2 n where n 2, then the Z-order sequence or Peano-Hilbert space filling curve can be used. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 18 / 26

34 Axial-Vectors as Memory Resident O 2 -Tree A new approach to maintaining the these axial-vectors in memory is with the use of O 2 -Tree. An O 2 -Tree is an augmented Red-Black Tree with data records stored only at the leaf nodes. A metadata file F m stores the records that correspond to the leaf nodes of the O 2 -Tree. These records in F m is used to reconstruct the memory resident O 2 -Tree. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 19 / 26

35 Axial-Vectors as Memory Resident O 2 -Tree A new approach to maintaining the these axial-vectors in memory is with the use of O 2 -Tree. An O 2 -Tree is an augmented Red-Black Tree with data records stored only at the leaf nodes. A metadata file F m stores the records that correspond to the leaf nodes of the O 2 -Tree. These records in F m is used to reconstruct the memory resident O 2 -Tree. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 19 / 26

36 Axial-Vectors as Memory Resident O 2 -Tree, Cont. General Structure of the O 2-Tree : Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 20 / 26

37 Experimental Results - I Average Access Cost without Extensions (in Memory) Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 21 / 26

38 Experimental Results - II Total Access Cost for Interleaved Extensions in Memory Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 22 / 26

39 Experimental Results - III Total Access Cost for Interleaved Extensions on Disk Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 23 / 26

40 Experimental Results - IV Storage Utilisation for Chunked Extendible Array Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 24 / 26

41 Summary and Future Work We have shown how chunked extendible dense arrays can be implemented as an array file. Array elements can be accessed into and out of memory in multiples of chunks with the aid of a computed access mapping function. The organisation of extendible arrays using such a mapping function is highly appropriate for most scientific datasets where the model of the data is perceived as a large multi-dimensional array. The appropriate APIs for integrating our scheme with the Global Array Toolkit and as a storage engine for SciDB - Elastic Partitioned Global Address Store (EPGAS) Main application areas is for hyper-spectral image processing and a host other scientific applications that are based on multidimensional data models. Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 25 / 26

42 Thanks! Thank You Questions? Otoo,Nimako&Ohene-Kwofie () Using Chunked Ext... CHPC 12, National Meeting 26 / 26

Chunked Extendible Arrays and its Integration with the Global Array Toolkit for Parallel Image Processing

Chunked Extendible Arrays and its Integration with the Global Array Toolkit for Parallel Image Processing Gideon Nimako A thesis submitted to the Faculty of Engineering and the Built Environment in fulfilment