Scientific File Formats

Transcription

1 SLAC 6 October 2010

2 1 Introduction 2 3 4

3 Introduction: why files? Files contain many (most?) scientific data Files last for a long time Explosion in data more, bigger files Figure: Magnetic tape drive com/photos/laughingsquid/ /

4 Scientific File Access Simple, non-transactional Data access: value lookups, statistics, plotting, Transformations: simple math + complex algorithms Logging/history: coarse Image C is the result of function(image A, parameter set B) not $X was deducted from Account Y and added to Account Z Longevity: >10, 50 or 100+ years

5 Themes and in Formats Sequential or random-access navigation Storage efficiency Self-description (i.e., metadata) Ordering: Object sequences, grids, images, N-D arrays (+ tables) Write-append (non-update) Machine portability (FP format, byte order) Standards

6 Flexible Image Transport System. Standard astronomical data format (NASA/IAU) Generic N-D arrays, images, and ASCII or binary tables Image tile-compression Not random access Human-readable header See also: AipsIO (used by casacore)

7 Primary HDU Extension HDU Extension HDU... HDU type Contents Primary ASCII header + n-d array Image* ASCII header w/image metadata + n-d array ASCII Table* ASCII header w/table metadata + fixed-width row data Binary Table* ASCII header w/table metadata + 2-D array * Extension HDU

8 extended Tagged Container (HEP, photon science) Object serialization: Vectors, trajectories, events, detections Not random-access No compression Lightweight, streaming

9 Datagram Datagram... Datagram type Sequence Xtc Env Notes Event transitions (e.g., {Begin,End}Run, L1Accept) User data objects More information:

10 ROOT Object I/O (HEP) Object serialization Tree-structured (similar to fs) Object deletion Compression (deflate) Ranged values See also: LCIO

11 Figure: From [Brun and Rademakers, 1996]

12 Network Common Data Form. (Geo) N-dimensional arrays Arrays appendable in one dimension (v4: or more) Named dimensions with explicit coordinates (allow irregular spacing) Machine portable Slabbed, sliced, random access 4: API on physical structure JJA wind power, courtesy Scott Capps

13 Section Notes Header magic, #records, dimension, global attr, variable meta Non-record data fixed-size variables, incl fixeddimensions Record data record variables (incl. record dimensions From [Rew et al., 1997]

14 Hierarchical Data Format N-D arrays Array nesting via pointers+vl datatypes Compression Parallel I/O (MPI I/O) Flexible, chunked data layout (slabs or custom tiles) Ragged arrays via variable-length datatypes fs-like, w/symlinks, heaps, freelists, Up to 255 byte offsets

15 Figure: From [HDF Group, 2010]

16 Many more formats! Irregular/non-rectangular grids (e.g., geodesic, radial, etc.) [Wadsley and Shell Internationale Petroleum, 1980], triangular meshes [Górski et al., 2005] See also Scientific Data Format FAQ [Stern, 1995] Figure: Hexagonal mesh http: // From HEALPix:

17 Final notes Data > formats Long-lived formats, long-lived software Figure: Looking for data in files. http: //

18 References Brun, R. and Rademakers, F. (1996). ROOT Object I/O System. Górski, K., Hivon, E., Banday, A., Wandelt, B., Hansen, F., Reinecke, M., and Bartelmann, M. (2005). HEALPix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622:759. HDF Group (2010). file format specification version Rew, R., Davis, G., Emmerson, S., and Davies, H. (1997). user s guide for C. Unidata Program Center. Stern, I. (1995). Scientific Data Format Information FAQ. Wadsley, W. A. and Shell Internationale Petroleum (1980). Modelling reservoir geometry with non-rectangular coordinate grids. In SPE Annual Technical Conference and Exhibition, Dallas, Texas. American Institute of Mining, Metallurgical, and Petroleum Engineers.