HDF Update. Elena Pourmal The HDF Group. October 16, 2008 IDL User Group Meeting 1

Size: px

Start display at page:

Download "HDF Update. Elena Pourmal The HDF Group. October 16, 2008 IDL User Group Meeting 1"

Hilary Barnett
5 years ago
Views:

1 HDF Update Elena Pourmal The HDF Group October 16, 2008 IDL User Group Meeting 1

2 The HDF Group The HDF Group is a not-for-profit company with its mission focused on the support and growth of the HDF technologies Spun-off NCSA University of Illinois in July 2006 Located on University of Illinois Campus, South Research Park Strong connection with NCSA and researchers at the University of Illinois IP was transferred from University to The HDF Group October 16, 2008 IDL User Group Meeting 2

3 HDF is HDF stands for Hierarchical Data Format A file format for managing any kind of data Software system to manage data in the format Designed for high volume or complex data Designed for every size and type of system Free and open format and software library, tools There are two HDF s: HDF4 and HDF5 IDL provides access to HDF4 and HDF5 files October 16, 2008 IDL User Group Meeting 3

4 Earth Science HDF - IDL Users Petabytes of data in HDF4 and HDF-EOS2 formats EOS AURA satellite data products are in the HDF5 and HDF-EOS5 formats NPOESS data products will be in the HDF5 file format Neutron scattering and small angle X-ray scattering NeXus file format based on HDF4 and HDF5 Standard to facilitate greater cooperation in the analysis and visualization of neutron, x-ray, and muon data October 16, 2008 IDL User Group Meeting 4

5 HDF status HDF4 is in a maintenance mode HDF5 is actively developed to address new requirements: High performance computing (HPC) Atmospheric sciences applications and NetCDF-4 Bioinformatics (DNA sequencing) Imaging (new generations of electron microscopes) Microsoft products (HPC,.NET, others) Indexing and database integration Improving concurrency, performance and storage efficiency October 16, 2008 IDL User Group Meeting 5

6 ITTVis and The HDF Group Collaboration Goal: Help HDF users who rely on the IDL to get timely access to the improved HDF libraries ITTVis and The HDF Group have begun collaboration on the development of HDF functionality in IDL. ITTVis provided The HDF Group with IDL software and licenses. The HDF Group developers will work with ITTVis technical staff to improve the process of integrating new versions of HDF with IDL. October 16, 2008 IDL User Group Meeting 6

7 Current HDF releases October 16, 2008 IDL User Group Meeting 7

8 Current HDF5 Releases HDF was released in February 2008 Major update of HDF5 1.6.* series (stable set of features and APIs since 1998) Addressed performance issues and file size overhead Extensions to the HDF5 file format Changes to APIs (200+ new APIs) Backward compatible May not be forward compatible HDF was released in June 2008 Minor bug fixes Included Fortran90 APIs for new C functions October 16, 2008 IDL User Group Meeting 8

9 Current HDF5 Releases HDF was released in February 2008 Addressed backward compatibility bug for reading files with corrupted object header information New maintenance releases scheduled for November 2008 HDF and Minor bug fixes Tools improvements Current plans are to support HDF5 1.6 and 1.8 until November 2009 October 16, 2008 IDL User Group Meeting 9

10 Information About Current Releases October 16, 2008 IDL User Group Meeting 10

11 HDF5, OPenDAP and IDL Our First Experience with IDL October 16, 2008 IDL User Group Meeting 11

12 What is OPenDAP? A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community HDF5 OPenDAP handler released in July 2008 October 16, 2008 IDL User Group Meeting 12

13 OPeNDAP Concept View HDF/ NetCDF OPeNDAP Visualization client Visualization tools (e.g. Tools GrADS) OPeNDAP client library HDF/NetCDF OPeNDAP HDF/NetCDF server dap server (e.g. Apache) DAP 2.0/4.0 October 16, 2008 IDL User Group Meeting 13

14 OPeNDAP IDL client OPeNDAP and IDL Available through OPeNDAP web site. Used by HDF developers to debug and enhance the HDF5 OPenDAP handler Gained experience helped us to enhance other OPeNDAP clients such as GrADS (Grid Analysis and Display System) October 16, 2008 IDL User Group Meeting 14

$he5' jj stat = OPENDAP_GET(url, data, mode='data', ce='cloudfraction') image = data.cloudfraction.cloudfraction.cloudfraction image2 = BYTSCL(image, MIN=0.$ 0) OPENDAP_GET() is part of DEVICE, DECOMPOSED=0 A HDF5 dataset is OPeNDAP a member of IDL Client. LOADCT, 34 IDL structure variable called data.

0) OPENDAP_GET() is part of DEVICE, DECOMPOSED=0 A HDF5 dataset is OPeNDAP a member of IDL Client. LOADCT, 34 IDL structure variable called data.

15 OPeNDAP IDL Client Example url = ' jj stat = OPENDAP_GET(url, data, mode='data', ce='cloudfraction') image = data.cloudfraction.cloudfraction.cloudfraction image2 = BYTSCL(image, MIN=0.0) OPENDAP_GET() is part of DEVICE, DECOMPOSED=0 A HDF5 dataset is OPeNDAP a member of IDL Client. LOADCT, 34 IDL structure variable called data. window,/free,xsize=720,ysize=360 image3 = REBIN(image2, 720, 360) tv,image3 The retrieved data can be manipulated like a normal IDL array. October 16, 2008 IDL User Group Meeting 15

16 Demo: MLS swath via GrADS GrADS is modified to use our library The client library provides grid mapping from swath data GrADS displays swath through HDF-EOS5 specific client library API calls October 16, 2008 IDL User Group Meeting 16

17 Thank you! Questions? October 16, 2008 IDL User Group Meeting 17

P2.1 Accessing HDF5 data via OPeNDAP. MuQun Yang* 1, Hyo-Kyung Lee 1, James Gallagher 2. I. Introduction

P2.1 Accessing HDF5 data via OPeNDAP MuQun Yang* 1, Hyo-Kyung Lee 1, James Gallagher 2 1 The HDF Group, 2 OPeNDAP, Inc. I. Introduction HDF[1] is a set of data formats and software libraries for storing