Overview of nagg 1.4.0

Transcription

1 The HDF Group Overview of nagg.4. Presentation and Demo for DEWG September 5, Larry Knox The HDF Group September 5, DEWG nagg tutorial

2 nagg.4. overview Purpose of presentation and demo:. Introduce, demonstrate and encourage trying out the nagg tool.. Check correctness of assumptions made in building the tool and of its output.. Invite feedback for improvement. September 5, DEWG nagg tutorial

3 nagg.4. overview What is nagg? Why would I use it? Where do I get it? What does nagg do? nagg command options nagg examples September 5, DEWG nagg tutorial

4 What is nagg? Nagg is a tool for rearranging NPP data granules from existing files to create new files with a different aggregation number or a different packaging arrangement. September 5, DEWG nagg tutorial 4

5 Why would I use nagg? Change aggregation number or packaging of previously downloaded npp data. Create aggregation or package combination not available for download. September 5, DEWG nagg tutorial 5

6 Where to get nagg The latest information and source for nagg can be found at A tar file containing these slides, example files and nagg.4. 64bit executable is available at ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/jpss/source/nagg/ nagg.4_demo.tar.gz. To build nagg: Download and extract the nagg-.4..tar.gz file for or 64 bit Linux. HDF5 and the hdf5_hl_region library are required. The hdf5_hl_region library source can also be downloaded from the links above. Build the source according to the doc/install file. September 5, DEWG nagg tutorial 6

7 Definitions Nagg - NPP granule aggregation and packaging utility Granule - A grouping of measurement or derived data (and/or data arrays) spanning a defined period (e.g., 8.6 seconds) and integer number of sensor scans. Definition varies for sensors and EDRs. The granule(s) can be accessed through the HDF5 reference regions provided in the NPOESS HDF5 Files. A granule within HDF5 is typically delineated with individually named and typed data arrays; each array is referenced with a separate object ID. RDRs, and Auxiliary/Ancillary data products delivered as HDF5, are in contrast binary structures stored purely as an array of bytes (unsigned char) referenced with a single object ID. Aggregation - A collection of granules, within an NPOESS HDF5 file. This will be a contiguous array for SDR/EDR/TDR/IP products. For RDR products, the aggregation s object ID dereferences (or points ) to an HDF5 group that contains one or more datasets. These datasets are the individual RDR granules. Granules are ordered temporally. The aggregation can be accessed with the HDF5 reference object. For a detailed explanation of aggregations, see Section.5., DDS Aggregation Methodology. Package Compatible NPP data products together or with corresponding geolocation product in common files. JPSS Common Data Format Control Book External Volume I, p 76 September 5, DEWG nagg tutorial 7

8 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial 8

9 Nagg options used in the examples Aggregation -n n where n is the number of granules in each output aggregation -A <seconds> n Default value of n is Packaging Default output is in packaged output files -S option produces simple unpackaged output files General -d <preexisting output directory> -h or help -t <list of product ids> (list is required) -g no - don t process geolocation product -g <geolocation product> September 5, DEWG nagg tutorial 9

10 Nagg --h or --help This option displays the list of available command options and a table of NPP product DPIDs, Short Names, Durations and GPIDs. The DPIDs and GPIDs are 5 letter ids used in the command option product lists and the output file names. There are currently 96 sensor data products and 9 geolocation products in the table, the first 7 shown here: DPID Short Name Duration GPID ICALI CrIMSS-CrIS-AVMP-LOS-IR-IP 997 GCRIO ICALM CrIMSS-CrIS-AVMP-LOS-MW-IP 997 GCRIO ICCCR CrIMSS-CrIS-CLOUD-CLEARED-RAD-IP 997 GCRIO ICISE CrIMSS-CrIS-IR-SURF-EMISSIVITY-IP 997 GCRIO ICMSE CrIMSS-CrIS-MW-SURF-EMISSIVITY-IP 997 GCRIO ICSTT CrIMSS-CrIS-SKIN-TEMP-IP 997 GCRIO ICTLI CrIMSS-CrIS-AVTP-LOS-IR-IP 997 GCRIO Apr. 7-9, HDF/HDF-EOS Workshop XV

11 nagg examples Aggregation (packaged) with geolocation product Aggregation of geolocation product only De-aggregation (packaged) with geolocation product Re-aggregation without geolocation product Packaging of sensor data products plus geolocation product De-aggregation and un-packaging of sensor data products plus geolocation product September 5, DEWG nagg tutorial

12 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial

13 Aggregation (Packaged) Increase number of granules per aggregation from to 4 Input files (8 + 8 geo) :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 SATMS GATMO Geolocation product is processed automatically and packaged with sensor data product by default. Command: nagg n 4 t SATMS d ex datafiles/satms*.h5 Input: 8 files with granule in each file Output: Produced 4 granules in ex/gatmo- SATMS_npp_d44_t_e99_b 5_c994578_XXXX_XXX.h5 Produced 4 granules in ex/gatmo- SATMS_npp_d44_t_e579_b 5_c99464_XXXX_XXX.h5 September 5, DEWG nagg tutorial

14 Aggregation (Packaged) Increase number of granules per aggregation from to 4 Input files (6) Output files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 SATMS GATMO September 5, DEWG nagg tutorial 4

15 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial 5

16 Aggregation (GEO only) Increase number of granules per aggregation from to 4 Input files (8) :: ::44 ::6 ::48 :: ::5 :4:4 :4:55 Command: nagg n 4 g GATMO d ex datafiles/gatmo*.h5 Input: 8 files with granule in each file Output: Produced 4 granules in ex/ GATMO_npp_d44_t_e99_b 5_c988788_XXXX_XXX.h5 Produced 4 granules in ex/ GATMO_npp_d44_t_e579_b 5_c _XXXX_XXX.h5 GATMO September 5, DEWG nagg tutorial 6

17 Aggregation (GEO Only) Increase number of granules per aggregation from to 4 Input files (8) :: ::44 ::6 ::48 :: ::5 :4:4 :4:55 Output files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:55 September 5, DEWG nagg tutorial 7

18 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial 8

19 De-aggregation (Packaged) Decrease number of granules per aggregation from 4 to Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 Command: nagg t SATMS d ex ex/gatmo- SATMS_npp_d44*.h5 Output (8 files): Produced granules in ex/gatmo- SATMS_npp_d44_t_e49_b 5_c _XXXX_XXX.h5 Produced granules in ex/gatmo- SATMS_npp_d44_t44_e59_b 5_c _XXXX_XXX.h5 Produced granules in ex/gatmo- SATMS_npp_d44_t456_e579_b 5_c _XXXX_XXX.h5 SATMS GATMO September 5, DEWG nagg tutorial 9

20 De-aggregation (Packaged) Decrease number of granules per aggregation from 4 to Input files () Output files (8) :: ::44 ::6 ::48 :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 :: ::5 :4:4 :4:56 SATMS GATMO September 5, DEWG nagg tutorial

21 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial

22 4 Re-aggregation without geolocation Decrease number of granules per aggregation from 4 to Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 SATMS GATMO -A 9 produces - second granules per aggregation for the SATMS and GATMO products Command: nagg g no A 9 t SATMS d ex4 ex/gatmo- SATMS_npp_d44*.h5 Output: Produced granules in ex4/ SATMS_npp_d44_t_e479_b 5_c _XXXX_XXX.h5 Produced granules in ex4/ SATMS_npp_d44_t48_e49_b 5_c _XXXX_XXX.h5 Produced granules in ex4/ SATMS_npp_d44_t44_e579_b 5_c _XXXX_XXX.h5 September 5, DEWG nagg tutorial

23 4 Re-aggregation without geolocation Decrease number of granules per aggregation from 4 to Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 Output files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 SATMS GATMO September 5, DEWG nagg tutorial

24 Partial aggregations and buckets Buckets (aggregation boundaries) are predetermined by the aggregation number and the granule duration starting from the IET* EPOCH, //958 Nagg does not produce leading or trailing fill granules for partial aggregations at the beginning or end of a set of granules. For a more extensive explanation see section.5., p 9 of the JPSS Common Data Format Control Book External Volume I * IDPS Epoch Time September 5, DEWG nagg tutorial 4

25 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial 5

26 5 Packaging Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 Package SATMS,TATMS,GATMO products SATMS TATMS GATMO Fill granules will be created for missing granules from missing files. Command: nagg t SATMS,TATMS d ex5 datafiles/ SATMS*.h5 datafiles/tatms*.h5 Output (8 files): Produced granules in ex5/gatmo-satms- TATMS_npp_d44_t_e7_ b5_c _xxxx_xx X.h5 Produced granules in ex5/gatmo-satms- TATMS_npp_d44_t44_e59 _b5_c _xxxx_xx X.h5 September 5, DEWG nagg tutorial 6 Produced granules in ex5/gatmo-satms- TATMS_npp_d44_t456_e579 _b5_c _xxxx_xx X.h5

27 5 Packaging Package SATMS,TATMS,GATMO products Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 Output files (8) :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 SATMS TATMS GATMO Fill granule September 5, DEWG nagg tutorial 7

28 Fill granules Are created when granules in a sequence are missing or when corresponding products do not have matching granules; however, files of entirely fill granules are not created. Have the same amount of data as regular granules, but all values are fill values as defined in control books or product profiles. Have the same structure and attributes as regular granules. Can be identified by the granule s N_Granule_Status attribute. Nagg creates fill granules with the value Missing at delivery time. Regular granules have the value N/A. September 5, DEWG nagg tutorial 8

29 Nagg operations Aggregation Packaging Aggregate data granules De-aggregate data granules Re-aggregate data granules Package granules of multiple compatible products in common files Un-package products into separate files for each product September 5, DEWG nagg tutorial 9

30 6 De-aggregation and Un-packaging De-aggregate and un-package SATMS,TATMS,GATMO products Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:55 Command: nagg -S t SATMS,TATMS d ex6 datafiles/ GATMO-SATMS-TATMS_npp_d44*.h5 Output (4 files): Produced granules in ex6/ SATMS_npp_d44_t_e7 _b5_c _xxxx_xx X.h5 Produced granules in ex6/ TATMS_npp_d44_t_e7_ b5_c _xxxx_xxx.h5 SATMS TATMS GATMO Fill granule September 5, DEWG nagg tutorial

31 6 De-aggregation and Un-packaging De-aggregate and un-package SATMS,TATMS,GATMO products Input files () :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 Output files (4) :: ::44 ::6 ::48 :: ::5 :4:4 :4:56 SATMS TATMS GATMO Fill granule September 5, DEWG nagg tutorial

32 Other nagg options -l like <example file> include products and use aggregation number as found in <example file> -g strict require exact match for geolocation files --version Print version information --debug Print all granules in input files, including those not selected by command options -O <ORIG> 4 character origin (output file name) -D <DOM> character domain ( output file name) RM.pdf has complete documentation of the nagg command options. September 5, DEWG nagg tutorial

33 Demo Aggregation (packaged) with geolocation product Aggregation of geolocation product only De-aggregation (packaged) with geolocation product Re-aggregation without geolocation product Packaging of sensor data products plus geolocation product De-aggregation and un-packaging of sensor data products plus geolocation product September 5, DEWG nagg tutorial

34 More examples: nagg/nagg-ug.pdf Help: September 5, DEWG nagg tutorial 4

35 Questions/comments? September 5, DEWG nagg tutorial 5

36 Thank you! September 5, DEWG nagg tutorial 6