ceda-di Documentation

Transcription

1 ceda-di Documentation Release Charles Newey December 09, 2014

2

3 Contents 1 Introduction 1 2 Project Goals 3 3 Command-line Usage Usage string Sample usage More information Configuration File Format More information Module Contents Command-Line Interface Module Contents Indices and tables Configuration File Indices and Tables 23 Python Module Index 25 i

4 ii

5 CHAPTER 1 Introduction The ceda_di module is a Python backend to interface with a wide range of historical scientific data formats. Within this project, you will find: Library interfaces to common scientific data formats (focusing mostly on data formats within the BADC archive), includin NetCDF HDF4 GeoTIFF ENVI BIL/BSQ Interfaces to the file format modules, to enable extraction of domain-specific metadata (including geospatial/temporal metadata) A command-line interface to a suite of tools designed to be run on a multi-core system with filesystem access to scientific d A tool to extract metadata from common scientific data formats A tool to index metadata into an Elasticsearch installation A suite of tools to query and visualise the data stored within Elasticsearch 1

6 2 Chapter 1. Introduction

7 CHAPTER 2 Project Goals The metadata extraction toolkit was developed with following goals: Extract geospatial, temporal and parameter metadata from a historical dataset with homogenous data stored in various file formats Generate JSON documents from the readable files containing geospatial and temporal metadata Ingest the created JSON documents into an enterprise full-text search system (ElasticSearch) Build a query system around the ElasticSearch module 3

8 4 Chapter 2. Project Goals

9 CHAPTER 3 Command-line Usage 3.1 Usage string The usage string for the command-line application is as follows: 1 """ 2 Usage: 3 di.py (--help --version) 4 di.py index [options] <path-to-json-docs> 5 di.py search <extents> [options] [--file-paths --json] 6 di.py extract [options] [--send-to-index] 7 [<input-path> (<output-path> --no-create-files)] 8 di.py test 9 10 Options: 11 --help Show this screen version Show version config=<path> Config file. [default:../config/ceda_di.json] 14 --host=<host> Specify ElasticSearch host port=<port> Specify ElasticSearch port index=<name> Specify ElasticSearch index name send-to-index Index metadata with ElasticSearch no-create-files Don t create JSON metadata files max-results=<num> Max number of results to return when searching 20 --file-paths Print out search results as file paths. 3.2 Sample usage Note: This is not a comprehensive guide! Please refer to the Command-Line Interface documentation for a more thorough explanation. # Extracts metadata from files in /path/to/input # Outputs data to Elasticsearch (--send-to-index) # Outputs JSON files to /path/to/output di.py extract --send-to-index /path/to/input /path/to/output # Extracts metadata from files in /path/to/input # Outputs data to Elasticsearch (--send-to-index) # Does not create JSON documents (--no-create-files) di.py extract --no-create-files --send-to-index /path/to/input 5

10 # Finds all JSON documents in /path/to/json and sends to Elasticsearch di.py index /path/to/json # Run unit tests di.py test 3.3 More information For more information, see the section on the Command-Line Interface. 6 Chapter 3. Command-line Usage

11 CHAPTER 4 Configuration File 4.1 Format The configuration for the command-line interface for di.py is written in a JSON file with various nested elements for various options. 4.2 More information For a full summary of all configuration options, read the section on the Configuration File. 7

12 8 Chapter 4. Configuration File

13 CHAPTER 5 Module Contents 5.1 Command-Line Interface Introduction ceda-di s command-line interface provides a simple way of accessing the basic functionality included in the ceda-di toolkit. The basic functions accessible through the CLI are: Indexing JSON files stored on disk into an Elasticsearch deployment Extracting metadata from scientific data files, and: Writing out the metadata as JSON files, and/or Indexing into an Elasticsearch deployment Searching an Elasticsearch deployment populated with metadata, including: Time/date ranges, Geographical extent (geospatial search), Full-text search over variables, etc Command-line Configuration Options --config [path-to-config] The file path to the configuration file for ceda-di. This defaults to the configuration file provided in the ceda-di repository (which should contain sensible defaults), but you may want to create your own customised configuration. --host [elasticsearch-host] The network path of your preferred Elasticsearch installation. This is where extracted JSON metadata will be stored and indexed if specified on the command-line. --port [elasticsearch-port] The port that your Elasticsearch service runs on. On most Elasticsearch installations this defaults to 9200 if you are unsure, try that first. --index [elasticsearch-index] This option specifies the Elasticsearch index to use when storing data. If it does not exist in the Elasticsearch installation at runtime, it will be created with the default settings specified in the Configuration File. 9

14 --max-results [num] This option defines the maximum number of results available per search request by default. When providing search parameters and an Elasticsearch index, this option will restrict the size of the reply Optional Command-line Parameters --send-to-index This option tells ceda-di to send any extracted metadata to the user s preferred Elasticsearch deployment. Any data discovered and extracted will be indexed and searchable shortly afterwards. --no-create-files This option tells ceda-di not to create any JSON files on the local machine. This is useful for environments with restricted disk space (for example, on a cloud platform or a shared space). --help Shows the help text. --version Shows the version Command-line Tasks (Commands) index [options] <path-to-json-docs> This command tells ceda-di to read and parses JSON documents in the path-to-json-docs directory, and send them to the specified Elasticsearch index. Essentially, this command is a convenient way of pushing JSON documents into an Elasticsearch instance. The Elasticsearch instance s configuration can either be specified in the Configuration File or via the Elastisearch command-line options specified above. search [options] <extents> This command tells ceda-di to search the defined Elasticsearch instance with the provided parameters and return --max-results-element long list of results. There are three simple modes of restricting your search, and the syntax is based on the JASMIN_CIS syntax. Latitude and longitude is restricted like so: # Syntax: di.py search x[lower_bound, upper_bound] y[lower_bound, upper_bound] # For example, this is a valid command: di.py search x[1.1111, ] y[3.3333, ] It is also possible to restrict your search temporally, like so: # Syntax di.py search t[lower_bound, upper_bound] # Please note that timestamps MUST be in ISO8601 format. # For example, this is a valid command: di.py search t[ , ] extract [options] <paths> This command is a little hard to describe with a paragraph of text, so read the provided examples below first. 10 Chapter 5. Module Contents

15 # Extract metadata from <input-path> # Don t create any files on disk # Send all metadata to the Elasticsearch index di.py extract --no-create-files --send-to-index <input-path> # Extract metadata from <input-path> # Create JSON files on disk in <output-path> # Send all metadata to the Elasticsearch index di.py extract --send-to-index <input-path> <output-path> # Extract metadata from <input-path> # Create JSON files on disk in <output-path> # Don t send to index (note the absence of the --send-to-index option di.py extract <input-path> <output-path> Looking at the examples above, it is obvious that there are three options: Send metadata to Elasticsearch ( send-to-index) Don t create files locally ( no-create-files) Or both ceda-di will not send metadata to Elasticsearch by default, but it will create JSON files locally by default. 5.2 Module Contents Module listing ceda_di package Subpackages ceda_di.metadata package Submodules ceda_di.metadata.product module Module for holding and exporting file metadata as JSON documents. class ceda_di.metadata.product.parameter(name, other_params=none) Bases: object Placeholder/wrapper class for metadata parameters Parameters name (str) Name of variable/parameter other_params (dict) Optional - Dict containing other param metadata get() Return the list of parameter items static make_param_item(name, value) Convert a name/value pair to dictionary (for better indexing in ES) 5.2. Module Contents 11

16 Parameters name (str) Name of the parameter item (e.g. long_name_fr, etc) value (str) Value of the parameter item (e.g. Radiance ) Returns Dict containing name:value information class ceda_di.metadata.product.properties(filesystem=none, spatial=none, temporal=none, data_format=none, parameters=none, **kwargs) Bases: object A class to hold, manipulate, and export geospatial metadata at file level. as_dict() Return metadata as dict object. Returns Dictionary describing metadata as_json() Return metadata as JSON string. Returns JSON document describing metadata. get_flight_info() Return a dictionary populated with metadata about the flight that the given data file was captured on - flight number, organisation, etc. Returns A dict containing flight metadata. static valid_lat(num) Return true if num is a valid latitude. Parameters num (float) Number to test Returns True if num is valid, else False static valid_lon(num) Return true if num is a valid longitude. Parameters num (float) Number to test Returns True if num is valid, else False Module contents Submodules ceda_di.envi_geo module Interface for reading data from ENVI BSQ/BIL packed binary files. Also contains methods for extracting metadata (geospatial/temporal). class ceda_di.envi_geo.bil(header_path, path=none, unpack_fmt= <d ) Bases: ceda_di.envi_geo.envi Sub-class of ENVI that uses the envi_io.bilfile class to read binary data from BIL files. read() Return a dict containing a summary of the file s data. Returns A dict containing a summary of the file s data. 12 Chapter 5. Module Contents

17 class ceda_di.envi_geo.bsq(header_path, path=none, unpack_fmt= <d ) Bases: ceda_di.envi_geo.envi Sub-class of ENVI that uses the envi_io.bsqfile class to read binary data from BSQ files. read() Return a dict containing a summary of the file s data. Returns A dict containing a summary of the file s data. class ceda_di.envi_geo.envi(header_path, path=none, unpack_fmt= <d ) Bases: ceda_di._dataset._geospatial get_data_format() Return file format information Returns A dict containing file format information get_geospatial() Read geospatial data parsed from binary file Returns A dict containing geospatial information get_parameters() Return a list of Parameter objects containing parameter information. Returns A list of Parameter objects containing parameter information. get_properties() Return a metadata.product.properties object describing the file s metadata. Returns Metadata.product.Properties object describing the file. get_temporal() Return a dictionary containing the start and end times of the data file. Returns A dict containing temporal data. ceda_di.envi_io module Interface for ENVI BIL and BSQ files (reading packed binary data) Used by envi_geo in ceda_di module Taken and adapted from: arsf-dan.nerc.ac.uk/trac/attachment/ticket/287/data_handler.py Original author: Ben Taylor (benj) class ceda_di.envi_io.bilfile(header_path, path=none, unpack_fmt= <d ) Bases: ceda_di.envi_io.envifile Child class of EnviFile. Provides correct wrappers of read() methods for reading BIL files in order. read() Read BIL file (reading bytes in correct order) Returns A multi-dimensional list containing unpacked BIL data. class ceda_di.envi_io.bsqfile(header_path, path=none, unpack_fmt= <d ) Bases: ceda_di.envi_io.envifile Child class of EnviFile. Provides correct wrappers of read() methods for reading BSQ files in order. read() Read BSQ file (reading bytes in correct order) Returns A multi-dimensional list containing unpacked BSQ data Module Contents 13

18 class ceda_di.envi_io.envifile(header_path, path=none, unpack_fmt= <d ) Bases: object Superclass for BilFile and BsqFile. Contains generic read() method that subclasses use to unpack binary data in the correct order. calc_from_xy() Calculate the number of pixels per line based on file size. check_valid_fmt_string() Check the format string for validity. Returns Number of bytes needed for type in the format string get_path(path, ext) Given the path of an ENVI header file, try to guess the path of the ENVI binary and return it. Parameters path Path to the BIL header file ext Extension of the binary data file to read Returns The path of the BIL data file process_hdr() Parse the provided header file. Returns Header file parsed into key/value pairs read(x_size, y_size, z_size) Read an ENVI binary file incrementally, returning arrays containing binary data. Parameters x_size Number of bands (BIL) Number of lines (BSQ) y_size Number of lines (BIL) Number of bands (BSQ) z_size Pixels per line ceda_di.exif_geo module Module containing classes to read and export XML metadata embedded in GeoTIFF files using the EXIF standard. class ceda_di.exif_geo.exif(fname) Bases: ceda_di._dataset._geospatial Class that handles extraction and export of EXIF metadata from GeoTIFF image files. get_geospatial() Return a dictionary containing geospatial extent metadata. Returns Dictionary containing geospatial extent metadata. get_properties() Return a ceda_di.metadata.product.properties object populated with the file s metadata. Returns A ceda_di.metadata.product.properties object get_temporal() Return a dictionary containing temporal extent metadata Returns Dictionary containing temporal extent metadata 14 Chapter 5. Module Contents

19 ceda_di.extract module Extract module - handles file crawling and metadata extraction. class ceda_di.extract.extract(conf ) Bases: object File crawler and metadata exractor class. Part of core functionality of ceda_di. conf(conf_opt) Return configuration option or raise exception if it doesn t exist. :param str conf_opt: The name of the configuration option to find. index_properties(filename, handler) Index the file in Elasicsearch make_dirs() Create directories for output files. prepare_logging() Initial logging setup process_file(filename) Instantiate a handler for a file and extract metadata. run() Run main metadata extraction suite. write_properties(fname, _geospatial_obj) Write module properties to an output file. class ceda_di.extract.handlerfactory(handler_map) Bases: object Factory for file handler classes. get_handler(filename) Return instance of correct file handler class. get_handler_class(filename) Return the class of the correct file handler (un-instantiated). ceda_di.hdf4_geo module Interface to extract and generate JSON from HDF4 EUFAR metadata class ceda_di.hdf4_geo.hdf4(fname) Bases: ceda_di._dataset._geospatial HDF4 context manager class. get_geospatial() Search through HDF4 file, returning a list of coordinates from the Navigation vgroup (if it exists). Returns Dict containing geospatial information. get_properties() Returns ceda_di.metadata.properties.properties object containing geospatial and temporal metadata from file. Returns Metadata.product.Properties object 5.2. Module Contents 15

20 get_temporal() Search through HDF4 file, returning timestamps from the Mission vgroup (if it exists) hdf = None v = None vs = None Returns List containing temporal metadata ceda_di.index module class ceda_di.index.bulkindexer(config, threshold=1000) Bases: object Context manager for indexing into an ES installation by pooling documents and submitting in large bulk requests when the document count reaches a certain threshold. add_to_index_pool(document, mapping=none) Add document to the correct pool, dependent on mapping type. :param str mapping: The mapping to index the document into. :param object document: The JSON-serialisable object to index. index_directory(path, mapping=none) Indexes all files in a given directory. :param str path: The path to the directory containing the data files. :param str mapping: The mapping type (doc type) for the document to be indexed as. submit_pool(mapping=none) Submit current document grouping (grouped by mapping) to the appropriate mapping in the ElasticSearch index. :param str mapping: The mapping to submit a to index. submit_pools() Submit all current document pools to the ElasticSearch index. ceda_di.index.create_index(config, elasticsearch) Set up an index in ElasticSearch, given a configuration file path. :param dict config: Application configuration dictionary, including ES config. :param str index_settings_path: Path to index settings JSON document. ceda_di.jascis module ceda_di.netcdf_geo module Metadata adapters for NetCDF files. class ceda_di.netcdf_geo.netcdffactory(fpath) Bases: object Factory for checking, handling and returning an appropriate metadata extraction class. Parameters fpath (str) Path to NetCDF file get_properties() Return correct metadata extraction class based on metadata format. class ceda_di.netcdf_geo.netcdf_base Bases: ceda_di._dataset._geospatial Base class - provides common NetCDF metadata extraction methods Parameters fpath (str) Path to NetCDF file 16 Chapter 5. Module Contents

21 static clean_coordinate(coord) Return True if coordinate is valid. static find_var_by_regex(ncdf, regex) Find a variable reference searching by regular expression. Parameters ncdf (Dataset) Reference to an opened netcdf4.dataset object regex (re) Regular expression to match with variable name static find_var_by_standard_name(ncdf, standard_name) Find a variable reference searching by CF standard name. Parameters ncdf (Dataset) Reference to an opened netcdf4.dataset object standard_name (str) The CF standard name to search for static geospatial(ncdf, lat_name, lon_name) Return a dict containing lat/lons from NetCDF file. Parameters ncdf (Dataset) Reference to an opened netcdf4.dataset object lat_name Name of parameter containing latitude values lon_name Name of parameter containing longitude values Returns Geospatial information as dict. static params(ncdf ) Construct list of Parameters based on variables in NetCDF file. Parameters ncdf (Dataset) Reference to an opened netcdf4.dataset object Returns list List of metadata.product.parameter objects static temporal(ncdf, time_name) Extract time values from Dataset using the variable name provided. Parameters ncdf (Dataset) Reference to an opened netcdf4.dataset object time_name (str) Name of the time parameter class ceda_di.netcdf_geo.netcdf_cf(fpath, convention) Bases: ceda_di._dataset._geospatial Metadata extraction class for CF-compliant NetCDF files. get_geospatial() get_parameters() get_properties() Return a metadata.product.properties object populated with metadata. Returns Properties object populated with metadata get_temporal() 5.2. Module Contents 17

22 class ceda_di.netcdf_geo.netcdf_raf(fpath, convention) Bases: ceda_di._dataset._geospatial Metadata extraction class for NCAR-RAF-compliant NetCDF. get_geospatial() get_parameters() get_properties() Return a metadata.product.properties object populated with metadata. Returns Properties object populated with metadata get_temporal() class ceda_di.netcdf_geo.netcdf_unknown(fpath) Bases: ceda_di._dataset._geospatial A wrapper to extract NetCDF metadata that is stored using an unknown metadata convention, or no known associated metadata convention. get_properties() ceda_di.search module class ceda_di.search.elasticsearchclientfactory Bases: object get_client(config_args) Return an appropriately configured Elasticsearch client. Parameters config_args Configuration dictionary. Should contain an Elasticsearch hostname under key es-host and an Elasticsearch port under the key es-port. Returns A configured Elasticsearch instance class ceda_di.search.jsonquerybuilder Bases: object build(extents_string=none, max_results=none) Build an Elasticsearch query dictionary from a given extents string. Parameters extents_string A string specifying temporal or spatial extents, e.g. t=[ T12:13:14, T17:18:19]. Returns A dictionary which is valid Elasticsearch query JSON. process_datetime_extents(start, end) Process a datetime extents search filter and add it to the query dictionary. Will parse partial datetimes to maximise the search window - e.g. start=2009, end=2010 will find all results from T00:00:00 to T23:59:59 Parameters start Start datetime string end End datetime string process_latitude_extents(lat_1, lat_2) Process latitude extents search filter and add it to the query dictionary. Will always include the region from the lowest latitude specified to the highest, regardless of the order in which they are passed to this function. 18 Chapter 5. Module Contents

23 Parameters lat_1 Latitude float in the range -90 to +90 degrees. lat_2 Latitude float in the range -90 to +90 degrees. process_longitude_extents(start, end) Process longitude extents search filter and add it to the query dictionary. Will automatically constrain start and end longitudes to be within the range -180 to +180 (so they may be specified e.g. as 370). The region searched is always the region from the start longitude to the end latitude :param start: Start latitude :param end: End latitude process_single_datetime(datetime) Process a single datetime search filter and add it to the query dictionary. Will parse partial datetimes to maximise the search window - e.g will find all results from T00:00:00 to T23:59:59 :param datetime: Start datetime string process_single_latitude(lat) Process a single latitude search filter and add it to the query dictionary. Parameters lat Latitude to filter by process_single_longitude(lon) Process a single longitude search filter. Will automatically constrain to within the range -180 to +180 (so values of e.g. 370 are acceptable). Parameters lon Longitude to filter by class ceda_di.search.searcher(config_args, json_query_builder=<ceda_di.search.jsonquerybuilder object at 0x7f8eb8d0af90>, elastic_search_client_factory=<ceda_di.search.elasticsearchclientfactory object at 0x7f8eb8d14050>) Bases: object Coordinates the searching of Elasticsearch nodes to output matching filepaths. run() Run the search and output the results matching the configuration belonging to this instance. Returns Outputs matching filenames to sys.stdout Module contents 5.3 Indices and tables genindex modindex search 5.4 Configuration File A sample JSON config file is provided in../../config/ceda_di.json. This contains all of the mandatory options, as well as a brief description of the role of each option. All options are described below in more detail if necessary Indices and tables 19

24 Note: The default configuration will work for most use-cases - all that most users will need to change are the Elasticsearch options and the file paths Options es-host [url] The master Elasticsearch node of the Elasticsearch cluster used. In the example configuration file, this is set to fatcat-test.jc.rl.ac.uk. es-port [port] The port that the Elasticsearch installation is running on. In the example configuration file, this is set to es-index [index name] The name of the Elasticsearch index to store extracted metadata in. In the example configuration file, this is set to badc. es-mapping [mapping name] Note: This is an advanced option for Elasticsearch. The defaults provided in ceda-di should be appropriate for most uses. The name of the Elasticsearch mapping. An Elasticsearch mapping exists so that Elasticsearch can infer data types from data stored in the index. This enables Elasticsearch to search across date/timestamps, do autocompletion on text fields, provide full-text search across all text data, and so on. In the example configuration file, the default mapping name is eufar. The default mapping eufar is provided in../../../elasticsearch/mapping/index_settings.json es-index-settings [path to index settings] Note: This is an advanced option for Elasticsearch. The defaults provided in ceda-di should be appropriate for most uses. The path to the settings JSON file for the Elasticsearch index. This includes any custom analyzers, tokenizers and so on. The default index settings are provided in../../../elasticsearch/mapping/index_settings.json For further reference, please read the Elasticsearch docs. max-results [number of results] The maximum number of results to return per search request. output-path [output directory] The directory to place all output files (JSON, log files, etc). log-path [log directory] The directory (as a subdirectory of output-path) to place all log files in. json-path [JSON directory] The directory (as a subdirectory of output-path to place all JSON output in. input-path [input directory] The directory to scan for files and extract metadata from. num-cores [number of cores] The number of processes to run in parallel to extract metadata. Usually best suited to 1-2 processors below the number of cores in your machine. For example, if di extract is run on a machine with 24 cores, it s best to set num-cores to between 20 and 22. logging [object containing logging info] Options for the Python logging module. The default options will log to a file with the format specified in the configuration file. This is mostly for development and should not be edited. 20 Chapter 5. Module Contents

25 handlers [object containing file handlers] An object containing filename pattern regular expressions as keys, and a handler class and priority as values. For example, if I have a class to handle JPEG files in the ceda_di.jpeg namespace, and I want to process metadata from all files ending with.jpg, I would write a configuration option like this: "handlers": { "\.jpg$": { "class": "ceda_di.jpeg", "priority": 1 } } And this would allow ceda_di to recognise all JPG files and extract metadata from them Configuration File 21

26 22 Chapter 5. Module Contents

27 CHAPTER 6 Indices and Tables genindex modindex search 23

28 24 Chapter 6. Indices and Tables

29 Python Module Index c ceda_di, 19 ceda_di.envi_geo, 12 ceda_di.envi_io, 13 ceda_di.exif_geo, 14 ceda_di.extract, 15 ceda_di.hdf4_geo, 15 ceda_di.index, 16 ceda_di.metadata, 12 ceda_di.metadata.product, 11 ceda_di.netcdf_geo, 16 ceda_di.search, 18 25

30 26 Python Module Index

31 Index Symbols config [path-to-config] command line option, 9 help command line option, 10 host [elasticsearch-host] command line option, 9 index [elasticsearch-index] command line option, 9 max-results [num] command line option, 9 no-create-files command line option, 10 port [elasticsearch-port] command line option, 9 send-to-index command line option, 10 version command line option, 10 A add_to_index_pool() (ceda_di.index.bulkindexer method), 16 as_dict() (ceda_di.metadata.product.properties method), 12 as_json() (ceda_di.metadata.product.properties method), 12 B BIL (class in ceda_di.envi_geo), 12 BilFile (class in ceda_di.envi_io), 13 BSQ (class in ceda_di.envi_geo), 12 BsqFile (class in ceda_di.envi_io), 13 build() (ceda_di.search.jsonquerybuilder method), 18 BulkIndexer (class in ceda_di.index), 16 C calc_from_xy() (ceda_di.envi_io.envifile method), 14 ceda_di (module), 19 ceda_di.envi_geo (module), 12 ceda_di.envi_io (module), 13 ceda_di.exif_geo (module), 14 ceda_di.extract (module), 15 ceda_di.hdf4_geo (module), 15 ceda_di.index (module), 16 ceda_di.metadata (module), 12 ceda_di.metadata.product (module), 11 ceda_di.netcdf_geo (module), 16 ceda_di.search (module), 18 check_valid_fmt_string() (ceda_di.envi_io.envifile method), 14 clean_coordinate() (ceda_di.netcdf_geo.netcdf_base static method), 16 command line option config [path-to-config], 9 help, 10 host [elasticsearch-host], 9 index [elasticsearch-index], 9 max-results [num], 9 no-create-files, 10 port [elasticsearch-port], 9 send-to-index, 10 version, 10 es-host [url], 20 es-index [index name], 20 es-index-settings [path to index settings], 20 es-mapping [mapping name], 20 es-port [port], 20 extract [options] <paths>, 10 handlers [object containing file handlers], 21 index [options] <path-to-json-docs>, 10 input-path [input directory], 20 json-path [JSON directory], 20 log-path [log directory], 20 logging [object containing logging info], 20 max-results [number of results], 20 num-cores [number of cores], 20 output-path [output directory], 20 search [options] <extents>, 10 conf() (ceda_di.extract.extract method), 15 create_index() (in module ceda_di.index), 16 27

32 E ElasticsearchClientFactory (class in ceda_di.search), 18 ENVI (class in ceda_di.envi_geo), 13 EnviFile (class in ceda_di.envi_io), 13 es-host [url] command line option, 20 es-index [index name] command line option, 20 es-index-settings [path to index settings] command line option, 20 es-mapping [mapping name] command line option, 20 es-port [port] command line option, 20 EXIF (class in ceda_di.exif_geo), 14 Extract (class in ceda_di.extract), 15 extract [options] <paths> command line option, 10 F find_var_by_regex() (ceda_di.netcdf_geo.netcdf_base static method), 17 find_var_by_standard_name() (ceda_di.netcdf_geo.netcdf_base static method), 17 G geospatial() (ceda_di.netcdf_geo.netcdf_base static method), 17 get() (ceda_di.metadata.product.parameter method), 11 get_client() (ceda_di.search.elasticsearchclientfactory method), 18 get_data_format() (ceda_di.envi_geo.envi method), 13 get_flight_info() (ceda_di.metadata.product.properties method), 12 get_geospatial() (ceda_di.envi_geo.envi method), 13 get_geospatial() (ceda_di.exif_geo.exif method), 14 get_geospatial() (ceda_di.hdf4_geo.hdf4 method), 15 get_geospatial() (ceda_di.netcdf_geo.netcdf_cf method), 17 get_geospatial() (ceda_di.netcdf_geo.netcdf_raf method), 18 get_handler() (ceda_di.extract.handlerfactory method), 15 get_handler_class() (ceda_di.extract.handlerfactory method), 15 get_parameters() (ceda_di.envi_geo.envi method), 13 get_parameters() (ceda_di.netcdf_geo.netcdf_cf method), 17 get_parameters() (ceda_di.netcdf_geo.netcdf_raf method), 18 get_path() (ceda_di.envi_io.envifile method), 14 get_properties() (ceda_di.envi_geo.envi method), 13 get_properties() (ceda_di.exif_geo.exif method), 14 get_properties() (ceda_di.hdf4_geo.hdf4 method), 15 get_properties() (ceda_di.netcdf_geo.netcdf_cf method), 17 get_properties() (ceda_di.netcdf_geo.netcdf_raf method), 18 get_properties() (ceda_di.netcdf_geo.netcdf_unknown method), 18 get_properties() (ceda_di.netcdf_geo.netcdffactory method), 16 get_temporal() (ceda_di.envi_geo.envi method), 13 get_temporal() (ceda_di.exif_geo.exif method), 14 get_temporal() (ceda_di.hdf4_geo.hdf4 method), 15 get_temporal() (ceda_di.netcdf_geo.netcdf_cf method), 17 get_temporal() (ceda_di.netcdf_geo.netcdf_raf method), 18 H HandlerFactory (class in ceda_di.extract), 15 handlers [object containing file handlers] command line option, 21 hdf (ceda_di.hdf4_geo.hdf4 attribute), 16 HDF4 (class in ceda_di.hdf4_geo), 15 I index [options] <path-to-json-docs> command line option, 10 index_directory() (ceda_di.index.bulkindexer method), 16 index_properties() (ceda_di.extract.extract method), 15 input-path [input directory] command line option, 20 J json-path [JSON directory] command line option, 20 JsonQueryBuilder (class in ceda_di.search), 18 L log-path [log directory] command line option, 20 logging [object containing logging info] command line option, 20 M make_dirs() (ceda_di.extract.extract method), 15 make_param_item() (ceda_di.metadata.product.parameter static method), 11 max-results [number of results] command line option, 20 N NetCDF_Base (class in ceda_di.netcdf_geo), Index

33 NetCDF_CF (class in ceda_di.netcdf_geo), 17 NetCDF_RAF (class in ceda_di.netcdf_geo), 17 NetCDF_Unknown (class in ceda_di.netcdf_geo), 18 NetCDFFactory (class in ceda_di.netcdf_geo), 16 num-cores [number of cores] command line option, 20 O output-path [output directory] command line option, 20 P Parameter (class in ceda_di.metadata.product), 11 params() (ceda_di.netcdf_geo.netcdf_base static method), 17 prepare_logging() (ceda_di.extract.extract method), 15 process_datetime_extents() (ceda_di.search.jsonquerybuilder method), 18 process_file() (ceda_di.extract.extract method), 15 process_hdr() (ceda_di.envi_io.envifile method), 14 process_latitude_extents() (ceda_di.search.jsonquerybuilder method), 18 process_longitude_extents() (ceda_di.search.jsonquerybuilder method), 19 process_single_datetime() (ceda_di.search.jsonquerybuilder method), 19 process_single_latitude() (ceda_di.search.jsonquerybuilder method), 19 process_single_longitude() (ceda_di.search.jsonquerybuilder method), 19 Properties (class in ceda_di.metadata.product), 12 R read() (ceda_di.envi_geo.bil method), 12 read() (ceda_di.envi_geo.bsq method), 13 read() (ceda_di.envi_io.bilfile method), 13 read() (ceda_di.envi_io.bsqfile method), 13 read() (ceda_di.envi_io.envifile method), 14 run() (ceda_di.extract.extract method), 15 run() (ceda_di.search.searcher method), 19 S search [options] <extents> command line option, 10 Searcher (class in ceda_di.search), 19 submit_pool() (ceda_di.index.bulkindexer method), 16 submit_pools() (ceda_di.index.bulkindexer method), 16 T temporal() (ceda_di.netcdf_geo.netcdf_base static method), 17 V v (ceda_di.hdf4_geo.hdf4 attribute), 16 valid_lat() (ceda_di.metadata.product.properties static method), 12 valid_lon() (ceda_di.metadata.product.properties static method), 12 vs (ceda_di.hdf4_geo.hdf4 attribute), 16 W write_properties() (ceda_di.extract.extract method), 15 Index 29