HiPS Hierarchical Progressive Survey

Transcription

1 International Virtual Observatory Alliance HiPS Hierarchical Progressive Survey Version 1.0 IVOA Note 15 th October 2015 Previous version(s): None Authors: Pierre Fernique [CDS] Mark Allen [CDS] Thomas Boch [CDS] Daniel Durand [CADC] Laurent Michel [SSC-XMM] Jesus Salgado [ESAC] Felix Stoehr [ALMA] Editor: Pierre Fernique Abstract This note presents HiPS, a hierarchical scheme for the description, storage and access of sky survey data. The principle is based on hierarchical tiling of sky regions at finer and finer spatial resolution which facilitates a progressive view of a survey, and supports multi-resolution zooming and panning. HiPS uses the HEALPix tessellation of the sky as the basis for the scheme and is implemented

2 as a simple file structure with a direct indexing scheme that leads to practical implementations. Status of This Document This is an IVOA Note. This is the first release of this document. This is an IVOA Note expressing suggestions from and opinions of the authors. It is intended to share best practices, possible approaches, or other perspectives on interoperability with the Virtual Observatory. The document describes a standard created by IVOA partners and is not a formal IVOA standard. A list of current IVOA Recommendations and other technical documents can be found at Contents Table of Contents 1 Description, context and background 3 2 HiPS principle 3 3 HiPS directory structure HiPS tile format Image tile format Catalogue tile format Cube tile format Low resolution enablers Order omission Allsky preview file Meta data Properties file MOC file Metadata file index.html file HiPS access and distribution 12 References 12 Appendix A HiPS mirroring management 13 HiPS node, HiPS list 13 HiPS node registry 14 Global HIPS list 14 HiPS node mirroring policy 14 Appendix B Suggested client algorithms for displaying HiPS 16 Image HiPS 16 Catalog HiPS 16

3 1 Description, context and background A Hierarchical Progressive Survey, also called HiPS, allows a dedicated client/browser tool to access and display a survey progressively, based on the principle that the more you zoom in on a particular area the more details show up. The HiPS method is based on HEALPix sky tessellation. It maps a set of astronomical images into a collection of HEALPix tiles that are organized in a hierarchical way as directories and files. A HiPS may be published via a HTTP server without requiring any specific WEB services including CGI or data base setup. A HiPS browser displaying a such survey is able to directly access the tiles corresponding to the current user view, without the need of a data base access to locate the data. It just needs HEALPix lib for computing the list of HEALPix tile indices covering the current user view, and in this way, it can build the proper URLs and get the required tiles. HiPS has been designed for image surveys with the idea of preserving the scientific information (astrometry and photometry). However, various extensions are described in this note for other data types including astronomical source catalogues, collections of pointed mode observations like HST and three dimensional data cubes like radio-data. HiPS was originally designed by CDS in 2010 for the Aladin project. This method has rapidly overtaken its original scope. Several HiPS-aware clients are already available, notably Aladin Desktop and Aladin Lite from CDS ( MIZAR from CNES ( JUDO2 from JAXA ( MMI from ESAC (prototype phase). There are already several HiPS providers, notably CDS, SSC-XMMM, CADC, IRAP, IAS, JAXA, ESAC that represent hundreds of HiPS for several terabytes of data. The list of public HiPS data sets can be retreived from this URL: Two HiPS generator tools are already available: Hipsgen.jar dedicated to image and cube surveys, and Hipsgen-cat.jar dedicated to catalogues ( The full context and motivation of HiPS can be found in Fernique et al This note describes HiPS version HiPS principle HiPS is based on the HEALPix sky tessellation and is essentially a mapping of survey data at various spatial resolutions into a collection of HEALPix tiles. The choice of the HEALPix tessellation is described in the MOC IVOA standard 1.0 section 1.4.

4 HEALPix is a curvilinear partitioning of the sphere that supports a hierarchical tree structure for multi-resolution applications. The detailed geometry and properties of HEALPix are described in Górski et al. (2005). The HEALPix partitioning of the sphere has a base resolution that divides the sphere into 12 diamonds, each of which is sub-divided into 4 diamonds recursively. Thus the sphere at order 1 consists of 48 diamonds, 192 diamonds at order 2, 768 at order 3 and so on. Each diamond at a given level is covers an equal area of the sphere. Each HiPS tile corresponds to a HEALPix diamond fully identified by its HEALPix order and its diamond number in this order (NESTED numbering scheme cf REC MOC 1.0 section 2.2.2). 3 HiPS directory structure A HiPS stores all its tiles as a collection of files in directory tree. The structure of these directories follows the simple hierarchy: order tiles, by using respectively the prefix Norder for orders, and Npix for tiles. To avoid too large directories, the tiles are grouped by items, using the subdirectory prefix Dir. A HiPS follows this directory structure: Norder3 Dir0 Npix0 Norder6 Dir0 Npix0 Npix1 Dir10000 Npix10000

5 Note: The actual implementation of HiPS as directories and files is not an obligation, only the view as directories and files is required (see HiPS distribution section). Internally, a HiPS can be stored in a data base, or any other packaging method (tar or zip files ) rather than a simple file system directory structure. 3.1 HiPS tile format The contents of the HiPS tiles depends of the nature of the original survey: pixel arrays for an image survey, a catalog source list for catalog survey, cube arrays for a cube survey, vector arrays for polarization data, properties for localized meta-data (progenitors), etc. This note describes the file format dedicated to images, catalogs and cubes providing respectively Image HiPS, Catalogue HiPS, and Cube HiPS. A HiPS tile contains the data (pixels, catalog sources...) located on its sky area. The tile format depends on the survey data type: FITS, JPEG, PNG for image or cube surveys; TSV for catalogs Image tile format It would be impractical to access and transfer individual HEALPix pixels, so the HiPS scheme groups pixels into tiles as the basic unit of a map that is transferred when any of its pixels are needed. The specification of the tiles makes use of the hierarchical properties of HEALPix to define the tiles as the HEALPix pixels of a lower order HEALPix grid, so that both the tiles and the pixels within the tiles are described by the HEALPix geometry. So an image tile is a bitmap, containing NxN pixels, usually N=512. These pixels represent the HEALPix sub-orders of the tile (9th sub-order for N=512, ie 2^9). The method for evaluating these pixel values depends on the HiPS creation resampling algorithm to map the original image pixels into the HEALPix grid. It can be the nearest pixel, a bilinear interpolation or any other algorithm. Also, in case of overlapping original images, an average method, weighted or not, can be applied. Example. The pixel (0,0) of the tile Npix32 of the Order3 correspond to the original pixel located on the sky at the HEALPix position Npix(32*512*512) for the Order(3+9). Note: If the original data are provided natively as HEALPix maps (for instance PLANCK mission), there is no need to resample the original pixels, just to map them in the adequate tile location. Note: Inside a tile, the pixels are kept in the natural spatial ordering (and not in the HEALPix NESTED numbering ordering). This way, an image tile can be displayed as a first approximation - as a simple bitmap image.

6 An image tile can be stored in FITS format keeping the original full dynamic range of the data values. Tiles stored in JPEG format provide good file compression, and tiles stored in PNG format provides the capability to manage transparency. The tile file extension corresponds to the data type (respectively.fits,.jpg,.png). A HiPS can be delivered simultaneously in various formats (ex: FITS and JPEG), thus different types are then identified by their respective file extension Catalogue tile format A catalogue tile is a list of sources (i.e. an extraction of some rows from a given catalog). These sources are located on the sky in the corresponding HEALPix diamond. If the number of sources in a tile is bigger than a certain limit, the supplemental sources are stored in the 4 sub-tiles in the following order, and so on, recursively. This limit is not necessary a constant and depends of the algorithm to generate the HiPS for the catalog (i.e. local density, level, ). A catalog tile is stored in TSV (Tab Separated Value) with the extension.tsv. All tiles must have the same columns (i.e. the same column ordering). The first line is a header line providing the column names Cube tile format To produce HiPS for data cubes, each cube have to be split in separate frames, each frame corresponding to a slice in the third direction of the cube (Z-axis) and

7 have to be homogeneous for each data cube in a given collection to be able to generate a uniform HiPs. Each of the frame tiles has a suffix _n where n is the depth index within the cube. The tile name without the suffix corresponds to the first frame (depth = 0). In this way a HiPS client that is not HiPS-cube compatible will still be able to display at least the first frame of the HiPS cube as an HiPS image. As with HiPS images, the cube frame tiles can be in FITS, in JPEG and/or PNG format. 3.2 Low resolution enablers Even with the HiPS hierarchical tiling method, the spherical nature of astronomical surveys can be complex/slow to draw for large regions (> 30 deg) due to: The number of tiles required for drawing large regions, even at low HiPS order. The high level of distortion of these tiles with a basic/fast 4 corners bilinear drawing method. To facilitate the client drawing algorithm, the following improvements MAY be implemented for the first 4 lower orders [0..3]: Order omission The low orders: Order0 (12 tiles), Order1 (48 tiles), Order2 (192 tiles) MAY be simply omitted, meaning that the survey files are not provided at these low resolutions Allsky preview file At low orders (0 to 3), the tiles MAY be concatenated into a unique file called Allsky located in the NorderX corresponding directory. However the associated regular tiles do not need to be removed (for instance for basic HiPS clients). HiPS image Allsky: The Allsky file is built as an array of tiles, stored side by side in the natural spatial order. The width of this array is the square root of the number of the tiles of the order. For instance, the width of this array at order 3 is 27 ( (int)sqrt(768) ). To avoid too large Allsky file, the resolution of each tile can be reduced (typically 64x64 pixels rather than 512x512). HiPS catalog Allsky: All sources at the order may be stored in a unique file called Allsky. HiPS cube Allsky : The Allsky file adapted to HiPS cubes follows the same rule that regular cube frame tiles: it is divided in Allsky_nnn files

8 for providing each frame separately. Without suffix, the Allsky file corresponds to the frame Meta data Four additional elements MUST, SHOULD or MAY be used for specifying survey meta data: 1- in the properties file, 2 - in the MOC file, 3 - in the metadata file, 4 in the index.html file Properties file A file named properties, stored in the root of the HiPS directory structure MUST be provided for generic meta information such as identification, copyright, creation date, pixel range... The syntax is ASCII, one line per property, following the syntax KEY = value. Example: publisher_did = ivo://ssc/p/xmm/epic obs_title = XMM-Newton stacked EPIC images obs_ack = HE-Team (Strasbourg), SSC XMM-Newton obs_copyright = (c) ESA / SSC XMM-Newton obs_copyright_url = t_min = t_max = obs_regime = X-ray em_min = 1e-10 em_max = 6e-9 dataproduct_type = image moc_sky_fraction = hips_creation_date = T14:07Z hips_builder = Aladin/HipsGen v8.175 hips_version = 1.4 hips_release_date = T08:46Z hips_publisher = SSC XMM-Newton hips_frame = equatorial hips_order = 7 hips_tile_width = 512 hips_service_url = hips_status = public master clonable hips_tile_format = png fits The following keywords are mandatory: publisher_did, obs_title, hips_frame, hips_order, hips_tile_format, hips_version. The table below describes the list of available keywords. The vocabulary re-uses as far as possible the ObsCore IVOA vocabulary and syntax. This list is not limited and new keywords MAY be added if required. Some keywords MAY be repeated for specifying several values (cf. *) publisher_did Unique ID of the data set - Format : IVORN - Ex : ivo://cds/p/2mass/j

9 publisher_id obs_collection obs_title obs_description obs_ack prov_progenitor (*) bib_reference (*) bib_reference_url (*) obs_copyright obs_copyright_url obs_regime (*) data_ucd (*) Unique ID of the HiPS publisher Format : IVORN - Ex : ivo://cds Short name of original data set Format : one word Ex : 2MASS Data set title Format : free text, one line Ex : HST F110W observations Data set description Format : free text, one paragraph Acknowledgment mention. Provenance of the original data Format : free text Bibliographic reference URL to bibliographic reference Copyright mention Format : free text URL to a copyright mention General wavelength Format: radio, IR, optical, UV, EUV, X-ray, Gamma-ray UCD describing data contents hips_version Number of HiPS version Format : 1.4 hips_builder hips_publisher hips_creation_date hips_release_date hips_status hips_estsize hips_frame hips_order hips_order_min Name and version of the tool used for building the HiPS Format : free text Institute or person who built the HiPS Format : free text Ex : CDS (T.Boch) HiPS first creation date - Format : ISO 8601 => YYYY-mm-ddTHH:MMZ Last HiPS update date - Format : ISO 8601 => YYYY-mm-ddTHH:MMZ Mirror HiPS status Format : master, mirror, partial, private Default : master HiPS size estimation Format : integer Unit : KB Coordinate frame reference Format : equatorial, ecliptic, galactic No default => must be provided Highest HiPS order Format : integer Lowest HiPS order Format : integer hips_tile_width Tiles width in pixels Format : integer Default : 512 hips_tile_format List of available tile formats, blank separated. The first one is the default suggested to the client Format : jpeg, png, fits hips_pixel_cut Default pixel cut range (physical values) Format : min max Ex : hips_data_range Default data range (physical values) Format : min max Ex : hips_sampling hips_overlay Sampling applied for the HiPS generation Format : none, nearest, bilinear Pixel composition method applied on the image overlay region during HiPS generation Format : add, mean, first, border_fading, custom hips_skyval Sky background subtraction method applied during HiPS generation Format : none, hips_estimation, fits_keyword hips_pixel_bitpix data_pixel_bitpix dataproduct_type dataproduct_subtype hips_progenitor_url hips_cat_nrows hips_cube_depth Fits tile BITPIX code Format : -64, -32, 8, 16, 32, 64 (FITS convention) Original data BITPIX code - Format : -64, -32, 8, 16, 32, 64 (FITS convention) Type of data Format : image, cube, catalog, meta Subtype of data Format : color, live URL to an associated meta progenitor HiPS Number of rows of the HiPS catalog Format: integer Depth of the HiPS cube Format : integer hips_cube_firstframe Initial first index frame to display for a HiPS cube Format : integer Default : 0 data_cube_crpix3 Coef for computing physical canal value (see FITS doc) Format : real

10 data_cube_crval3 data_cube_cdelt3 data_cube_bunit3 hips_rgb_red Coef for computing physical canal value (see FITS doc) Format : real Coef for computing physical canal value (see FITS doc) Format : real Third axis unit (see FITS doc) Format : string Original HiPS red component parameters of a HiPS color composition Format: HipsID [cutlow cutmiddle cuthight TransferFunction] hips_rgb_green Original HiPS green component parameters of a HiPS color composition Format: HipsID [cutlow cutmiddle cuthight TransferFunction] hips_reg_blue hips_initial_ra hips_initial_dec hips_initial_fov hips_pixel_scale Original HiPS blue component parameters of a HiPS color composition Format: HipsID [cutlow cutmiddle cuthight TransferFunction] Default RA display position Format : real (ICRS frame) Unit : degrees Default DEC display position Format : real (ICRS frame) Unit : degrees Default display size Format : real Unit : degrees HiPS pixel angular resolution at the highest order Format : real Unit : degrees s_pixel_scale Best pixel angular resolution of the original images Format : real Unit : degrees t_min Start time of the observations Format : real Unit: MJD (*) t_max Stop time of the observations Format : real Unit: MJD (*) em_min em_max Start in spectral coordinates Format: real Unit: meters Stop in spectral coordinates Format: real Unit: meters client_category '/' separated keywords suggesting a display hierarchy to the client Ex : Image/InfraRed client_sort_key Sort key suggesting a display order to the client inside a «client_category» Format : free text Sort : alphanumeric addendum_did moc_sky_fraction In case of live HiPS, provider_did of the added HiPS (must be multiple) Fraction of the sky covers by the MOC associated to the HiPS Format : real between 0 and 1 (*) MJD can be easily computed from Unix time: MJD = (Unixtime / 86400) MOC file A file called Moc.fits, stored in the HiPS root, SHOULD be provided for specifying the coverage map of the survey. This file format and content have to follow the published IVOA MOC standard. The client can use it as a summary of the coverage of the HiPS and to hence avoid trying to load the tiles outside the coverage area of the HiPS. Note: As the MOC 1.0 standard only accepts equatorial system, the galactic or ecliptic HiPS, not covering all the sky, are presently provided without MOC Metadata file A file called metadata.xxx, stored in the HiPS root MAY be provided to specify the metadata information such as column description, FITS keywords. This information depends on the survey data type:

11 HiPS image: the meta data dictionary is stored as a FITS header keywords (FITS convention) in metadata.fits file for providing generic FITS header keywords. This file can be a simple FITS header, without any data value or could even be a regular FITS file from the original survey in order to capture the global metadata information of the survey in the second case the image array is simply ignored by the client. HiPS catalog: the meta data are stored in metadata.xml as a fully defined VOTable header (column descriptions, units, ucds, ). In this way all column information associated with the source tiles can be provided via the metadata file. HiPS cube: same as Hips image index.html file A file called index.html, stored in the HiPS root MAY be provided to offer a basic HTML presentation of the survey. This way, any WEB browser loading a given HiPS root directory (i.e. loading the index.html file) will display basic information on this HiPS.

12 4 HiPS access and distribution Some HiPS clients, notably Aladin Desktop, is able to display a local HiPS by using its root directory file name and by deriving the file names of the various HiPS components (tiles, properties, MOC etc.). However, the HiPS are generally used for remote access. For remote access, the distribution of HiPS is generally implemented via a classical HTTP server (apache or similar server). This is implemented by simply copying the HiPS directories and files onto the HTTP server. The HiPS can be distributed via HTTP and/or HTTPS protocol, freely available or protected by any authentication method supported by the http server. Since the remote client knows the base URL of a dedicated HiPS, it will be able to directly query the tiles and the various additional elements of the HiPS structure. Example: XMM-Newton EPIC HiPS URL: Tiles => baseurl/norder{o}/ndir{d}/npix{n}.{ext} where {O} is the order, {N} tile number, {D}={N}/10000, {ext} is fits, png,... ex: properties file => baseurl/properties ex: MOC file => baseurl/moc.fits ex: Allsky files => baseurl/norder{o}/allsky.{ext} where {O} is the order, {ext} is fits, jpg, png, ex: etc. Note: It is not mandatory to package the HiPS into a fixed number of files and directories. It is possible to store the same information using very different methods (like data base storage, etc..) and simply produce and fetched the right information when given the fixed format URL. References Gorski K. et al., HEALPix: A Framework for High-Resolution Discretization and Fast Analysis of Data Distributed on the Sphere, The Astrophysical Journal, Volume 622, Issue 2, pp , 2005ApJ G Fernique et al., Hierarchical progressive surveys. Multi-resolution HEALPix data structures for astronomical images, catalogues, and 3-dimensional data cubes, Astronomy & Astrophysics, Volume 578, p 114, 2015, 2015A&A...578A.114F IVOA MOC 1.0 REC:

13 Appendix A HiPS mirroring management Several astronomical institutes are deploying a such http HiPS server distributing their own HiPS, or mirrored HiPS from other sites. In this context, this appendix describes a minimal basic protocol for managing a HiPS site and the HiPS mirror mechanism based on the IVOA registry. HiPS node, HiPS list A HiPS node is a basic HTTP site which provides HiPS surveys. A HiPS node MUST implement one dedicated URL (static ASCII file or CGI, ) for publishing the list of HiPS surveys. This list is called the HiPS list of the node. The format of this list is directly derived from the properties file associated to each HiPS. It is a collection of ASCII records (key1 = property1\nkey1 = property2...) that are separated by blank lines and have the same syntax and vocabulary that HiPS properties file. The lines beginning with # are considered as comments. Each record MUST provided at least these properties: publisher_did, hips_release_date, hips_service_url, hips_status. Other properties MAY be provided: hips_order, hips_tile_format, hips_estsize, moc_sky_fraction,... See the previous table in the meta data section for property description. The table below describes two additional properties dedicated to HiPS node description: hips_service_url hips_status Base URL of the HiPS Status of the HiPS node Format: list of keywords: public private: allowed for all clients (public), or only for dedicated authorized clients (private) master mirror partial: original HiPS (master), full copy of HiPS (mirror), partial copy of HiPS (partial) clonable unclonable: HiPS mirroring allowed (clonable) or not (unclonable) Default: public master clonable Example: # Hipslist of HiPS node # Date: T13:42Z # Do not copy any HIPS with unclonable status! publisher_did hips_release_date hips_service_url hips_status = ivo://cadc/p/hst/f850lp/r3 = T12:00Z = = public master clonable

14 publisher_did = ivo://cdsp/2mass/h hips_release_date = T12:00Z hips_service_url = hips_status = public mirror unclonable hips_estsize = hips_order = 9 hips_tile_format = fits jpeg dataproduct_type = image moc_sky_fraction = 1 Note: There are no constraints for the HiPS list URL, but a default choice MAY be: where xxxx is the HTTP server hostname (ex: ) Help: This Perl script code: can be adapted in order to generate on the fly the hipslist from a list of HiPS directories. HiPS node registry The HiPS registry represents the list of available HiPS nodes and their associated HiPS list URLs. This role should be managed by the IVOA registry as a new registry capability pointing to the HiPS list of each HiPS node. Global HIPS list The global HiPS list is an URL providing the full list of available HiPS over all HiPS nodes, with all of its associated properties. This global HiPS list is designed to be used by HiPS clients. This role could be done by the IVOA registry directly, or by a dedicated server harvesting regularly the list of Hips lists provided by the VO registry. A prototype of a such server is already deployed at this address: This list is a collection of ASCII records (key1 = property1\nkey1 = property2...) blank line separated (same syntax and vocabulary that HiPS properties file). The lines beginning with # are considered as a comments. An alternative JSON syntax is also provided at this address: And a human readable alternative of this globalhipslist is provided at this URL : HiPS node mirroring policy FIRST OF ALL: BEFORE DOING ANY KIND OF COPY OF HIPS SURVEYS, THE INSTITUTE HOSTING YOUR HIPS NODE MUST HAVE THE AGREEMENT OF THE ORIGINAL DATA PROVIDER FOR COPYING THE

15 DATA PACKAGED AS HIPS AND/OR TO REDISTRIBUTE THE RESULTING MIRRORED HIPS. PLEASE CAREFULLY OBSERVE THE COPYRIGHT NOTICE OF EACH HIPS THAT YOU WOULD LIKE TO MIRROR. THE FACT THAT A COPY IS TECHNICALLY POSSIBLE DOES NOT MEAN THAT YOUR INSTITUTE OR YOURSELF ARE AUTHORIZED TO MAKE SUCH A COPY. After this legal prerequisite, any HiPS node can mirror a HiPS survey from other HiPS node on condition that the HiPS survey is described by the HiPS node list, and without the property hips_status = unclonable. There is no dedicated protocol for the mirroring process. It can be done via wget, rsync, Hipsgen (MIRROR action) or any other method used for synchronized HTTP web sites. A HiPS node which provides a copy of a HiPS survey MUST specify hips_status = mirror... for this HiPS. A HiPS node which provides a copy of a HiPS survey MUST NOT modify the hips_release_date of the master site in order to detect out of date copies. If the mirroring is partial (lower hips_order, or only a subset of tile formats), the node who has done the mirroring must specify in its HiPS list hips_status = partial, and adjust the relevant properties (hips_order, hips_tile_format), in the HiPS list and in the properties file of the concerned HiPS.

16 Appendix B Suggested client algorithms for displaying HiPS Image HiPS For a given region: 1. Compute the relevant HiPS order (one screen pixel should be cover by one tile pixel) 2. Compute the list of HEALPix tile numbers covering the region (HEALPix lib function) 3. Retrieve the corresponding tiles (locally or via the net) 4. Draw each tile on its sky projected location based on the 4 corner sky localizations (=> affine transformation), or based on more control points by subdividing each tile in sub HEALPix tiles for reducing the projection distorsions. Client display algorithm adaptation for taking into account Allsky preview files: the client/browser drawing a HiPS at very low resolution (whole sky area) should look first if a Allsky.xxx file is existing in the NorderX directory, and if yes, it should load it, split it as a very low resolution tile collection, and draw them. A client can decide to also use the regular individual NorderX tiles as an intermediate zoom level before drawing the Norder(X+1) tiles. Catalog HiPS For a given region: 1. From the lower order: 1. Compute the list of HEALPix tile numbers covering the region (HEALPix lib function) 2. Retrieve the corresponding tiles (locally or via the net) 3. Draw the sources inside the tiles (position on the sky and/or associated measurement information) 4. Continue with the next order if required (depending of the algorithm: number of drawn sources, order number compared to the view size ) Client display algorithm adaptation for taking into account Allsky preview files: the client/browser who must display a HiPS cat at very low resolution (whole sky area) should look first if a Allsky.xxx is existing in the NorderX, and if yes, it should load it and display the corresponding sources. In this case, the regular NorderX tiles must be ignored for avoiding duplicates sources.