UNIVERSITY OF HAWAII AT MANOA Institute for Astrononmy

Transcription

1 Pan-STARRS Document Control PSDC-xxx-xxx-01 UNIVERSITY OF HAWAII AT MANOA Institute for Astrononmy Pan-STARRS Project Management System PS1 Postage Stamp Server System/Subsystem Description Grant Award No. Prepared For Prepared By Document No. Document Date Revision : F : panstarrs team : Jan Kleyna : PSDC-xxx-xxx-01 : October 12, 2006 : 01 DISTRIBUTION STATEMENT Approved for Public Release Distribution is Unlimited c Institute for Astronomy, University of Hawaii 2680 Woodlawn Drive, Honolulu, Hawaii An Equal Opportunity/Affirmative Action Institution

2 Submitted By: [Insert Signature Block of Authorized Developer Representative] Date Approved By: [Insert Signature Block of Customer Developer Representative] Date Postage Stamp Server ii October 12, 2006

3 Revision History Revision Number Release Date Description First draft Postage Stamp Server iii October 12, 2006

4 TBD / TBR Listing Section No. Page No. TBD/R No. Description xxx-xxx Resolve issue of difference-image postage stamps Postage Stamp Server iv October 12, 2006

5 Referenced Documents Internal Documents Reference PSDC Title Data Store SDD Postage Stamp Server v October 12, 2006

6 Contents 1 The PS1 Postage Stamp Server Desired qualities Potential Concerns Implementation of Postage Stamp Server Postage Stamp Server Software Interfaces Network Layer Transport Layer Session Layer Presentation Layer Postage Stamp Server Data Store Protocol Postage Stamp Query Postage Stamp Response Postage Stamp Image Data Store Data Transfer Parameters vi

7 1 The PS1 Postage Stamp Server The PS1 Image Processing Pipeline (IPP) will store a complete archive of all exposures. It is desirable to have the ability to retrieve postage stamp sub-images from the archive. Postage stamps will be particularly useful for the study of transient events and moving objects. Hence it is necessary to develop a Postage Stamp Server to serve postage stamps to various clients, using a published protocol. This document describes a preliminary specification of such a client/server protocol. 1.1 Desired qualities The postage stamp server should have following qualities 1. It should communicate using the Data Store interface (PSDC ), whereby requests for one or more stamps to the postage stamp server are submitted as FITS table files, the response from the Data Store is FITS table file, and the individual stamps are FITS image files, the names of which are listed inside the response FITS table file. The client will submit the query FITS table, retrieve the response FITS table by polling the Data Store, and then retrieve the individual stamps in the response table by polling the Data Store. 2. The ability to retrieve a sub-image of a specified size, around a specified RA and Dec, at a specified time. If no such image exists, or if the position falls onto a blind region of the focal plane, an appropriate error flag should be returned in the response FITS table file. 3. The ability to specify whether the returned image is a raw image, or a difference image relative to the current static sky. If it is impossible to create a difference image, an appropriate error flag should be returned in the response FITS table file. 4. The ability to specify whether 1) the pixels of the output stamp image are raw pixels with a WCS containing focal plane distortion terms, or 2) the stamp coordinates are warped onto static sky pixel coordinates, with a simple WCS like a tangent projection. 1.2 Potential Concerns Some potential concerns about the postage stamp server model described above are: 1. The computational burden of serving stamps may be significant. If a total of 10 8 moving object detections are anticipated over 10 3 days, and if a stamp is desired for each detection of a moving object, then the system will have to produce 10 5 stamps per day, or one stamp per second. This figure is probably an upper limit, however, because it is unlikely that a stamp will be requested for each transient detection, if only from analysis limitations at the client side. 2. The computational burden of serving difference image postage stamps will be far greater than the burden of serving raw image stamps. Given the current update of the static sky, static sky subtraction requires the creation of a kernel for each image (FPA). The creation of a kernel is expensive, requiring computations involving the entire image, not merely the stamp region. Even if a kernel exists for a particular image at a particular time, the updating of the static sky will necessitate the computation of a new kernel. Postage Stamp Server 1 October 12, 2006

8 These computational difficulties are ameliorated slightly by the fact that a kernel may be reused for a particular image as long as the static sky has not changed, so that the number of kernel computations required is constrained by the number of images from which stamps are extracted, not by the number of stamps. 2 Implementation of Postage Stamp Server 2.1 Postage Stamp Server Software Interfaces Network Layer Internet Protocol v4 will be used. Data Store s HTTP server will have an IP address visible (routable) to the client Transport Layer TCP will be used on top of IPv Session Layer Postage Stamp client-server communication will take place entirely via the Data Store mechanism, described in the Data Store SDD PSDC Presentation Layer All data exchanged between the Postage Stamp client and server will be in the form of FITS Tables and FITS images. 2.2 Postage Stamp Server Data Store Protocol Postage Stamps will be retrieved according to the following protocol, transmitted over HTTP, as illustrated in Client will submit a Postage Stamp Query to the Postage Stamp Server Data Store. The query will consist of a FITS table, with one row per stamp. The rows of the FITS table will contain the requested stamp positions, sizes, times, and filters. 2. Client will poll the Data Store until it receives a Postage Stamp Response consisting of a FITS table having one row per stamp. Each row will contain flags denoting success or failure, as well as the file name of the fits stamp image file for that row. 3. Client will poll the Data Store for each image (file name) that was successfully created. Postage Stamp Server 2 October 12, 2006

9 C S P o s t a g e S t a m p Q u e r y ( F I T S T a b l e ) P o s t a g e S t a m p l i e n t C l i e n t p o l l s f o r R e s p o n s e... P o s t a g e S t a m p R e s p o n s e ( F I T S T a b l e ) C l i e n t p o l l s f o r F I T S s t a m p i m a g e 1... P o s t a g e S t a m p I m a g e 1 ( F I T S I m a g e ) P o s t a g e S t a m p S e r v e r D a t a t o r e C l i e n t p o l l s f o r F I T S s t a m p i m a g e N... P o s t a g e S t a m p I m a g e N ( F I T S I m a g e ) Figure 1: Diagram showing protocol over HTTP between postage stamp server and a client. Client submits query as a FITS table, then polls the server until it receives a FITS table response. The client then retrieves the postage stamps listed in the response by again polling the server for each stamp image Postage Stamp Query A client will send a Postage Stamp Query to a Postage Stamp Server implemented through the Data Store protocol. The query will consist of a fits table file with EXTNAME=POSTAGE STAMP QUERY, with FITS headers given in Table 1, and with FITS table columns given in Table 2. The client will then poll the Data Store for the response named after the query s QUERY ID until it receives a FITS response table. The QUERY ID must be be alphanumeric with optional underscores, and must be unique to each client or user. Hence the query ID will be of the form CLIENT xxxx where CLIENT is a unique alphanumeric string of 8 or fewer characters assigned to each client, and xxxx is an alphanumeric code of 21 or fewer characters. It is the responsibility of the client to ensure that the code xxxx is unique to the client. Some potential components of xxxx are the current Universal Time in seconds, the local user name, the client s Unix process ID, the client s IP address, or an incremented counter. If two queries with the same QUERY ID are submitted to the server, then the later query will over-ride the previous query, although the server will not guarantee the non-existence of invalid intermediate states. Table 1: Postage Stamp Query Keywords Column Precision Description QUERY ID char(30) client query ID for this batch query, with prefix unique to each client OBSCODE char(3) site identifier (MPC observatory code) Postage Stamp Response The Postage Stamp Server will respond to a Postage Stamp Query in a Postage Stamp Response FITS file transmitted through the Data Store protocol. This file will be named after the QUERY ID followed by the suffix.fits. The Postage Postage Stamp Server 3 October 12, 2006

10 Table 2: Postage Stamp Query Table Columns Column Precision Description ROWNUM char(20) identifier for this row in the table RA DEG F64 coordinate at the start of exposure, in degrees DEC DEG F64 coordinate at the start of exposure, in degrees MJD REQ F64 requested time, which may fall anywhere in exposure, MJD REQFILT char(3) filter used for the exposure (string); allowed values of g, r, i, z, y, B, V, w, ANY or NULL NX U32 width of stamp in pixels (RA direction if warped into static sky WCS) NY U32 height of stamp in pixels (Dec direction if warped into static sky WCS) DIFF Boolean TRUE if image is a difference image, rather than raw image WARPSS Boolean TRUE if image is to be warped onto static sky WCS, rather than raw pixel coordinates MAKE PS Boolean TRUE if an actual postage stamp image is to be made, in addition to a response table. Columns of the Postage Stamp Query. REQFILT is the requested filter, which may be the string any or NULL if any filter is acceptable. NX and NY are either raw image pixel sizes, in the raw image chip convention, or pixels in the RA and Dec directions if WARPSS is set to TRUE to warp the image onto the static sky WCS. stamp response will have the header EXTNAME=POSTAGE STAMP RESPONSE and the additional headers in Table 3. The columns of the response will be as given in Table 4. The ERRORCODE column of the response table will be set by performing a bitwise OR of the values in 5. If no fatal error was encountered, and if the MAKE PS field of the Postage Stamp Query was TRUE, then the MADE PS column of the stamp s row will be set to TRUE, and the FILENAME will contain the complete name of the FITS file containing the postage stamp image in the Data Store. Table 3: Postage Stamp Response Keywords Column Precision Description QUERY ID char(30) client query ID for this batch query, with prefix unique to each client Keywords in the Postage Stamp Response. The QUERY ID is the same as the client-generated QUERY ID in the Postage Stamp Query Postage Stamp Image Upon receiving a Postage Stamp Response, a client will poll the Data Store for the FITS images contained in the FILE- NAME column of the response, for the cases that the FILENAME is not NULL. Each FITS postage stamp image file will contain headers corresponding to all the stamp s columns in the Postage Stamp Response (Table 4), in addition to the headers in Table 6. The FITS keyword names of these additional headers from Table 4 will be the same as the column names in Table 4, and the FITS keyword values will be printed representations of the table values, following the normal FITS conventions for representing integers, floating point numbers, character strings, and booleans Data Store Data Transfer Parameters The Data Store will support Postage Stamp Response and Postage Stamp Image lifetimes and polling intervals as given in Table 7. Assuming an upper limit of 10 5 stamps per day, at 10 5 bytes per stamp, one day of requests will require 10 Postage Stamp Server 4 October 12, 2006

11 Table 4: Postage Stamp Response Table Columns Column Precision Description ROWNUM char(20) identifier for this row in the table RA DEG F64 requested J2000 coordinate of the stamp center, in degrees DEC DEG F64 requested J2000 coordinate of the stamp center, in degrees MJD REQ F64 requested exposure time, MJD REQFILT char(3) requested filter used for the exposure as a string, allowed values of g, r, i, z, y, B, V, w, ANY NX U32 width of stamp in pixels (RA direction) NY U32 height of stamp in pixels (Dec direction) DIFF Boolean TRUE if image is a difference image, rather than raw image WARPSS Boolean TRUE if image is to be warped onto static sky WCS, rather than raw pixel coordinates MAKE PS Boolean TRUE if an actual postage stamp image is to be made, in addition to a response table. MJD OBS F64 actual time of start of exposure, MJD RA OBS F64 J2000 field center RA at start of exposure, in degrees DEC OBS F64 J2000 field center Dec at start of exposure, in degrees MADE PS Boolean TRUE if an actual postage was made. Some errors may not preclude making a stamp. IMG NAME char(40) name of the postage stamp, if no fatal errors, and if MAKE PS is TRUE ERRORCODE U64 OR-combined numerical code enumerated below, containing error, if any FILT char(3) actual filter retrieved; may differ from REQFILT if REQFILT was ANY or NULL EXPTIME U64 exposure time of parent image FPA ID char(20) original FPA ID used at ingest FILENAME char(40) server-determined FITS image stamp file to retrieve from the Data Store Columns of Postage Stamp Response. Columns with names identical to those in the Postage Stamp Query are duplicates of corresponding values in the query. A non-zero ERRORCODE may still produce a stamp, if the error was non-fatal, such as a central RA,Dec that lies on a chip gap. If no stamp is produced, then MADE PS is FALSE and FILENAME is NULL. gigabytes of storage. Hence the 10 day file lifetimes specified in Table 7 will require 100 gigabytes of storage, or one disk. Postage Stamp Server 5 October 12, 2006

12 Table 5: Postage Stamp Response Error Codes Name Value Description NO ERROR 0 No Error NO IMAGE FOR TIME 1 No image at this observation MJD RA DEC NOT ON A PIXEL 2 The central RA,Dec are on a blind area, like a gap RA DEC NOT IN FOV 4 Wrong position on sky for MJD given NO IMAGE FOR FILTER 8 Image is not in the filter requested NOT AVAILABLE PERM 16 Image was taken, but is permanently unavailable NOT AVAILABLE TEMP 32 Image was taken, but is currently inaccessible Error codes, combined in bitwise OR, to be returned in ERRORCODE column of a postage stamp response. Not all error codes preclude the creation of a stamp. Table 6: Postage Stamp Image Keywords Column Precision Description QUERY ID char(30) client query ID for this batch query, with prefix unique to each client ROWNUM char(20) identifier for the corresponding row in the query and response tables FPA ID char(20) IPP-assigned identifier that can be used to trace back to this FPA MJD-OBS F64 midpoint time of the exposure as a MJD and day fraction RA HH:MM:SS.SSS field center RA at exposure midpoint, string DEC sdd:mm:ss (s is + or -) field center declination at exposure midpoint, string EXPTIME F64 exposure time in seconds ROTANGLE F64 image rotation of the y-axis in degrees, from local N toward E FILT char(3) effective filter used for the exposure, one of g, r, i, z, y, B, V, w AIRMASS F64 airmass at exposure midpoint LIMITMAG F64 limiting magnitude of detections for this FPA DE1 through DE10 F64 detection efficiency coefficients OBSCODE char(5) MPC three-character observatory code TEL ALT F64 telescope altitude at exposure midpoint, in degrees TEL AZ F64 telescope azimuth at exposure midpoint, in degrees Keywords in the Postage Stamp images. The QUERY ID is the same as the client-generated QUERY ID in the Postage Stamp Query. Additionally, the image should contain headers corresponding to the columns of the Postage Stamp Response, and valid WCS headers. Table 7: Data Store data transfer parameters Parameter Postage Stamp Response Postage Stamp Image Max polling Interval 60s 60s File Lifetime 10 days 10 days Postage Stamp Server 6 October 12, 2006