ILOSTAT SDMX Gateway. Dissemination API Reference Guide. Department of STATISTICS Publication Date:

Transcription

1 Dissemination API Reference Guide Author: Department of STATISTICS Publication Date:

2 2

3 Table of Contents Document changes since last version ii I Data and metadata dissemination 3 I1 Description 3 I11 Overview 3 I12 URL format 3 I2 Structural metadata 7 I21 Common query string parameters 7 I22 Agency scheme 7 I23 Concept scheme 7 I24 Code lists 7 I25 Data structure definitions 9 I26 Data flows 10 I3 XML Validation Schema 11 I4 Data and metadata dissemination 11 I41 Selection of a dataflow 12 I42 Filtering 12 I43 Query string parameters 13 I44 Attributes and descriptive metadata 14 Page i

4 Document changes since last version Version Date Author Comment 20 E Greising * Included notes annotations * Corrected codes used for aggregate reference areas in CL_COUNTRY * Clarification on the usage of version when querying items on structural metadata queries * Removed references to versioning system (deprecated) * Described new implementation for dataflow queries including partial indicator codes and MULTI keyword * _Z replaces _@NA@ for not applicable Page ii

5 I Data and metadata dissemination I1 Description I11 Overview The dissemination service is a partial implementation of the standard SDMX RESTful API, as defined in the SDMX Standards: Section 7 Guidelines for the use of Web services This API defines a set of HyperText Transfer Protocol (HTTP) resources which give access to: Structural metadata (code lists, concepts, definition of data structures, etc) Data and informational metadata (values, notes) I12 URL format The defined resources are all accessible through HTTP GET requests, using a Web browser or any HTTP-enabled software component Each resource is associated to a Uniform Resource Locator (URL), as summarized in the picture next page From a technical point of view, the Base URL is the deployment path of the Java servlet that implements the SDMX RESTful interface The SDMX connector is currently deployed under the following Base URLs: Development/Integration testing: Production/public service: Structural metadata queries: /resource/agencyid/resourceid/version/itemid?querystringparameters Resource: agencyid: resourceid: The type of metadata to be returned Valid values: datastructure, conceptscheme, codelist, dataflow Not implemented queries: metadatastructure, categoryscheme, agencyscheme, hierarchicalcodelist, organisationscheme, organisationunitscheme, process, reportingtaxonomy, structure, dataproviderscheme, dataconsumerscheme, metadataflow, provisionagreement, structureset, categorisation, contentconstraint, attachmentconstraint Agency maintaining the artefact (eg: ILO) Default: ILO Artefact ID (eg: CL_FREQ) Default: all Page 3

6 version: itemid: querystringparameters: detail: Arterfact version (eg: 10) Default: latest ID of the Item to be returned (eg: Q) Valid for Item schemes only Default: all Desired amount of information to be returned References: Valid values: allstubs, referencestubs, full Default: full References to be returned with the artefact Valid values: none, children, descendants, any type of resource Default: none Data queries: Not implemented: parents, parentsandsiblings, all /resource/flowref/key/providerref?querystringparameters Resource: flowref: key: providerref: querystringparameters: Type of query Valid value: data Not implemented: metadata Dataflow ref (eg ILO,DF_MIG_EMP_TEMP_SEX_AGE_MIG_NB,latest) Key of the series to be returned (eg: KIFRA2854AEAP_DWAF_RT) Wildcarding (eg: STIESPQ) and ORing (eg: YIKHI+JPNA) is supported Data provider (eg: ILO) Default: all startperiod: Start period (inclusive) ISO8601 (eg ) or SDMX reporting period (eg 2014-Q3) endperiod: End period (inclusive) ISO8601 (eg ) or SDMX reporting period (eg 2014-M10) If dates are written in ISO8601 format (with dashes, ie YYYY-MM-DD), every point-in-time which reference period is entirely comprised between startperiod and endperiod (inclusive) will be included in the query response firstnobservations: Maximum number of observations starting from the first observation lastnobservations: Maximum number of observations counting back from the most recent observation dimensionatobservation: Id for the dimension attached at the observation level Default: TIME_PERIOD detail: Not implemented: updatedafter: Desired amount of information to be returned Valid values: serieskeysonly, dataonly, nodata, full Default: full Last time the query was performed Used to retrieve deltas Must be percent-encoded (eg: T14%3A15%3A00%2B01%3A00) includehistory: Whether to return vintages Default: false Page 4

7 The format parameter allows selecting the message format requested The valid options are: generic_2_0 will deliver a message in Generic Version 20 format (This is the default format) generic_2_1 will deliver a message in Generic Time Series Data Version 21 format compact_2_1 will deliver a message in Structure Specific Time Series Data Version 21 format json will deliver a message in json format (Valid only for data messages) The following sections will describe the resource locators in more details Page 5

8 Page 6

9 I2 Structural metadata I21 Common query string parameters Each URL is composed of a mandatory path and an optional query string, respectively the parts of the URL before and after the question mark (? ) sign The parameters allowed in the query string of a structure query are defined at paragraph 4322 of the SDMX standard, section 7 In the ILO implementation, the detail parameter is not mandatory, defaulting to full Furthermore, in this version the references parameter is restricted to the values none, children and descendants, since the cross-referencing of parent structures was not yet implemented in the underlying library at the time it was developed The format parameter allows selecting the message format requested The valid options are: generic_2_0 will deliver a message in Generic Version 20 format (This is the default format) generic_2_1 will deliver a message in Generic Time Series Data Version 21 format compact_2_1 will deliver a message in Structure Specific Time Series Data Version 21 format I22 Agency scheme The agency resource gives access to the agency scheme The ILO implementation currently defines only the ILO agency The identifier and version variables are ignored, the agency scheme is thus accessible under /agency/ilo I23 Concept scheme The conceptscheme resource gives access to the concept scheme The ILO implementation defines the following concept schemes: CS_ILOSTAT: defines the concepts common to the whole ILOSTAT information model and that are not included in the following two concept schemes CS_CLASSIF_TYPE: defines the concepts used as classifications (indicator-bound dimensions) in the ILOSTAT information model CS_NOTE_TYPE: defines the concepts used as notes (descriptive metadata presented as Tablelevel or value-level attributes) in the ILOSTAT information model CS_DISSEM: defines those concepts that are only relevant for dissemination purposes and can be discarded in data collection/data reporting The version variable is ignored (defaulting to latest or 10) I24 Code lists The codelist resource gives access to the code lists used to represent the coded dimensions and attributes (ie classifications and coded notes) All the code lists include Code, Name and Description elements (when available) and some additional fields in the form of annotations that provide useful information for dissemination and/or collection purposes (eg Sort gives an order Page 7

10 of prevalence of the item; Default indicates if certain attribute should be taken into account in a collection process or not) The ILO implementation defines the following code lists: Classification code lists: o One general list per classification type (as found in the T_CLY_CLASSIF_TYPE table in ILOSTAT metadata subsystem), accessible under /codelist/ilo/cl_<cly_code> It includes the description of the type and codes and descriptions for all the classifications items of all classifications versions of this type (eg /codelist/ilo/cl_eco) This code lists include a virtual classification item code of the form _Z which description is Not applicable to be assigned to all classifications not belonging to a certain indicator in a Dataflow based in the generic DSD (See I26 below) o One per classification version (as found in the T_CLV_CLASSIF_VERSION table in ILOSTAT metadata subsystem), accessible under /codelist/ilo/cl_<clv_code> It includes codes and descriptions for the classifications items of this version (eg /codelist/ilo/cl_eco_isic4) Indicator code lists: These code lists are constrained by collection They are accessible under /codelist/ilo/cl_indicator_<collection_code> This query is particularly useful for querying the list of indicators belonging to a collection Subject code lists: o One general list (as found in the T_SUB_SUBJECT table in ILOSTAT metadata subsystem), accessible under /codelist/ilo/cl_subject It includes codes and descriptions for all the subjects in the information model o One per indicator which includes the list of subjects this indicator is related to It is accessible under /codelist/ilo/cl_subject_<indicator_code> o One per collection which includes the list of subjects each indicator in the collection is related to It is accessible under /codelist/ilo/cl_subject_<collection_code> Notes code lists: o One general list per note type (as found in the T_NTY_NOTE_TYPE table in ILOSTAT metadata subsystem), accessible under /codelist/ilo/cl_note_<nty_code> o Special constrained code lists for notes applicable to a source type ( /codelist/ilo/cl_note_<src>) or a Topic( /codelist/ilo/cl_note_<top>) o The notes applicable to an Indicator ( /codelist/ilo/cl_note_<ind>) are the union of those applicable to its allowed types of source, its topic and the indicator itself (To be implemented in next version) Notes code lists include the following annotations: o Sort: The sort/prevalence order o Default (Y/N): Indicates if this note should be included in a collection process The Survey code lists: o One general list for all surveys, accessible under /codelist/ilo/cl_survey Page 8

11 o One per country restricted to the surveys related to this country, accessible under /codelist/ilo/cl_survey_<cou_iso3_code> Country code list: Accessible through /codelist/ilo/cl_country, it will provide the country code according to the ISO-3166 ALPHA3 standard and the name in the three official languages The alphabetical sort order in each language is given as the annotation Sort The annotation Group is set to Y if this item represents a country group and Parent indicates which group it belongs to (only for groups, N for countries) All the codes for items representing groups of countries are defined in the open space X of the ISO-3166 standard using numbers X01 has been assigned to World, and numbers have been assigned to code Income level categories as defined by the World Bank and ILO Geographical regions Africa, Americas, Arab States, Asia and the Pacific and Europe, and its sub-divisions, respectively Common dimensions and attributes: for the representation of indicator-invariant dimensions (frequency, collection, represented variable, etc) and attributes (time format, value status, etc) The version variable is ignored (defaulting to latest or 10) If the identifier variable is not specified, all existing code lists in the information model will be returned - ie /codelist/ilo/ However, to query for one specific item in a scheme, a version must be specified in the URL, and it will default to latest or 10 eg /codelist/ilo/cl_country/1/x01 To get all the code lists related to an indicator definition use the DSD query with the references=children or references=descendants parameter (See I25) I25 Data structure definitions The datastructure resource gives access to the Data Structure Definitions (DSD) which describes the data flows used to collect or disseminate data In the ILO implementation, attributes within the same DSD are used to convey descriptive metadata (notes) Each DSD is identified by the {collection; country; indicator} triplet, where: collection stands for the business code of a collection (ie CL_COLLECTION Code values or T_COL_COLLECTIONCOL_CODE in ILOSTAT metadata subsystem) country stands for the ISO alpha-3 code of a country (ie CL_COUNTRY Code values or T_COU_COUNTRYCOU_ISO3_CODE in ILOSTAT metadata subsystem), or ALL for the generic, dissemination DSD valid for all countries for one specific indicator indicator stands for the business code of an indicator (ie CL_INDICATOR Code values or T_IND_INDICATORIND_CODE in ILOSTAT metadata subsystem) The keyword ALL is allowed to generate a generic DSD for the country with all the dimensions collapsed into a single hypercube, including all classifications used by every indicator Including the TOPIC code instead of the indicator, a DSD for the Topic will be generated by collapsing in a single hypercube all the valid combinations of dimensions and attributes according to the indicators defined in ILOSTAT Similarly, the REPRESENTED_VARIABLE code can be used to obtain the DSD for this Represented Variable The resource /datastructure/ilo/<collection_code>_all_all?references=children can be used to get all code lists in one single query Page 9

12 The keyword MULTI generates a multi-indicator collection DSD for a given country It can also be combined with the Topic-DSD and Represented Variable-DSD features This is useful for querying individual table structures per indicator Under this scheme, the following resources are valid: /datastructure/ilo/yi_mex_emp_temp_sex_age_nb: country specific DSD, for the yearly collection of the Employment by Sex and Age indicator for Mexico /datastructure/ilo/yi_mex_emp_temp_sex_age_nb?references=descendants: same DSD, with full references to the code lists and concepts embedded /datastructure/ilo/yi_all_emp_multi?references=descendants: a single xml message with multiple DSDs corresponding to all the indicators in the topic EMP, including all referenced code lists and concepts /datastructure/ilo/yi_all_une_tune_sex_geo_nb: generic DSD, for the dissemination of the Unemployment by Sex and Geographical coverage indicator /datastructure/ilo/yi_mex_all: country specific generic DSD for Mexico, including all indicators in the Yearly Indicators (YI) collection (200+ as of writing) collapsed in a single hypercube of 19 dimensions: Collection, Country, Frequency, Survey, Represented Variable, 13 classification breakdowns and Time /datastructure/ilo/yi_mex_multi: country specific multiple DSD for Mexico, including all indicators in the Yearly Indicators (YI) collection (200+ as of writing) as independent Key families into a single xml file I26 Data flows The dataflow resource gives access to the data flows used to collect or disseminate data In the ILO implementation, one dataflow is defined for each DSD, identified by DF_<DSD_ID> The default behaviour is the same as when querying DSDs, either for one indicator, grouping by Topic or Represented variable or using the keywords ALL or MULTI (See I25) Under this scheme, the query /dataflow/ilo will return a message including all the dissemination dataflows (ie indicators) with data available, so it is the best way of discovering data availability For simplification, the dataflows included in the response to this query are those representing indicators and the generics by collection (<col>_all_all) Similarly to the datastructure resource query, querying a dataflow based in a generic ALL indicators flow reference like in the following example: /dataflow/ilo/df_yi_mex_all/<filters>?<parameters> will return a single dataflow corresponding to the country specific generic DSD for Mexico, including all indicators in the Yearly Indicators (YI) collection (200+ as of writing) collapsed in a single hypercube On the other hand, the query /dataflow/ilo/df_yi_mex_multi/<filters>?<parameters> will generate a message including all the valid DF for Mexico corresponding to indicators in the YI collection with data available In other words, the API s behaviour is the same as specifying MULTI for DSDs, but constraining the result by data availability, since the main intention of the API is data dissemination Page 10

13 Individual dataflows can also be queried; the following URL gives access to the dataflow for yearly indicators, France, Employment by Sex and Age: /dataflow/ilo/df_yi_fra_emp_temp_sex_age_nb/<filters>?<parameters> As explained for DSDs above, under this logic, the following resources are valid dataflows: /dataflow/ilo/df_yi_mex_emp_temp_sex_age_nb: country specific, for the yearly collection of the Employment by Sex and Age indicator for Mexico /dataflow/ilo/df_yi_mex_emp_temp_sex_age_nb?references=descendants: same dataflow, with full references to the code lists, concepts and data structures embedded /dataflow/ilo/df_yi_all_emp?references=descendants: a xml message with a dataflow making use of the generic DSD for the all the indicators in the topic EMP, including all referenced code lists, concepts and datastructure /dataflow/ilo/df_yi_all_emp_multi?references=descendants: a single xml message with multiple dataflows corresponding to all the indicators in the topic EMP, including all referenced code lists, concepts and respective datastructures /dataflow/ilo/df_yi_all_une_tune_sex_geo_nb: generic dataflow, for the dissemination of the Unemployment by Sex and Geographical coverage indicator /dataflow/ilo/df_yi_mex_all: country specific generic dataflow for Mexico, colapsing all indicators in the Yearly Indicators (YI) collection in a single hypercube /dataflow/ilo/df_yi_mex_multi: country specific multiple dataflow for Mexico, including all indicators in the Yearly Indicators (YI) collection with data available I3 XML Validation Schema The schema URL space gives access to XML Schema files that can be used to validate data flows in SDMX-ML files I4 Data and metadata dissemination The data URL space gives access to the disseminated data and metadata The URL scheme allows: The selection of a given dataflow The optional filtering of data/metadata across dimensions Optional query string parameters The format parameter allows selecting the message format requested The valid options are: generic_2_0 will deliver a message in Generic Version 20 format (This is the default format) generic_2_1 will deliver a message in Generic Time Series Data Version 21 format compact_2_1 will deliver a message in Structure Specific Time Series Data Version 21 format json will deliver a message in json format (Valid only for data messages) Page 11

14 I41 Selection of a dataflow The dataflow variable of the URL scheme is of the form <agency_id>,<dataflow_id>,<version> In the ILO implementation, agency_id is always ILO and version can be omitted So in order to get data from the yearly collection of the Employment by Sex and Age indicator for all countries, the dataflow identifier is: ILO,DF_YI_ALL_EMP_TEMP_SEX_AGE_NB It should be noted that, when querying for data using the generic dataflows (ie those with ALL in the indicator portion of the id, or referring to a topic or represented variable), data is formatted according to the generic hypercube including all the breakdowns in the indicators referenced and thus, certain key combinations may be invalid (do not exist in any indicator) All the classification dimensions of the generic hypercube not corresponding to a certain key family (ie a valid indicator) will be filled with the global Not applicable code _Z (See I24 above) Please take into account that this type of queries in a big collection like YI or STI can be very time consuming I42 Filtering The next component of the URL is called the key in the SDMX specification It is a dot-separated list of values for the dimensions, in the order defined by the DSD related to the dataflow An omitted dimension is wild carded; the logical or operator is supported using the + character The ALL keyword wildcards all dimensions, allowing to get all the data for a given DSD Following on the previous example, the DSD for Employment by Sex and Age 1 defines the following dimensions, most general to most specific: COLLECTION, COUNTRY, FREQ, SURVEY, REPRESENTED_VARIABLE, and afterwards the classifications involved in the indicator like CLASSIF_SEX, CLASSIF_AGE, etc in the order given in the DSD Any filter can be specified either explicitly (by providing a value or expression) or omitted by just adding the separator dot to its right It is not necessary to put all the dots for all the dimensions; the dots to the right of the last filtered dimension can be omitted and will be wildcarded One particular use case is the ability to select data for one specific classification version: all the classification item s codes for this version can be implicitly specified by entering the prefix that identifies that version codes followed by an asterisk (*) to wildcard the classification item codes (ie <CLA>_<CLV>* or AGE_5YRBANDS*) in the proper dimension position to fetch data only for this classification version Similarly, all data from the indicators belonging to one Topic can be obtained by specifying <TOP>*, eg EMP* (see example below) The characters M, Q and A in the FREQ dimension can be used to filter data points with Monthly, Quarterly and Annual time reference periods It is important to note that, while dimension codes and in particular classification item codes, can be ORed with the +, a wildcarded code in such expression will be ignored Examples: 1An Indicator is defined as a Represented variable broken-down by none, one or more classifications The Represented variable code is composed by the Topic (ie EMP), the variant (ie TEMP for Total EMPloyment) and the Type of measure (ie NB for NumBer, RT for RaTe, etc) The indicator code is obtained by inserting the Classification Types codes (ie SEX & AGE) after the variant code: EMP_TEMP_SEX_AGE_NB Page 12

15 /data/ilo,df_yi_all_emp_temp_sex_age_nb/all: fetch all data for the yearly collection /data/ilo,df_yi_all_emp_temp_sex_age_nb/: idem (all dimensions wildcarded) /data/ilo,df_yi_all_emp_temp_sex_age_nb/mex+esp: fetch data for México and Spain, with all other dimensions wildcarded /data/ilo,df_yi_all_emp_temp_sex_age_nb/sex_tage_5yrbands_total: filter on the total classification items (same information as in the multi-country report of the ILOSTAT website) /data/ilo,df_sti_all_emp_temp_sex_age_nb/deu+fra+gbr+itam: fetch monthly data only for Germany, France, the United Kingdom and Italy /data/ilo,df_yi_all_emp_temp_sex_age_nb/yifraa1142sex_fage_5yrbands_15-19+age_5yrbands_60-64: fetch data for France, survey 1142, for female workers, ages and /data/ilo,df_yi_esp_une_tune_sex_age_nb/age_5yrbands_y00-04+age_5yrbands_y05-09+age_5yrbands_y10-14+age_5yrbands_y15-19+age_5yrbands_y20-24+age_5yrbands_y25-29+age_5yrbands_y30-34+age_5yrbands_y35-39+age_5yrbands_y40-44+age_5yrbands_y45-49+age_5yrbands_y50-54+age_5yrbands_y55-59+age_5yrbands_y60-64+age_5yrbands_yge65+age_5yrbands_total: fetch data for Spain for 5-year age bands classification version /data/ilo,df_yi_esp_une_tune_sex_age_nb/age_5yrbands*: The same as above example /data/ilo,df_yi_aut_all/emp_temp_nb: fetch data for all indicators based on the represented variable Total Employment Level (EMP_TEMP_NB) for Austria Note that the filtering of the represented variable is only meaningful for an ALL-indicators type dataflow /data/ilo,df_yi_aut_all/emp*: fetch data for all indicators belonging to the topic Employment (EMP) for Austria Note that this particular case of filtering of the represented variable is only meaningful for an ALL-indicators type dataflow as well I43 Query string parameters The standard SDMX REST interface defines a number of query string parameters that can be used to further tailor the data flow retrieved These parameters are to be specified after a? and concatenated with & The ILO implementation interprets a reduced set of these parameters, as described below: startperiod: the starting date for the period for which results should be supplied (inclusive) endperiod: the ending date for the period for which results should be supplied (inclusive) Accepted formats: ISO8601 (eg ) or SDMX reporting period (eg 2014-Q3) If dates are written in ISO8601 format (with dashes, ie YYYY-MM-DD), every point-in-time which reference period is entirely comprised between startperiod and endperiod (inclusive) will be included in the query response firstnobservations: includes data for the first N time points lastnobservations: includes data for the last N time points detail: the desired amount of information to be returned; must be one of: Page 13

16 o full: default value, return everything (data, annotations, attributes) o dataonly: values only (no attributes) o serieskeysonly: can be used for performance reasons to fetch the series (with specified dimensions) matching a given query o nodata: only attributes and annotations format: The message format Valid options are: o o generic_2_0 will deliver a message in Generic Version 20 format (This is the default format) generic_2_1 will deliver a message in Generic Time Series Dara Version 21 format o compact_2_1 will deliver a message in Structure Specific Time Series Data Version 21 format o json will deliver a message in json format (Valid only for data messages) So, going further on the previous examples, the following request is possible: /data/ilo,df_yi_all_emp_temp_sex_age_nb/yifraa1142sex_tage_5yrbands_total?startp eriod= &endperiod= &detail=dataonly&format=json: fetch the data point for France, survey 1142, total classifications, year 2009 in a json data message I44 Attributes and descriptive metadata For each observation value there will be a set of attributes describing the data point These attributes include certain characteristics of the measured dimension like Unit of measure, Unit multiplier, Currency, Time format, etc and useful information for presentation purposes like the default number of decimal positions and the suggested sort order for the indicators Two particular attributes provide data structure information: the INDICATOR that provides the indicator code and SUBJECT1, SUBJECT2, SUBJECT3 which refer to up to 3 subjects the indicator belongs to The descriptive metadata in ILOSTAT is coded and will appear as attributes associated to each observation value There are three types of metadata which are recognized by the name of the attributes: VALUE_STATUS is a flag associated to the observation value, and there will be only one per observation MET_<Note type> holds descriptive metadata and will probably be repeating in all the data points for the table defined by the main dimensions (COLLECTION, COUNTRY, FREQ, SURVEY and REPRESENTED_VARIABLE) plus TIME OVN_<Note type> holds a footnote It is attached to one observation and denotes certain exception for that particular data point Page 14