Integrating Statistical Documentation: A Feasibility Study

Transcription

1 INTERACTION WITH USERS SESSION D Integrating Statistical Documentation: A Feasibility Study Boris Lorenc, Heather Bergdahl, Klas Blomqvist Statistics Sweden v

2 Integrating Statistical Documentation: A Feasibility Study Boris Lorenc 1, Heather Bergdahl 2, Klas Blomqvist 3 Abstract. We report on a feasibility study aimed to investigate possibilities of integrating Statistics Sweden s formal templates for quality declaration and for methodological description of the production process into a single structure. In the study, we approached the subject from two perspectives. (1) A qualitative approach comprising of a survey of users and customer support personnel. (2) Preliminary formal considerations for integrating the documentation. Resulting from these considerations, we propose a single, structured set of metadata and quality data from which the content can be provided to the existing templates, as well as made available for other kinds of uses. It has a layered structure, which means that the users are freer than presently to select the information which they are interested in, thereby supporting its increased use. Key words: Beskrivningen av statistiken, BaS, SCBDOK, MetaPlus, SIMS, ESQRS, ESMS. 1. Introduction [Heading 3, Arial, Bold, 16 pt] In addition to producing statistics, producers of official statistics also produce its documentation. By doing so, they aim to put the statistics into its context (thereby promoting its understanding and use), indicate strengths and weaknesses of the statistics (thereby providing quality declaration for the produced statistics), and inform on the methodology used to produce the statistics for transparency and trust (thereby ensuring repeatability). For achieving these goals, the producers of official statistics rely on metadata, which also enables formal storage, sharing and reuse of the statistics. 1 Statistics Sweden, Senior Methodologist, Stockholm, Sweden, boris.lorenc@scb.se 2 Statistics Sweden, Quality Coordinator, Örebro, Sweden, heather.bergdahl@scb.se 3 Statistics Sweden, Senior Advisor/Enterprise Architect, Stockholm, Sweden, klas.blomqvist@scb.se

3 Templates for document types that meet these different goals have not always been developed in synchronicity with each other. As these templates tend to refer to similar (and sometimes same) information, better integration between them would be beneficial. Treating them as completely separate, independent implies that double, triple, etc work is carried out in applying the templates in statistics production, as well as a risk thereby introduced that there will be a lack of consistency between the published documents. As an example among others, Statistics Sweden uses a number of templates to document produced statistics, the stored data, and so on. One template for describing statistics, that includes a quality declaration, is Beskrivningen av statistiken (BaS) (Description of statistics). Another, with a stress on methodological description of the statistics production process and outcome, is SCBDOK (i.e. a template for documenting statistics, developed at Statistics Sweden). Further, there is a formal structure developed that supports storing metadata pertaining to statistical products, called MetaPlus. In addition to that, when providing the statistics to Eurostat, quality and metadata templates developed within the European Statistical System that is, the templates pertaining to ESQRS (the ESS Standard for Quality Reports Structure) and ESMS (the Euro-SDMX Metadata Structure) are used. However, on reflection, the information that these templates refer to does overlap. In practice, the same or at least similar text can be found in more than one document. This tends to create confusion as to where the user can find which information, an uncertainty that engulfs both the users and the personnel providing customer support. Additionally, when a part of the description is changed, this often needs to be implemented in more than one place, implying unnecessary work burden for documentation maintainers as well as a risk for inconsistency by missing to perform all the updates. Finally, providing similar information to Eurostat implies yet another level of duplication and yet another threat to information consistency. We report on a feasibility study aimed to investigate possibilities of integrating the aforementioned templates at Statistics Sweden into a single structure. The paper starts by presenting the methods used (Section 2), which included a qualitative survey of users and customer support personnel, as well as some further sources. Another method was a preliminary formal consideration for integrating the documentation, which relied on the Generic Statistical Business Process Model (GSBPM) but also took note of different user needs. Findings stemming from the former group of methods are reported thereafter (Section 3). Resulting from these findings, and the formal analysis, we propose a single, structured set of metadata and quality data from which the content can be provided to the existing templates, as well as made available for other uses (Section 4). The proposal is structured in layers, which means that the users are freer than presently to select the information which they are interested in, thereby supporting its increased use.

4 2. Method In pursuing its goals, the study took two main approaches: 1. A mostly qualitative approach designed to capture the external users and internal users/stakeholder views and requirements regarding documentation needs and practices. 2. A technical approach that started from cataloguing the content of the known documentation requirements in the templates, from which ideas emerged about main outlines of a future solution. 2.1 The Qualitative Approach External users needs We explored external users practices, needs and attitudes regarding documentation of statistics by means of semi-structured in-depth interviews with one participant each from these four Statistics Sweden s customer categories: Media; Industry and Trade (Businesses); Public Sector (incl. Government); and Teachers and Students (including Researchers). The interviews were conducted by a member of the project team based on an interview template which covered these main areas: the participant s experiences with Statistics Sweden s statistics and statistics from other statistics producers, and a comparison between these statistics; the participant s understanding of the quality of the statistics, including which reasoning the participant uses to assess the quality of statistics; the participant s comments and suggestions regarding the statistics documentation at Statistics Sweden; in-depth evaluation of parts of the program Quality improvement of quality declarations, which included requirements regarding uncertainty measures in certain areas. The interviews, lasting between minutes, have been recorded and transcribed. An analysis was first made of each of the interviews separately, using thematic analysis. These individual analyses were then related to each other in order to identify and characterise main emerging topics. Internal users and stakeholders needs To get a similar insight into internal users experiences, needs and views regarding the statistics documentation they come in contact with while performing their work tasks, we asked a number of them to fill in a questionnaire that we have prepared, containing five broad questions with an open-text response format. The questions related to: description of the participant s typical usage situations where they come into contact with the documentation,

5 their intended purpose of the documentation (including description of any external user needs); perceptions and experiences in terms of content and presentation, which needs are not covered so far regarding documentation and quality reports, further improvements. Five participants, working in the areas of customer support, archiving, documentation in a subject matter department, process management, and online access to micro-data for researcher at Statistics Sweden responded to the questionnaire. These responses were also analysed first each on its own using thematic analysis, and afterwards the individual results were related to each other to identify common topics and shared understanding. Although not qualitative, with this approach we included Analysis of visitor statistics on as this also reflects users experiences and behaviour. Benchmarking with other national statistical institutes work on this matter 2.2 The Technical Approach Among the activities in the pilot study focused on an overview of the frameworks and their relations, the following occurred: A workshop held to discuss a future coherent documentation approach and future vision Review of Eurostat s work to map the GSBPM against the Single Integrated Metadata Structure (SIMS) Mapping of Statistics Sweden s existing documentation templates (BaS, SCBDOK, MetaPlus) and Eurostat metadata standards (ESMS and ESQRS) against GSBPM (or, to be more correct, to the version of it used within Statistics Sweden, which is but for a few minor differences identical to the GSBPM (ref?)) Review of ISO ( Market, opinion and social research -- Vocabulary and service requirements ), according to which Statistics Sweden is certified since 2014, with respect to documentation requirements Review of SOU 2014: 45, Unique knowledge through research records and SOU 2012: 83, What is official statistics? A review of the official statistics system and Statistics Sweden, to take stock of what is proposed in the documentation area 3. Findings This section presents the main results of the findings of the qualitative component of the feasibility study.

6 3.1 User Documentation The feasibility study has found that there are clear indications stemming from both external and internal users that the differing purposes of BaS and SCBDOK, and thereby their partial similarity as well as the differences, have not been communicated clearly enough. External users After analysing the interviews with external users, the study found the following general themes: 1. Users experience difficulties in grasping the distinction between SCBDOK and BaS External users have difficulties in understanding the distinction between SCBDOK and BaS. Because of that, the interviews show, the users cannot easily access the information they need, they confuse these two types of documents, and are frustrated that they cannot find the information that they in previous searches have been successful in locating. 2. Users have different levels of needs for documentation Not all users need all of the information that a documentation template provides. Bearing this in mind, to reach the users, it would be beneficial to present the contents of the documentation in a way that takes into account different types of users with different kinds of interest. We suggest that this can be achieved by structuring the information into a hierarchy, where only the highest level is presented. Further levels of detail are chosen by the user depending on what information they need. We develop this somewhat more in Section 4; but, in a development stemming from the current study, we are further exploring ways of achieving this which we aim to report on in a separate paper. 3. Documentation needs can be grouped into those addressing the what and those addressing the how The need for documentation regarding statistics can roughly be grouped into the need for knowing the what of the statistics and knowing the how of the statistics. The first refers to the subject matter of the statistics (i.e. definition of young unemployed person, which countries are measured in the LFS for 2013, etc), the other to processes and methods (i.e. how was the sampling done, which nonresponse adjustment method has been used in the LFS, etc). Overall, users seem to be more interested in learning about what than about how. The latter is simply assumed to be done according to the high standards of official statistics production. Thus, interest for the how is mostly internal and inter-agency-wise. Exceptions to this seem to be the researchers as a user group, because they often have a broader understanding of statistical methods than the other categories of users. At the same time users confirmed that it is assuring to know that information on how actually exists, if needed. In fact, many users themselves regard the existence and accessibility of such documents as best practice for an NSI even though they are not frequently read.

7 4. Users stress importance of providing essential documentation adjacent to the published statistics Essential information documenting the statistics should be published close to the statistics that it relates to. For instance, any changes in the production and/or quality of statistics (in the definitions or in the methodologies that have been used) that may have affected the comparability of the statistics in a time series is requested by the users to be found in close spatial vicinity to the published statistics. While there are examples to the contrary, there are still examples of the practice in which documentation for the published statistics even when it might have gone through changes in the production which may risk comparability is to be found only way down in the structure of Statistics Sweden s website. This is why especially researchers are on the lookout for any changes in methodology or definitions as such changes may invalidate their research conclusions. This information needs to be extremely accessible. 5. Information about precision of published point estimates is not in the focus of users needs As already mentioned, documentation of how seems to attract less interest than documentation of what. Uncertainty information (i.e. confidence intervals, standard error, etc) seem to fall under how, perhaps because knowledge about how to interpret this information apparently requires some survey methodology background. Precision and accuracy are quality components of official statistics. The results show a limited use of such quality information. However, the results show that users are confident regarding quality of the statistics as declared, and further they trust that even more detailed quality information is available, would it be needed (without necessarily finding and reading this information). Instead, it is simply take as a given that Statistics Sweden provides statistics of high quality. 6. Direct contact is preferred to the use of the documentation Among the interviewed users, there was a high tendency to contact Statistics Sweden for explanations or help with interpretation, instead of using the documentation. We interpret this as an indication of shortcomings in the organization of the documentation. To some extent, the result may be a consequence of the fact that in this feasibility study we have not used a probability sample to select the persons that we interviewed. However, this conclusion from the interviews is consistent with other indications that we have found in the study about the deficiencies in the content and organization of documentation, so that its validity should still be considered to be high. Internal users/stakeholders An analysis of the responses by internal users/stakeholders identified the following general topics: 1. Access to the documentation needs to be simplified Facilitate for all users (external and internal) finding information that they need, which seems mostly to be about the content of the statistics (the what); clarify what can be found where.

8 2. BaS and SCBDOK should be joined into a single document Avoid duplication of work for SCBDOK and BaS by joining them into a single document template. Apply also to MetaPlus those parts of the joined template that are feasible. 3. Documenting longitudinal statistics should be improved Users who need documentation on longitudinal micro-data are struggling with Statistics Sweden s current way of documenting the statistics, which is geared towards cross-sectional statistics, produced in a single statistical program cycle. This result is related to the above recognized need to clearly highlight the changes since the last production run: because a longitudinal data series needs comparability over time in order to be useful, information is needed that clearly describes any changes, and the degree to which the comparability is threatened. 4. Documentation is important for archiving The importance of documentation for archiving is illustrated by noting that older archived data, which do not have their documentation, is difficult/impossible to reuse. 5. The web interface for Meta Plus should be revised The Web Interface for Meta Plus is perceived as cumbersome, both for internal users and external. This has two main reasons. The first it is difficult to find the information on the website, partly due to that the technical solution: the Web Interface to Meta Plus is physically separated from This means that there are no direct links from the statistics, the user has to click several times in order to access the information. And the second reason is that the Web Interface application itself has a complex structure. The result of this is that in order to fully understand the information, the user has to have quite a good insight into the statistical activities that they are looking for information about. Users do not find the tool to be intuitive. 3.2 Further insights Statistics from As a way to illustrate the use of the documentation, the feasibility study obtained data on visitor statistics on and the number of downloads of the two most widely used types of documentation, BaS and SCBDOK. The aim was to get an idea of the degree of use and its distribution in some relevant accounting categories. 1. Use of the "Documentation" tab of the website In a year, between 1 November 2013 and 31 October 2014, the number of external hits on the Documentation tab has been 77,085. Of those 5,353 went on to the Documentations and 6,344 went to the Classifications and Standards, indicating approximately same level of interest in these two types of documentation. It should be noted that the documentation can also be accessed through the product pages, that is, that the statistics are not complete.

9 Figure 1 Visitor statistics on Statistics Sweden s website, 1 November October, 2014 On 1,848 of the visits, the visitor has gone on to the Quality and production where SCBDOK and BaS can be found, and on 968 of the visits the visitor has gone on to Micro-data and classifications, which leads to Meta Plus and the classifications database on the Statistics Sweden s website. Based on the statistics, explicit interest in these visits for SCBDOK and BaS is relatively small, about 2% of those who clicked on the tab Documentation have gone on to pages containing these documents. Because the frequencies of subcategories of Documentation in Figure 1 sums up to approximately 37% (from 77,805 visits), one can assume that in the rest of the visits the visitors have stayed at Documentation and not gone to any of the pages further below in the hierarchy. One difficulty in drawing any conclusions about the needs of users based on visits statistics is that the pages sometimes have similar names (e.g. there is both Documentation and Documentations, both Microdata and classifications and Classifications and standards, and both Quality and production and Publications regarding statistical methods ), so that naming in the structure appears to have some potential for improvement. 2. Downloading BaS and SCBDOK Use of BaS and SCBDOK can also be studied directly through information about their downloading. For a period covering almost a year, from November 5, 2013 to September 30, 2014, the number of external downloads of these documents has been approximately 130,000. This is a significantly higher figure than in the previous figure (1,848), which may mean that the majority of external users do not come to the documents through the tab Documentation.

10 SCBDOK has been downloaded more frequently than BaS, approximately 90,000 or 70% of the downloads, while the rest (40,000 or 30% of the downloads) related to BaS. As of subject matter areas that the documents related to, the first five in the order of frequency were: the labor market, education and research, population, economic activities, and accommodation. Distribution by year of publication provides generally that the older a document, the fewer the downloads it has, but with the exception of documents that may be references to significant changes in production, which is then downloaded more often than their age would suggest. It is worth noting that SCBDOK has more downloads than Description of Statistics (BaS). This is unexpected because BaS is at Statistics Sweden considered to be more accessible than SCBDOK and is intended as a first step in getting to know more about the statistics. It could be because users do not understand what which document is relating to and covering, and click the link that is closest at hand, or it may be that the users who use the documentation know what they want and deliberately seek out the detailed information. The feasibility study had not been able to examine which of these two interpretations is more coherent with reality. The outside world 1. Statistics Denmark's customer survey An examination of the documentation of statistics that users want has been done by Statistics Denmark. In the report How do end users of statistics want the metadata? findings are presented that fairly well correspond to the picture of the needs of external users in Sweden: The users do not know about the structure of the documentation and do not understand it. They find the documentation on the comparability of statistics over time insufficient. They want more detailed information about the variables that can be found in the statistics database. 2. Benchmarking with Germany, discussion with Mirko Herzner, quality expert from Destatis Destatis has been doing quality declarations since 2005 for practically all statistics. Originally, there was a Word template for that, but since 2012 there is a database solution. When ESMS came in 2009, Destatis concluded that it has roughly the same content as in the internal templates for quality declarations. In 2011, changes to the structure of the quality reports were made in order to get them in a structure in line with the ESMS. Destatis considered using ESMS outright, but decided to use its own structure that can be mapped to the ESMS. 3. Examples from some Nordic statistical institutes websites The following two examples are from SSB and DST.

11 10?fane=om#content Comment: Short information becomes easily accessible to the user. Comment: Quality declaration can be expanded directly on the screen. 4. Integration This section provides the project team s evaluations and proposals on what a potential main study of a unified documentation system should reasonably aim at, and what the system should be able to achieve. The content is based on the areas identified in the previous section. The project team believes that after the implementation of the main study, an implementation of its results will lead to: increased and more effective use of the documentation by internal and external users, more effective production of quality reports and statistics documentation, and higher quality of the produced documentation. 4.1 Integrate templates for BaS and SCBDOK The feasibility study has found that there are clear indications from both external and internal users that the distinction between BaS and SCBDOK has not been communicated clearly enough. However, having two partially overlapping but separate document templates implies a double work for placing content in them, as well as a risk for divergence between them if the updates have not been done simultaneously. Therefore, a major boost to effectiveness of the documentation production and its quality can be made by integrating the two templates into one, which also is one of the main messages from internal users. As a byproduct, it is reasonable to expect that eliminating some of the hinders identified in the qualitative parts of this study will lead to a better understanding of the documentation system, easier navigation, and faster access to the required documentation. As the next step, after an integration between BaS and SCBDOK templates, an effort needs to be made to make a mapping with the European frameworks represented in SIMS, in order to prepare for future interoperability. 4.2 Improve accessibility of the documentation by introducing levels of the documentation content Further, the preliminary study found that the level at which the statistics documentation and quality is communicated has not been sufficiently adapted to the needs of users, based on their existing knowledge and the intended use, does not enable accessibility to the extent necessary, and therefore,

12 does not meet its goal to convey the information about produced statistics (including their quality) and the production process in an easily graspable and usable manner. Therefore, in terms of the documentation content and how to present it (its communication), a classification into levels and types of content (such as the what and the how, as presented in Section 3), adapted for typical usage situations, looks promising. Precise principles and forms of it should be developed in a future study. 4.3 To integrate the frameworks, use a database solution The technical inventory that has been made in the feasibility study shows that it is fully possible to integrate at least BaS and SCBDOK, with a good potential to add also ESMS and ESQRS (i.e. the SIMS). As above, when working on the integration of the templates, a new, hierarchical structure should also be developed, making possible for the users to choose the scope and focus of the documentation content that they in fact need. A future study should examine the conditions for storing this integrated, potentially hierarchical structure in a database, because the current solution with storage in Word documents makes it difficult to directly access and reuse a specific level and type of information. It has proven difficult to map the quality and production documentation to the statistical production process as it is described in Verksamhetsstödet (the Process Support System, a tool supporting design and conduct of statistical programs at Statistics Sweden which is based on the GSBPM). The efforts towards integrating them should continue, on the basis of process models that are produced within the business architecture, as well as relying on the activities conducted through Eurostat to relax SIMS to GSBPM. 4.4 Bring documentation closer to the statistics that it describes Documentation should be presented as close to the actual statistics as possible. Any changes in the production of statistics (referential definitions, used methods, or other aspect affecting comparability) should more clearly than now be indicated, adjacent as much as possible to the statistics in question. This, in order to support the use of the statistics by those users who are interested in the comparability over time. Principles and specific practical approaches for such a display of statistics together with the metadata (documentation) need to be investigated in a future study. 4.5 Generate documentation during the process A principle advocated for is that the documentation should be generated when the process that it refers to has been carried out (rather than generated as an afterthought, long after the process has been completed). A corollary of this is that some parts of documentation generation can be automatised. This track also reiterates the need to devote efforts to mapping the documentation frameworks

13 (BaS, SCBDOK, ESQRS, ESMS, SIMS) with Verksamhetsstödet, GSBPM and other process-related frameworks. 4.6 Verify solutions with users Any solution designed to implement the ideas above should be tested involving real users, in as much real situations as feasible.