IMPLEMENTING STATISTICAL DOMAIN DATABASES IN POLAND. OPPORTUNITIES AND THREATS. Central Statistical Office in Poland

Transcription

1 IMPLEMENTING STATISTICAL DOMAIN DATABASES IN POLAND. OPPORTUNITIES AND THREATS. Central Statistical Office in Poland

2 Agenda 2 Background Current state The goal of the SDD Architecture Technologies Data structure System architecture Conclusions Threats and challenges

3 The aim of the presentation 3 To present the model of system architecture and database structure of Statistical Domain Databases (SDD) that was implemented by CSO in Poland To list problems that needs to be concerned before presenting data in pivot tables through world wide web To show the possibility of using OLAP tools to provide data from official statistics to external users

4 Current state 4 Data are accessible via website but the views are very limited Data presented in public databases are mostly the same as in statistical yearbook Users cannot change table structures by themselves It was necessary to identify new ways to present statistical data

5 General outline 5 The goal was to implement a system that can present official statistical data in pivot tables The data layer was implemented by regional statistical offices (based on their specialization) The presenting layer was developed by Statistical Computing Centre The data model is the result of cooperation between different local authorities

6 The goal of the system 6 Present data from different statistical domains to external users Data should be presented in pivot tables that users would adjust them to their own preferences It covers 27 different statistical domains from education to labour market

7 Technologies 7 For pilot implementation the following systems were used: Microsoft SQL Server 2008 R2 with Analysis Services Microsoft SQL Server Business Intelligence Development Studio (ASP.NET) Microsoft IIS Server on Windows Server 2008 (Virtual Machine)

8 Components Key system components 8 Staging area each statistical domain has its own staging area managed by proper statistical office raw data Data warehouse different repositories with microdata OLAP repository only macrodata Database portal website Metadata repository terms, classifications and variables

9 General outline of system architecture 9 deployment DBD Regional Statistical Offices Statistical Computing Centre «device» Staging area - MS SQL Serv er 2008 MS SQL Server Data Export Wizard «device» Data warehouse and OLAP - MS SQL Serv er 2008 Enterprise with Analysis Serv ices ODBC / OLE DB Provider «device» Website - Microsoft IIS Serv er TCP/IP «device» Client - website browser «device» Metadata serv er - MS SQL Serv er

10 Data gathering 10 Moving data to staging area Statistical Offices Transferring data to central data warehouse Statistical Computing Centre Aggregate data in OLAP Cubes Statistical Computing Centre Data verification Statistical Offices

11 Data structure 11 The general data structure of SDD is known as a star schema in data warehousing Due to the necessity of quick response to users queries data structure has been developed individually for each domain Usually there are up to 8 dimension tables per cube The common dimension tables are: 1. Time 2. Region 3. Variable 4. Unit of measure

12 Logical model of data structure 12 Time dim. Variable dim. Unit of measure dim. Region dim. Fact table Custom dim. 5 Custom dim. 6 Custom dim. 7 Custom dim. 8

13 13 Typical data structure (domain: Education)

14 Data exploration via WWW DOMAINS 2. DIMENSIONS 3. MEASURES 4. FILTERS

15 Rules of data gathering 15 Metainformation contains description of variables that are usually combined from statistical term with its profile All data are stored in multidimensional cubes (MOLAP) Therefore all the data need to be aggregate before putting it into database

16 Threats and challenges 16 In some cases the possible data profiles can exceed the limit of the data included in the OLAP cube All domains cannot be put in one common set of tables Integrate SDD with Information System of Official Statistics as a part of its complexity (Public Data Warehouse)

17 Conclusions 17 Full automation of data gathering process is not possible Verification is the process in which working staff should be particularly engaged Almost every statistical domain has its own rules To assure a reliable and high performance system it is necessary to treat each statistical domain individually Using variables instead of terms can mitigate various problems with methodological changes