Periodic Report on WP3 (1/1 31/12/2008) NKUA Athens, February 2009
Document information Project Project acronym: NMDB Project full title: Real-time database for high resolution Neutron Monitor measurements Grant agreement no: 213007 Funding scheme: Combination of Collaborative Projects & Coordination and Support Actions Project start date: 01.01.2008 Project duration: 24 months Call topic: INFRA-2007-1.2.1 Scientific Digital Repositories Project web-site: www.nmdb.eu Document Periodic Report: NKUA Period Covered: 01.01 31.12.2008 Actual Submission Date: February 2009 Editors: H. Mavromichalaki Authors: H. Mavromichalaki, A. Papaioannou, C. Sarlanis Draft/Final : Final No of pages (including cover) 9 Keywords: Periodic Report on WP3 2
Contents..3 1. Summary...4 2. Project objectives....4 3. Work progress and achievements......4 4. Explanation of the use of the resources /Deliverables and milestones Tables 7 5. Project managements.... 8 3
1. Summary The progress of the package WP3 during the year 2008 has been developed according the schedule of the NMDB Project and the Kick-off meeting (Paris 2008). An online questionnaire, which included all information about Neutron Monitors of the participated groups, was created and all stations managed to send data to the NMDB database. 2. Project objectives According to the schedule program during the time period 01.01.08 31.12.08, the objective of WP3 was to issue a standard set of rules for data acquisition in order to achieve both standard format and resolution (1 minute every 1 minute). A pre-request was to technically prepare all participating stations in order to feed their data into the central database (NMDB). WP3 was committed to take all specified inputs and help adjusting the existing infrastructure in order to be compatible with the NMDB database. 3. Work Progress and achievements Deliverable 3.1 report: Archive creation 3.1.1 Collection of all information regarding the status of the NMs 3.1.2 Check the specific needs of every station 3.1.3 Control of necessary updates 3.1.4 Create a compatibility report Start: 01 March 2008 / End: 30 June 2008 In the frame of this deliverable an online questionnaire was created from NKUA, including general information as well as analytical information for every participating station regarding the station s electronics, acquisition system and local server 4
(http://cosray.phys.uoa.gr/nmdb%20athens%20website/nmdb%20athens.html). The result was to have a clear view of the work that should be performed for every single participating station. All information regarding the participating NM stations is collected in an archive. All groups involved in this effort were setting specifications for a system compatible for a central database. The aim of these tasks was focused into the ability of the participating NM stations to provide real-time neutron monitor data with 1- min resolution, updated every 1-min. Summarizing according to the above questionnaire, we can say that at the initiation of the NMDB project 8 NM stations (OULU, JUNG1, JUNG2, AATB, MOSC, ESOI, ATHN and ROME) provided real-time data of 1 min resolution updated every 1 min. 2 NM stations (KIEL, BRBG) provided real-time data of 1min (or higher resolution), with a slower updating rate. 3 NM stations (ArNM, NANM, APTY) provided real-time data of high resolution (1 sec) with an update rate of 1 min. 1 NM station provided non real-time data of 1 min resolution, updated every 1 min (LOMSTIT). 1 NM station (BERN MNM) provided real-time data of 5 min resolution updated every 5 min. 2 NM stations (TERA, KERG) provided real-time data of 1 min updated every 1 day. 7 NM stations (ROME, MOSC, KIEL, AATB, ArNM, NANM, ATHN) provided data corrected in real-time. Therefore, according to the presentation of WP3 working plan at the Paris Kick-off meeting (January 17-19, 2008), software should be implemented in order to make it possible for: 2 NM stations (KIEL, BRBG) to upgrade their updating rate 3 NM stations (ArNM, NANM, APTY) to provide real-time data of 1 min resolution. With the completion of the station s upgrade via software, at least 13 NM stations were in place to feed the database with real-time data of 1 min resolution, updated every 1 min. This was the first accomplishment of WP3.. A compatibility report for every station regarding NMDB s needs was created on 5 July, 2008 with responsibility of NKUA.. Deliverable 3.2 report: Support existing infrastructure/ Trial period Start: 01 June 2008 / End: 30 November 2008 Every indicated participating station had to improve either their registration system and/or the ability to provide 1-minute data every 1 min. NKUA was responsible to 5
support any asking participant. Every station provided extensive reports on the upgrading procedure (possible problems, system runs, efficiency issues, final results, system performance etc). The output of the trial period was existing infrastructure to be upgraded and ready to feed data to the database (1min/1min) and final documentation regarding the efforts of the upgrade was produced on 27.01.2009 Any remaining problem with the quality of the data in the NMDB database is continuously examined. A brief summary of the procedures that have been done is given below: Almaty: Implementation of a script for forming 1-minute data every 1-min and and 1- hour as well as sending these data to NMDB. This script was developed for Windows XP in IZMIRAN. Aragats: Development of a Database Bridge (DB) that deploys data from Data Visualization Interactive Network (DVIN) node MySQL server to NMDB MySQL server for retrieving data from Aragats and Nor-Amberd neutron monitors to NMDB. Athens: Implementation of three scripts for data manipulation and one for system reliability. a) The first one gets the data from binary files and stores them in the local MySQL database. b) The second one opens hourly text files that have been created in local directory from the Super Editor Software that is on IZMIRAN format and stores them in our Local MySQL Database and c) The third one gets data from the local MySQL database sending them to the Central Database in NMDB. The supervisor script checks if everything is executed well including the system kernel that tries to overcome possible problems. IZMIRAN: Development of the software for the automatic updating of the NMDB taking into account the specific properties of each station. One of them is developed for operating under OS Windows in Delphi-7 and another one under OS Linux+Perl. The algorithms for each part are: a) Updating database for OS Windows+Delphi. b) Updating database for OS Linux+Perl+MySQL c) Reading data from NMDB for OS Windows+Delphi d) Reading data from NMDB for OS Linux+Perl. All this software including the initial texts is accessible by the address: ftp://cr0.izmiran.rssi.ru/programsupport_nmdb-izmiran. KIEL: Development of scripts for submission of the Kiel NM data to NMDB. These data is taken from the FTP server and is converted into CSV files suitable for sending to NMDB. The CSV files are sent to NMDB with LOAD DATA LOCAL INFILE. The 6
python scripts convert and send the data to NMDB. The scripts are described at:http://www.nmdb.eu/?q=node/65 KOSICE: Development of software a) Communication level programs, specifically Daemon SAPI in standard C language. b) Application level program in php language, including weather data for local MySQL and NMDB. OULU: On the registration system, the acquisition program stores data in a text file, one file per day. A perl script uploads these file(s) every minute onto server via ssh, and stores these in server's "spool" folder for newly created data files. On the server, another program checks for these files each minute, fixes all data problems (such as gaps), fixes these if necessary, and stores data both to the local MySQL database and to NMDB. Also, one-hour averages are computed for "completed" hours and also stored in respective tables in real time. Finally, text file is moved from "spool" to "archive" folder to be kept there. We use C++ and Perl software and MySQL++ library to handle connections to databases. Data are in NMDB with 2 minute delay. OBSPARIS: Development of automatically check for new available data on the mail server, adapt the format to CSV, and send the newly created file to the database. Another script checks some parameters once a day to create the files needed to feed the revised table. An IDL program is running in order to send the revised data to NMDB. UBERN: Development of Python scripts to transfer the data files of the neutron monitors at Jungfraujoch (JUNG and JUNG1) to Bern and to convert the timestamp from the end of the interval to the beginning of the interval and the given time from Mideuropean Winter Time to Universal time. Afterwards the script writes the data into a MySQL database in Bern and finally the tables JUNG_ori and JUNG1_ori of the database NMDB in Kiel are synchronized with the corresponding content of the Bern database. UNIROME3: Added of linux server with FTP server to receive data from the acquisition unit and developed a script for data sending to NMDB server. TAU: New software for data acquisition from NM of ESOI-TAU was created on local computer-registrator (worked under DOS support) and was installed on the ESOI-TAU Observatory. The aim of the program to transfer the automatically prepared matrix of 1-min observational data into buffer-computer accumulator (worked under Windows XP) for the next data reduce and transfer to local data server. Special complex of software under MATLAB was created on the computeraccumulator for reducing the primary data to a set of files in special format for transfer through Internet (remote desktop mode) into local server of ESOI-NMDB situated in Qazrin office of the Israel Cosmic Ray and Space Weather Center. 7
Special software under MATLAB for data transfer to global NMDB server is now in preparation. 4. Explanation of the use of the resources/ Deliverables and milestones I: Deliverables Del no. Deliverable name 3.1 Archive creation/ Specifications 3.2 Support Existing infrastructure/ Trial period W P no Lead beneficiary Nature Disseminatio n level Delivery date from Annex I (proj.month) Delivered Yes/No Actual/ Forecast delivery date 3 NKUA R PU 6 YES 05.07.08 3 NKUA O PU 12 YES 27.01.09 8
Table II: Milestones Milestone No Milestone name 1 Upgrade needs/specifications 2 Metadata adapted to NMDB WP nos. Lead beneficiary Delivery date from Annex I Actual delivery date Comments WP3 BIRA (?) Month 3 Month 4 Online report of current NM stations situation WP3 IZMIRAN Month 4 Month 6 Report on input data (del. 3.1) & extensive discussions 5. Project Management NKUA supervised all items of WP3 and performed deliverables 3.1 & 3.2. 9