Database Driven Processing and Slow Control Monitoring in EDELWEISS

Transcription

1 Database Driven Processing and Slow Control Monitoring in EDELWEISS CRESST / EDELWEISS / EURECA Software Workshop München, 21. Mai 2014 Lukas Hehn, KIT Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft

2 Talk Overview Database Driven Processing about CouchDB EDELWEISS databases and where they are hosted sample CouchDB document used for processing database driven processing scheme and Slow Control Monitoring in EDELWEISS available slow control data examples for visualization with CouchApps summary

3 Database: About CouchDB CouchDB is an open source document-oriented database. Store your data with JSON documents. Access your documents with your web browser, via HTTP. Query, combine, and transform your documents with JavaScript. Accessing CouchDB: administration and document modification via Futon web interface access with all programs supporting HTTP calls (i.e. with CURL) output of JavaScript Object Notation (JSON) objects documents are unstructured: data extracted with MapReduce views libraries existing for most languages mostly used by us: python scripts using the library couchdbkit EDELWEISS uses commercial hoster Cloudant Cloudant provides distributed databases over many servers that appear as one instance

4 CouchDB Usage in EDELWEISS data transfer & modification DAQ LSM (scripts or DAQ software directly) replication Centre de Calcul Lyon (IN2P3) information about files KIT server one-time replication analysts user computer Cloudant CouchDB instances

5 Main Data Base for Processing: datadb currently: 1.5 GB in >20K documents (1 document per 1h partition file existing for several detectors read out by the same Mac + some metafiles for a ~24h run) CouchDB document: meta information for these files: SAMBA-DAQ header location of the LSM processing information alpha counter data pulse templates muon veto HV cryogenics data muon veto position

6 Sample Document datadb (Beginning Only) full revision control number (either integer or float) string boolean (true/false value) array set of key/value pairs (also possible: attach files, i.e. png or ROOT TF1 with a fit result) file location on Mac computers at LSM

7 Database Driven Processing From LSM (Modane Underground Laboratory): Daemon processes regularly run KData scripts SAMBA-DAQ to CouchDB: create new documents with SAMBA header hourly Local Processes: Proc0: copies RAW files from LSM-Mac to Computing Center in Lyon metaproc0: copies meta files Proc1: creates ROOT files from RAW and copies them to Lyon SPS to HPSS: archives data on long term storage in Lyon weekly Proc2, etc...

8 Database Driven Processing 1 CouchDB document per file with status keys python routine on a LSM-Mac asks database with different MapReduce views which files need processing triggering by scheduled cron job or CouchDB _changes feed routine passes file location to processing code for each file a process can be easily restarted by deleting the specific key (i.e. proc0 ) in the corresponding database document

9 End of the datadb Sample Document backup of RAW data + metafiles on HPSS long term storage: i.e. run305 (oct13 jan14): 4784 files/282 tar files/4,315 GiB copy of RAW files from LSM to Computing Center Lyon datadb documents track the processing of each file creation of root-files creation of amplitude files with pulse trace filtering

10

11 Slow Control Monitoring in EDELWEISS Available monitoring: cryostat & bolometers radon monitor: alpha spectrometer measuring air volume around cryostat muon veto system: high voltage supply for scintillator modules PMTs position of the 2 movable shield parts (properly closed?) DAQ system: multiple Apple Macs Data from all these monitoring systems written to individual data bases Easy online access from everywhere via CouchApps: small websites that display data base content graphically

12

13 define time range and value to plot or zoom into plot with mouse

14

15

16

17 Summary Existing EDELWEISS infrastructure: data processing in EDELWEISS relies on CouchDB data base that tracks the location and (processing) status of every file revision control of CouchDB allows to inspect the history of every file monitoring systems (alpha counter for radon level, active muon veto system, cryogenic systems) data is transferred directly to the data base CouchApps allow to easily visualize data Currently worked on: data bases with calibration coefficients, data quality parameters, different data cuts apply same processing to analysis results (pulse filtering,...)