CLAS Data Management

Transcription

1 CLAS Data Management G.Gavalian (ODU) CLAS Collaboration Meeting! Supported by the Data Mining Collaboration, S. Kuhn, L. Weinstein, M. Sargsian, and M. Strikman

2 CLAS Data Management Charge The Mandate: Jefferson Lab requires that valuable data generated in connection with the lab s research program be managed in a way that allows future and outside researchers to be able to work with the data, either to validate a result or to conduct additional studies on the same data. The scope of this mandate includes the preservation of the data, documentation of the data format, the preservation of associated data such as run conditions and calibration databases, and the preservation of software used to read and process the data. The proposed solution:! Extend the existing CLAS nuclear data mining software to serve as a data management system.

3 CLAS Data Management A compact and portable data format (gzipped ig5): ged File System (): Allows sorting run files according to run conditions, targets, beam type and energy. Can add meta-data and run properties (such as beam current and total charge) to the run collections. CLARA Distributed Environment (V. Gurjyan): Cloud computing environment for accessing the data from anywhere and for pre-processing data to decrease network traffic. Multi-node Dynamic Process Environments () for local network distributed computing and data synchronization. Data Mining Analysis Framework: Distributed experimental data framework, with search and discovery using. Analysis codes for specific experiments, including particle identification, momentum corrections and fiducial cuts.

4 Data Formats A custom data storage library (ig5) was developed for the data mining project to store the data of nuclear target experiments in more compact form: The data format is a column-wise mixed data storage algorithm mimicking CLAS data structures in BOS. The library provides ability for lossless compression (GZIP) of the stored data. Interface is developed for data access in C++ and Java. The EVIO container is used to store the compressed data buffers, which provides multiplatform portability. The new data format reduces the size of stored data by about 70%: Data Formats Data Formats: BOS - CLAS standard data format. EVIO(iG5) - new data format. EVIO (ig5)comp - new data format with compression BOS EVIO(IG5) EVIO(IG5) COMP

5 ged File System Developed a tagged file system to sort experimental data files on NAS disks. Meta-Data (tags) for each file are stored, including run conditions, target, beam type and energy. Developed algorithms to search through data using tags. User User User Client Programs CLARA PLATFORM ged File System (T) Metadata Storage NAS Data Storage

6 example Meta-data for each file on the tag file system has attributes: IP address for containing the physical file and IP address of Proxy platform that provides access to the. set associated with run conditions of the run, including beam type, target type beam energy and so on.. Property set describing attributes of each individual run file. Proxy: FILE Meta-Data Acces : Property electron FCUP= TAGS energy-4.7 Entries=145 MD5=2c5482d45 target-h2

7 CLARA Distributed Platform Data Mining project uses CLARA for distributed computing environment CLARA Improvements needed: CLARA-1.6 object model was changed to introduce mime-types, for unambiguous data exchange. CLARA-1.7 platform was running independent server on one computer, without platform discovery. CLARA-2.0 introduced Dynamic Processing Environments () which allows several platforms to be connected into one environment. CLARA-2.0 Mesh style design had limitations, using multi-computer setup behind the firewall was problematic. A new STAR-style design was introduced to provide access to data processing environments behind the firewall, enhancing the power of distributed computing. Significant development work invested in making CLARA useful for data mining. More development will be needed to make CLARA adequate for this larger mission.

8 CLARA Distributed Platform (STAR) In CLARA 2.0 MESH was introduced to have several computers connected to main platform and be discoverable, each of them with their own. In CLARA 2.0 STAR-type environment was implemented, where all communication with s goes through main Proxy platform. Each has its own storage and runs a, the data processing speed is only limited to internal servers readout speed. One computer (main Proxy platform) has to have open communication with outside world. CLARA 2.0 (MESH) CLARA 2.0 (STAR) USER CLARA PLATFORM USER CLARA PLATFORM

9 CLARA Distributed Cloud User can connect to multiple platforms (running on different University clusters): data query, discovery by experimental meta-data data analysis (basic and experiment specific code) and download. USER USER CLARA PLATFORM USER USER CLARA PLATFORM USER CLARA PLATFORM

10 Data Mining Software Data Mining Software: based on CLARA for cloud data processing, written in Java. data tagging and search interface based on. select files (beam energy, target etc..) using. experiment specific analysis codes, using review committee approved particle identification, momentum corrections and fiducial cuts: select cuts and corrections (PID method, momentum corrections ) select final state to analyze (e-,pi+,p..) select data to download : full data 4-vectors or 4-vectors with PID info Converted Files on ODU Server e2b eg2 e6 g11

11 Data Access Interface GUI allows user to search through tags and experiment list on the server. Listing provides run numbers, number of files for run and integrated charge for run.

12 Data Analysis Interface The data can be distributed on different servers, user can chose the server to use. User selects the analysis code for the data to pass through before downloading (particle id, momentum corrections..) User selects the final state for exclusive or inclusive reactions: 11:2212:22:211 11:2212:X+:1-:Xn based on either SEB or run group specific PID

13 Data Cuts and Corrections (eg2a) Custom analysis codes for eg2a run group: particle identification vertex corrections momentum corrections fiducial cuts

14 Nuclear Data Mining Analyses pp SRC correlations in nuclei, Or Hen, under CLAS review backward Deltas in deuterium, Chris Wooten (ODU senior thesis) EMC and SRC, Barak Schmookler (MIT) d(e,e p) and (e,e n), M. Braverman (TAU) SRC Proton Transparency: Or Hen (TelAviv), PLB772(2013)63 Mean Field Proton Transparency: Or Hen (TelAviv), under CLAS review Double Spin asymmetry in d(e,e p)n, Mike Mayer (ODU), soon to be under CLAS review Many proton knock out from nuclei to explore the potential to reach r- process path below 208Pb, Dan Watts (Edinburgh) 3He(e,e n)/(e,e p), Uttar Pudaisini (ODU) p-and n-single nucleon momentum distributions in asymmetric nuclei, Tel Aviv Hadronization in semi-inclusive DIS, H. Hakobyan (UTM)

15 World Wide Data Analysis The Data on ODU servers is being actively analyzed by users from Scotland, Germany, Chile, India, Israel, MIT, U, ODU and others. ODU

16 Data Mining Summary This software puts the developed wisdom of the run group in one easyto-access location. no need to wander the halls looking for experts. no need to cut and paste codes (this happens a lot) allows reanalysis even after the experts have moved away allows students not associated with CLAS to do analysis. Efficient data format, and CLARA. Efficient data compression saves valuable disk space. tagged file system allows efficient sorting and discovery for the data. cloud distributed CLARA system allows access to the data from anywhere. new CLARA design allows multiple data sources (located at different sites) to be analyzed in parallel. Data from nuclear and photon experiments (e2, eg2, e6, g11 and g12) is converted to EVIO(iG5) format and stored on ODU servers. Run group specific cuts and corrections are implemented for eg2, e2, g11, e6.

17 Summary Developed CLARA framework further to allow universities to host data storage, analysis and distribution centers. Developed efficient data format to reduce the data size ~70%. Developed ged File System for cataloging and selecting data distributed over the large number of platforms. Created Java interface for easy deployment of programs over a variety of platforms. Jython interface allows easy scripting for deploying jobs and running analysis on local computer. Data Mining framework is widely used for analysis by CLAS and non-clas physicists. Data Mining project framework can be extended to serve as the data management system for CLAS-12, and possibly for other halls.

18 Backup BACKUP SLIDES

19 Data Analysis Custom analysis codes for each run group perform particle identifications and corrections Analysis committee Approved cuts and corrections implemented. Can implement multiple versions of cuts and corrections example: E2B Vertex Corrections

20 CLARA Distributed Platform (MESH) In CLARA 2.0 was introduced to have several computers connected to main platform and be discoverable, each of them with their own. Communication and data exchange will be directly with, which requires for computer to have access from outside of the network. CLARA 2.0 (MESH) USER CLARA PLATFORM

21 SRC Analysis (B. Schmookler MIT) Short Range Correlations Analysis of eg2a data (B. Schmookler) EMC ratio a2 plateau Preliminary

22 Data Compression Test of the data compression for CLAS-6 data. SEB BOS - standard CLAS library. EVIO(IG5) - IG5 Buffered Data Stream wrapped inside EVIO format. EVIO(IG5) COMP - IG5 Compressed Buffered Data StreamWrapped inside EVIO byte buffer. EVNT BOS SEB (P) EVIO EVIO(IG5) EVNT (P) BOS EVIO(IG5) EVIO(IG5) COMP SEB - particle identification hit banks (HEAD,EVNT,DCPB, ECPB...) EVNT- only particle 4-vector banks with header bank (HEAD,EVNT)

23 CLAS-12 Plans Developed framework can be extended and used for CLAS-12 production data. Some fraction of the experimental data can be transferred to CLARA cloud for data monitoring, calibration and physics analysis Final reconstructed data (after calibration) can be stored at participating universities for physics analysis and will be accessible through CLARA. The data correction codes (momentum, vertex corrections and fiducial cuts) can be easily shared. The multitasking provided by CLARA cloud will significantly speed up data analysis for users.

24 CLARA Distributed Platform In CLARA 1.7 the platform is running the computing environment with, the NAS disks are mounter to the system through switch. The data reading is limited to the switch speed. The number of processed (threads) are limited to one host computer. CLARA 1.7 USER CLARA PLATFORM

25 Data Mining Summary Data Access and Analysis: data from nuclear target experiments (and g11 and g12) is converted to EVIO(iG5) format and stored on ODU servers. data is discoverable by experiment name, target type and beam type. run group specific cuts and corrections are implemented for eg2, e2, g11, e6. Simulations: meson production and decay framework is implemented for light meson production and decay (radiative, hadronic, dalitz). Fast-Monte-Carlo is developed for photo-production experiments using single particle efficiency maps derived from experimental data.