Building a multi-pb Object Database for the LHC

Transcription

1 Buiding a muti-pb Database for the LHC Buiding a muti-pb Object Database for the LHC The RD45 Project Jamie Shiers Appication Software and Databases Group Information Technoogy Division

2 Buiding a muti-pb Database for the LHC Overview The Large Hadron Coider (LHC) The RD45 Project Choices: technoogies and standards Buiding a muti-pb object store Product choices Scaabiity and performance Interface to Mass Storage Short term pans Risk Factors Summary and Concusions Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 2

3 Buiding a muti-pb Database for the LHC The Large Hadron Coider New faciity currenty under construction at Schedued to enter operation in 2005 wi operate for years Wi re-use existing infrastructure incuding 27km ong tunne currenty housing LEP Some 4 arge experiments panned for the LHC ATLAS, CMS, ALICE, LHC-B Wi generate data voumes of 5PB/year at data rates of 100MB/second - 1.5GB/second Gives rise to a tota data voume of some 100PB MTimescae & Lifetime wi be equay difficut probems Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 3

4 Buiding a muti-pb Database for the LHC Data Voumes & Rates Data voumes and rates for 1 LHC experiment Tota voume per year ~5PB Tota LHC data voume ~2 x 1bps/ifetime of Universe Time interva Data voume Equivaent 1 second 100 MB 1 inear metre of books 1 minute 6 GB 1 Exabyte hour 360 GB trees-worth of paper 1 day 8.6 TB US Library of Congress 1 week 60 TB The NCAR MSS today 1 month 260 TB The ECMWF MSS in year (100 days) 1 PB 3 years EOS data (2001) 1 mienium (a) 5 EB A words ever spoken by humans Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 4

5 Buiding a muti-pb Database for the LHC The RD45 Project Started in 1995 to address the issues of object persistency for the LHC experiments Strong focus on standards Assumed that LHC woud use C++ How does this compare with previous, Fortran-based soutions? How can we adapt to changes, e.g. Java? Have identified a soution buit on ODBMS + MSS Project is sti oriented towards the LHC Soutions adopted by many other experiments COMPASS (): 400TB/year, 1999 on BaBar (SLAC): 200TB/year, Phenix, STAR (BNL): 300TB/year each Currenty setting up production services production data ~100GB, test data ~1TB Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 5

6 Buiding a muti-pb Database for the LHC Why Use Standards? Widey-used stds are an essentia part of our strategy Cannot predict the changes between now and 2020 Can be sure that major changes wi happen Stick with the rest of the pack Do not use proprietary interfaces/soutions Even (especiay) if you own the code, migration is a sow, painfu process, typicay measured in years Which standards? è C++, incuding STL-based coections Object Database Management Group (ODMG) ƒ IEEE MSS(?) Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 6

7 Buiding a muti-pb Database for the LHC Persistent Object Manager: Choices Strong feeing (before project started) that commercia soutions woud not scae Investigated many potentia soutions anguage extensions (E, O++) ight-weight persistent object managers OMG persistent object service fu-bown Object Databases [ see Catte s book Object Data Management ] Strong preference for a uniform soution or at east interface Candidates: OMG POS, ODMG Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 7

8 Buiding a muti-pb Database for the LHC The ODMG The Object Database Management Group is a consortium of object-oriented database management system (ODBMS) vendors and interested parties working on standards to aow portabiity of customer software across ODBMS products The programmer shoud perceive the binding as a singe anguage for expressing both programming and database operations, not two anguages with arbitrary boundaries between them. Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 8

9 Buiding a muti-pb Database for the LHC ODBMS vs RDBMS Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 9

10 Buiding a muti-pb Database for the LHC ODMG vs Embedded SQL Init(); // initiaise the db session startupdate(); // start an update transaction // create a new database (fie) HepDatabaseRef mydb = db("mydatabase"); // create a new container in this database HepContainerRef cont = container("mycontainer"); if (cont == 0 ) fata("coud not find or create MyDatabase"); for (short i=0; i<1000; i++) { // create a new event in my container HepRef(HepEvent) event = new(cont) HepEvent; if (event == 0) fata("coud not create a new event"); } EXEC SQL WHENEVER SQLWARNING GOTO 1100 EXEC SQL LOCK TABLE FATMEN IN SHARE MODE EXEC SQL SELECT * INTO :DBASE, :EXPER FROM FATMEN 2 WHERE DATABASE =:DBASE AND EXPERIMENT =:EXPER STMT1 = 'LOCK TABLE GNAMES_' // EXPER(1:LEXP) // ', FILES_' // 1 EXPER(1:LEXP) // ', FXV_' // EXPER(1:LEXP) // 2 ',VOLUMES_' // EXPER(1:LEXP) // ' IN EXCLUSIVE MODE' CMD = 'STMT1' EXEC SQL EXECUTE IMMEDIATE :STMT1 Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 10

11 Buiding a muti-pb Database for the LHC L There is no best ODBMS ODBMS Seection What are the requirements for your appication? What are the priorities? What compromises can you accept? See Barry s Object Database Handbook Key requirements for Scaabiity, scaabiity, scaabiity Performance Support for heterogeneous environments anguages, compiers, h/w etc. J Some important simpifying factors... Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 11

12 Buiding a muti-pb Database for the LHC Features of HEP Data J Data is essentiay read-ony J Sma number of concurrent users (1-100) J Very ow transaction rate J Can accept data oss(!) MData voumes and rates (up to 100PB, 1.5GB/s) MFuy distributed MProject Lifetime (~25 years) Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 12

13 Buiding a muti-pb Database for the LHC LHC Lifetime Schedued to enter operation in 2005 Wi run unti What wi happen in the next 25 years? What has happened in the ast 25 years? At : Introduction of computer terminas and phase-out of cards Instaation of genera-purpose networks Introduction of interactive services The death of the ine-printer and pyjama paper Migration from mainframes to workstations to PCs Introduction of OO and move away from Fortran Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 13

14 Buiding a muti-pb Database for the LHC Can we use a commercia ODBMS? Conventiona Wisdom Databases does not scae to PB range Besides, their too sow HEP requirements are just different We can t afford them anyway And even if we coud, they won t be around in 2005 RD45 resuts ODBMS products with architectures that scae to PB range exist today (O)DBMS products se on performance HEP requirements are different from the norm, but in some cases simper! (WORM, ow transaction rate etc.) We can t afford to buid them industry standards are more ikey to be around than HEP ones... Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 14

15 Buiding a muti-pb Database for the LHC Buiding a muti-pb Database Ceary cannot buid a singe DB of 100PB However, DBs of 1-10GB in production today Coud we buid a system out of mutipe databases? What are the reaistic imits on database size? Fiesystem imits (64-bit, no probem) Disk/tape imits (50-100GB possibe today) Transfer time: seconds Effectivey imits maximum fiesize to around 10GB today L Need 1M databases to reach 10PB but this is the same number of fies that we are deaing with today... Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 15

16 Buiding a muti-pb Database for the LHC Architecture of Objectivity/DB ~ODMG compiant; C++, Java, Smatak bindings Permits up to 2 16 physica databases (i.e. fies) in a singe, ogica database (federation) tested up to 25GB/fie (database) tested up to imit of 2 16 (sma) databases Fies may be distributed across database servers in LAN/WAN System/user data may be repicated avoid singe points of faiure; bottenecks J Provides easiy sufficient scaabiity to reach PB region L Requires arge (100GB) fies for 100PB federations L Cannot assume that a data wi be on disk... Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 16

17 Buiding a muti-pb Database for the LHC Muti-PB ODBMSs To extend the ODBMS to 100PB or more, require: è An interface to a suitaby scaabe mass storage system Architectura changes permitting muti-pb federations without requiring very arge fies required regardess of capabiities of MSS cannot assume that this wi be instaed on a systems, e.g. ap-tops An interface has been designed and tested between our two product choices: è Objectivity/DB (ODMG-compiant ODBMS) HPSS Interface deveoped by coaboration of Objectivity and SLAC Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 17

18 Buiding a muti-pb Database for the LHC Architecture of Objectivity/DB Federation: databases can be stored on different nodes in LAN/WAN Database: currenty maps to a singe fie [ containers, pages, sots, objects ] Each object has a unique 64bit Object Identifier database, container, page, sot database names stored in a cataogue which can be repicated user does not have to know where objects reside Objectivity cient: object-aware Objectivity server: deas with fies, pages etc. Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 18

19 Buiding a muti-pb Database for the LHC Interfacing Objectivity/DB to HPSS Objectivity/DB server uses a sma number of I/O cas open(), cose(), read(), write(), etc. For each of these, an HPSS equivaent exists hpss_open(), hpss_cose(), hpss_read(), hpss_write(), etc. Essentiay just re-ink Objectivity/DB server against HPSS cient API Such a proof-of-concept prototype was deveoped & announced at SC 97 Currenty being tested at Catech,, SLAC Expect a product by end-1998 numerous issues to be resoved, incuding performance questions reated to use of simpe HPSS API Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 19

20 Buiding a muti-pb Database for the LHC Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 20

21 Buiding a muti-pb Database for the LHC HPSS Performance Measurements HPSS "simpe API" performance time/ca[s] Data Rate[MB/s] Bock Size [KB] ODBMS page size Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 21

22 Buiding a muti-pb Database for the LHC Objy-HPSS Interface Aternatives: Via the HPSS NFS interface, Using a future HPSS interface to DFS or DMIG, Using the "simpe API" (the current interface), Using the "advanced API" (permits mutipe bocks to be requested), Using a DB-fauting mechanism ("staging"). For 98+, pan to use staging mechanism DBs wi be copied out of HPSS into Unix fiesystems at open Can use existing hooks into Objectivity/DB to force stage of DB at open time Future pans incude passing of hints from cient to server, possibiity of oad-baancing across mutipe servers etc. Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 22

23 Buiding a muti-pb Database for the LHC Accessing A Database OO_FD_BOOT environment variabe points to boot-fie User initiaises DB Can now access any object in any database in the federation Obviousy, just iterating through the entire federation is not very convenient Can scan databases, containers, coections, ook-up by name etc. Contains basic DB parameters ocation of ock-server journa-fies cataog & schema int status = ooinit(); DB host & fiename determined from Object ID (OID) If fie is not on disk, wi be automaticay restored by HPSS Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 23

24 Buiding a muti-pb Database for the LHC Data Organisation Physica Databases, Containers, Pages Convenient for data that wi be accessed together Different parts of the same ogica structure can have different physica custering e.g. infrequenty accessed data is stored separatey from hot data custering interesting parts of interesting events Logica e.g. a events with certain characteristics may span mutipe physica containers may be sparse may overap / be intereaved with other ogica coections a events with 2 eectron candidates a events with 2 eectrons & 2 µ Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 24

25 Buiding a muti-pb Database for the LHC Exampe Event Structure (BaBar) Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 25

26 Buiding a muti-pb Database for the LHC Meta-Data Conventiona definition: Meta-data = data about data Prefer: Meta-data = data which makes other data usabe Meta-data for an LHC experiment ~TB dominated by detector caibration information See significant benefits in using same system for both data & meta-data can make ogica associations between the two can use different custering strategies meta-data permanenty on disk Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 26

27 Buiding a muti-pb Database for the LHC Short Term Pans Use of Objectivity/DB by existing experiments Growing number of experiments at and outside Even adopted by Fortran-based experiments! LEP data archive pan to archive some 30TB of LEP ( ) data and keep for 20 years understood that reprocessing wi be a major effort wi piggy-back on LHC soution Objectivity/DB, HPSS Pans for COMPASS, NA45: ~400TB/year simiar data voumes at other abs (SLAC, BNL, ) Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 27

28 Buiding a muti-pb Database for the LHC Comparisons with Existing Systems Significanty more powerfu than existing HEP systems Consistent interface to a persistent data Data and meta-data handed by the same system Physica storage mode de-couped from ogica mode Appications much faster to buid e.g. caibration database: 1 student-year vs 10 expert-years Major performance and usabiity improvements seen read ony the needed objects use meaningfu predicates rather than semi-arbitrary bit-mask Much better integration with programming anguage Can hande >> data & provide enhanced functionaity Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 28

29 Buiding a muti-pb Database for the LHC Risk Factors The surviva of a given database vendor The surviva of a given MSS system Neither can be guaranteed over a period of ~20 years! Production depoyment in a fuy distributed environment has yet to be demonstrated Best protection: use widey-used soutions for which there is a demand Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 29

30 Buiding a muti-pb Database for the LHC Summary We have demonstrated that a powerfu, scaabe data management soution can be buit using off-the-shef, standards-infuenced components We beieve that commercia Object Database Management Systems are a viabe soution for buk data storage A combination of an ODBMS + MSS can hande data voumes we into the PB region Parts of the system have been demonstrated in production Fu production is expected in 1999 By 2000, expect > 1PB in Objectivity/DB + HPSS Jamie.Shiers@cern.ch NASA/IEEE MSS symposium, March 98 30