2 Databases for calibration and bookkeeping purposes

Transcription

1 Databases for High Energy Physics D. Baden University of Maryland. B. Linder ORACLE corporation R. Mount Califomia Institute of Technology J. Shiers CERN, Geneva, Switzerland. This paper will examine the two important questions conceming databases for I-IEP: Should we buy or build database systems for our calibration and bookkeeping needs? Will database technology advance sufficiently in the next 8 to 10 years to be able to provide byte-level access to petabytes of SSC/LHC data? 1 Introduction This main thrust of this session was a discussion of the future role of database management systems in HER Should physics analysis of Gigabyte, Terabyte or Petabyte datasets be regarded as just another database problem to be solved using tools provided by commercial systems? M11 commercial database systems of the year 2000 routinely handle Petabyte databases? If we use commercial systems, to what extent must our approach to physics analysis be re-thought? In addition to this important but necessarily inconclusive debate, some time was devoted to a brief review of the old debate about the relative merits of commercial and HEP made database systems for the modestly sized calibration and bookkeeping databases. 2 Databases for calibration and bookkeeping purposes Experience and LEP and FNAL clearly show that a database system is a requirement for today s experiments. The question as to whether such a system should be built or bought has been discussed many times, included at previous conferences in this series. [3] It is clear that there are two sides to this argument: Is it technically possible to use a commercial system? Would it be managable administratively and financially? Answers to these two questions are somewhat more positive than they were in the early days of LEP planning. Technically speaking, it would have been possible to use a commerical system with some pain in Today, database systems are much more mature and powerful and so what would have been a possibility in 1984 becomes a probability in However, the administrative and financial burdon would have been unsupportable in Today, the problems in this area could conceivably be solved, although they are by no means trivial. 81 OCR Output

2 2.1 Technical arguments In terms of performance and efficiency, home grown solutions have a clear advantage. However, few HEP applications are database bound and commercial systems have certainly improved over the last 8 years in this respect. Commercial solutions certainly have the advantage in terms of data integrity and reliability, but there is no system that is 100% reliable. Commercial systems also provide interactive interfaces, e.g. through SQL, ranging from adequate to good. However, FORTRAN does not mix well with SQL, although there is hope that there will be significant improvement in this area in the future. Finally, a significant amount of HEP specific code would have to be added to any com merical system. The amount of this code is roughly comparable in size (within a factor or two or so) to existing home- grown solutions. 2.2 Administrative and financial arguments Commerical systems tend to require well trained support staff at every site. HEP solutions also require effort for development, maintenance, installation and support, but experience shows that this is considerable less than the total requirement for commerical systems. The sheer size and diversity of HEP collaborations also poses problems in terms of licensing and standardisation. The SQL standard alone is too basic to ensure portability of applications across systems, and there are currently many sites running different database systems for different applications. The sheer necessity of buying a license per site or worse, multiple licenses per site also poses a problem, which is only intensihed if the support is not provided by a vendor with whom the institute already deals. 3 Use of existing packages for data management in the LHC/SSC era A possible scenario for LHC and SSC experiments would be to use existing packages, such as those provided in the CERN Program Library. More specifically, one can ask the following questions: Can Zebra be used in the year Should Zebra be used in the year Can FATMEN be used as a file catalogue for multiple PB of data Can we use existing data access methods and management tools How should we store the data Actually in a database In some super file system, e.g. what one hopes the IEEE Hass Storage Systems Reference Model [1] will become 82 OCR Output

3 3.1 Use of Zebra We already know of some areas where Zebra [2] needs to be improved. However, given the time scale of LHC and SSC, one would hope that one could exploit features of languages such as Fortran 90 and/or c++ to provide the functionality currently offered by Zebra in a more natural and powerful way. Currently, opinion is divided as to whether Fortran 90 is sufficiently powerful for this purpose. Initial investigations suggest that it could indeed provide the functionality that we require, provided that we do not simply try to rewrite Zebra in Fortran 90, but rather re examine our data suucture requirements and see how well these map on to the Fortran 90 model. 3.2 Use of FATMEN The FATMEN package is described elsewhere in these proceedings [4]. There is currently a restriction in the Zebra RZ package [5] which means that one can only reference 4 million files using a single FATMEN catalogue. This restriction will be removed in the near future. It is clear that a system for the storage of meta-data is required, although one would strongly hope that the NAMES SERVER component of the IEEE MSS reference model would address and solve this problem in a manner suitable for HEP by the end of the current millenium. 3.3 Data access and management We are currently seeing a move from the traditional, sequential method of accessing data to random access. This area still requires more development, but one predict with a reasonable degree of confidence that this trend will accelerate. In the short term, there are a number of problems that remain to be solved conceming the efficient usage of high capacity to media, and network tape staging issues. Data management tools are still rather primitive and often experiment specific. More and more experiments are moving towards standard software, but this remains an area where a great deal of work is still required. 3.4 Storage of the data In addition to the native filesystems that come with the operating system, there are a number of packages that were developed according to earlier versions of the IEEE MSS model. Unfortu nately, none of these systems are sufficiently powerful to solve the HEP data storage problem. We must therefore ensure that we provide sufficient input to the various IEEE working groups so that such deficienies are corrected in future revisions of the model and any standards based upon 1t. Databases will certainly play a role in storage of meta-data, although one could argue that the current predominant relational model is not well suited to this task. It is, however, much less likely that the data itself will be stored in the database, particular in view of the requirement of efficient byte level access to multiple petabytes of data in the fully distributed, heterogenous environment. 83 OCR Output

4 4 Towards alternative data management schemes The CDF experiment at the Fermilab collider collects on the order of 1 Terabyte of raw data per year. The SDC experiment, planned for the SSC, is expected to take some three orders of magnitude more data per year, i.e. a few Petabytes. If one assumes a file size of 100 MB, being a size that is convenient to handle, this leads to 10 million new files per year. Although the currently tools for data management and access work, it is less obvious that they will scale by three orders of magnitude. Areas that have to be solved include Processing, history (reconstruction, sorting etc.) File location and retrieval/access Data disuibution A project is currently under way to investigate alternative approaches to data management. The project includes members from the Universities of Maryland and Illinois at Chicago plus LBL, ANL, FNAL and the SSCL. This research is organised around a few basic principles: HEP data is not intrinsically relational one should read only the data that is required Data duplication should be avoided Follow EEE Mass Storage Systems Reference Model investigate new datastructures More details of this work can be seen in the paper presented by C. Day, in the proceedings of this conference. [4] The requirements cited above have led to an investigation of object oriented database technology. Whilst still in its infancy, this technology provides a number of interesting ideas for altemative approaches to data management and structures. In particular, Object Design Incorporated produces the Dbj ectstore product for CAD/ CAE. With this tool, data is managed as c++ structs, which are managed as persistant entities via Db j ectstere. In this manner one can map an entire data set to virtual memory, and hence reduce the data management problem to the management of virtual pages. This satisfies the criterium whereby only the required data is read, but inroduces numerous other complications due to concurrency etc. Much work still remains to be done to optimise the above scheme for HEP and a number of open questions still remain: Can we influence, or at least keep abreast of, progress in the area of the EEE MSS reference model? 84 OCR Output

5 Can data subsets, c.g. mini-dsts ctc. bc implcmcntcd using pointcrs t0 more complctc data samples? Since this work involves rethinking data structures, can one also use this opportunity to extend current attempts at physics without programing? 5 DBMS futures Database management systems provide a number of important features, including Common methods for data access Data management Data models Portability at both the hardware and applications level Concurrency management In the future, one can expect orders of magnitude growth in the size of databases, the number of users and the complexity of database applications. Equally, whereas databases are primarily stored on disk today, one can anticipate the use of magnetic, optical and solid state media in the future. Access to databases will become more and more distributed, underlining the requirement for high bandwidth connectivity. Future database models will include the relational, object and fiat models, plus combi nations of the above. One can expect to see the different database models sitting on top of a common data access management layer, which in tum uses common concurrency, recovery and consistency tools. ORACLE directions include the use the Massively Parallel machines, cached databases and a variety of media including RAM, disk and tape. In addition, one can expect particular emphasis to be placed upon data management issues, including Disk management Backup/recover Very large database issues Finally, one can expect further evolution of the database model. 85 OCR Output

6