Code-independent Database

Transcription

1 The NOMAD 1 (Novel Materials Discovery) Laboratory a European Centre of Excellence Code-independent Database Document date: 02 May 2016 Document version: 1.0 Deliverable No: 2.1 Full Name Short Beneficiary Role Name Number MAX-PLANCK-GESELLSCHAFT ZUR FOERDERUNG DER MPG 2 1 Coordinator WISSENSCHAFTEN E.V. KING'S COLLEGE LONDON KCL 2 Beneficiary HUMBOLDT-UNIVERSITAET ZU BERLIN HUB 3 Beneficiary THE CHANCELLOR, MASTERS AND SCHOLARS OF THE CAM 4 Beneficiary UNIVERSITY OF CAMBRIDGE UNIVERSITAT DE BARCELONA UB 5 Beneficiary AALTO-KORKEAKOULUSAATIO AALTO 6 Beneficiary DANMARKS TEKNISKE UNIVERSITET DTU 7 Beneficiary BAYERISCHE AKADEMIE DER WISSENSCHAFTEN LRZ 8 Beneficiary CSC-TIETEEN TIETOTEKNIIKAN KESKUS OY CSC 9 Beneficiary BARCELONA SUPERCOMPUTING CENTER - CENTRO BSC 10 Beneficiary NACIONAL DE SUPERCOMPUTACION PINTAIL LIMITED PT 11 Beneficiary 1 Note that the abbreviation has been changed from NoMaD to NOMAD, and this will be used from now on. 2 Beneficiary MPG includes three groups active in the NOMAD Laboratory CoE - FHI, MPCDF and MPSD. D2.1 Code-independent Database 1

2 Abstract Processing, management, and storage of data coming from materials science calculations lies at the very heart of the NOMAD Laboratory CoE. This deliverable report describes the planning and implementation of the central data storage, called the NOMAD Database. It consists of the established Metadata structure, that provides a formal description of the calculation data, as well as the NOMAD Archive and the Encyclopedia Database where the code-independent output of the socalled Conversion Layer are stored. It also covers various layers of data processing up to a model graphical user interface. Copyright 2016 by the NOMAD Consortium. The information in this document is proprietary to the NOMAD Consortium. This document contains preliminary information and is not subject to any license agreement or any other agreement with the NOMAD Consortium. This document contains only intended strategies, developments, and functionalities and is not intended to be binding upon to any particular course of business, product strategy, and/or development of the NOMAD Consortium. The NOMAD Consortium assume no responsibility for errors or omissions in this document. Furthermore, the NOMADConsortium does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. The NOMAD Consortium shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. In addition, the materials presented and views expressed here are the responsibility of the author(s) only. The EU Commission takes no responsibility for any use made of the information set out. D2.1 Code-independent Database 2

3 Code-independent Database 1 Introduction Background Role and scope of this deliverable Interaction with other work packages Approach Institutions involved Approach Results System-infrastructure Metadata NOMAD Archive Preprocessing Encyclopedia Database Encyclopedia API and Mini-GUI Next steps and further evolution of the database Impact Conclusion D2.1 Code-independent Database 3

4 1 Introduction NOMAD is a truly interdisciplinary project where computer scientists meet experts from physics, chemistry, materials science, and applied mathematics. The report reflects this situation, and gives at certain points explanations that might appear trivial to an expert in a certain field. The key (long term) goals of The Novel Materials Discovery Laboratory (NOMAD) Centre of Excellence (CoE) ( are to develop a Materials Encyclopedia and Big-Data Analytics Tools for materials science and engineering. Computational materials science is using many different simulation packages involving many different types of theoretical approaches, from empirical to first principles. These packages are usually called codes. The NoMaD Repository ( is an open access database for data from computational materials-science. It is maintained by the groups of Matthias Scheffler (FHI-MPG), Claudia Draxl (HUB), and Stefan Heinzel (MPCDF-MPG). All three are also Principal Investigators (PIs) of the NOMAD Laboratory CoE. In contrast to any other such database, the Repository accepts (and requests) all input and output files of computations produced by any of the popular codes. It fulfills scientific needs and legal requirements well, e.g. by guaranteeing data storage for at least 10 years and optionally keeping the files private for up to three years. At the time of writing this report, it contains already over 3 million calculations, which corresponds to more than a billion CPU-hours. Due to its wealth, this repository represents the ideal starting point for the purpose of the NOMAD Laboratory CoE. However, the data structures within the Repository are too heterogeneous to be directly usable for the goals of the NOMAD Laboratory CoE. This problem is overcome by the development of a Conversion Layer (WP1) that generates the necessarily code-independent data. The conversion has already been described in deliverable D1.1 Data Formats and Compression. For making the data generated by the Conversion Layer available for the challenging applications of the NOMAD Laboratory CoE, a stable way of describing, storing, and accessing the data is most crucial. This is what we call the NOMAD Database, which lies at the very heart of the NOMAD Laboratory CoE and is subject of this deliverable. The report at hand describes its implementation, the basic concepts and choices behind it, and the next steps to follow. The following terms are used extensively throughout this report: The NOMAD Laboratory CoE is the name of this European Centre of Excellence that develops a Materials Encyclopedia and Big-Data Analytics tools for materials science and engineering. The NoMaD Repository is an open-access repository independent of the NOMAD Laboratory CoE. It hosts input and output files of materials-science calculations, and is the main source of data for the NOMAD Laboratory CoE. The NOMAD Archive contains the data obtained from the NoMaD Repository after parsing all its information and converting these data to a code-independent format. This is part of the NOMAD Laboratory CoE. The Materials Encyclopedia transforms the calculation-oriented information in the Archive to materials-oriented knowledge and provides a user-friendly public access point to it. The Data Management and Storage System (DMSS) is a module allowing to store the codeindependent data in a form suitable for the Materials Encyclopedia. Data can have various meanings in the context of this project. Thus we use this word only in the generic sense, and for its specific meanings: D2.1 Code-independent Database 4

5 o o Calculation: A collection of everything related to a single computation executed with a materials-science code. This corresponds to one entry in the Repository. Quantity: Each calculation depends on a set of parameters and retrieves various results. Each of those is regarded a quantity. Metadata is the name or label that characterizes corresponding values, which can be quantities, but also information about the relation between calculations. For example, program_name is a metadata and VASP may be the corresponding value. A Materials Ontology classifies materials and describes relations between them. The Demonstrator is the software product of deliverable D2.2 (October 2016), providing a fully functional graphical user interface (GUI) for characterizing one class of materials. For the sake of easy reading, these terms are often abbreviated by leaving out NOMAD or other qualifiers that are not needed in the specific context; for example, we may use Encyclopedia instead of Materials Encyclopedia. 2 Background 2.1 Role and scope of this deliverable The data in the NOMAD Laboratory CoE comes from the NoMaD Repository, a hub for uploading and storing input and output files in their raw - code-dependent - form. This data is then processed by a Conversion Layer into a code-independent form and stored in the Archive. The Encyclopedia provides a materials-oriented view on the calculation-based code-independent data of the Archive. To this extent, the information contained in the Archive is complemented by further processing and stored in the so-called Data Management and Storage System (DMSS). Outside users, as well as the Visualization and Analytics tools, have access to this data. Obviously, these tools may also access the Archive directly (see Figure 1). The scope of this deliverable is the storage of data in the Archive and the Encyclopedia Data Management and Storage System. Hereby, we focus on the infrastructure rather than the actual data contained, as they are continuously evolving. 2.2 Interaction with other work packages A major effort of the first phase of the CoE has been devoted to the Conversion Layer (WP1) as described in D1.1 Data formats and compression. In turn, the Big-Data Analytics work package (WP4) needs access to code-independent data provided by the Archive and DMSS (D4.1 Analytics toolkit initial deployment, due M12). The same applies to the Visualization work package (WP3), which also provides tools that are (partly) incorporated in the Encyclopedia graphical user interface. The development and deployment of the database and storage infrastructure is sustained by the High-performance computing (HPC) Infrastructure work package (WP6). It also provides consulting regarding technologies to use and performance aspects. D2.1 Code-independent Database 5

6 Figure 1: Part of the software-infrastructure required for the Demonstrator, and interaction with other work packages. Abbrevations in this and the following graphics are: API Application Programming Interface, GUI Graphical User Interface, CLI Command Line Interface 3 Approach 3.1 Institutions involved The partners responsible for the realization of the here described infrastructure are MPG-FHI for the Archive (WP1) and HUB for the Materials Encyclopedia (WP2). In the scope of WP6, MPG-MPCDF provides the infrastructure for the development and the initial production systems. MPG-MPSD supports this action with a responsibility for linking between WP1 and WP2. Further valuable input comes from BSC s GUI development team, and MPG-MPCDF members, who bring in their experience with the NoMaD Repository database. 3.2 Approach The NOMAD Metadata is used to identify, describe, and classify the quantities stored in the NOMAD Database, as well as relations between calculations. It can be regarded as a central register (dictionary) that provides a clear nomenclature for accessing the actual data. Often metadata is limited to this dictionary function, but in this project we also use it as our conceptual model of the data. This means we give it structure, also defining how the different pieces of data are organized using a relational model, including hierarchical structuring. This is important both for humans and automated data processing to enable complex workflows and analysis. The Archive, as the output of the Conversion Layer (see D1.1), holds the parsed and normalized calculated data that are openly accessible in the NoMaD Repository at the time. Within each entry (corresponding to a calculation), all quantities recognized by the parsers are present. New openaccess data in the Repository is automatically ingested into the Archive. For different applications D2.1 Code-independent Database 6

7 developed in various work packages it serves as code-independent and calculation-oriented source of data. In contrast, the Encyclopedia provides a material-oriented view on this data, using all the calculations and quantities that can be processed by the Encyclopedia. For example, the atomic structure of a material is visualized using the information about the atomic positions and unit cell, thus this data is available in the Encyclopedia. At the same time, as the first development stage is based on bulk materials, only corresponding calculations will be presented in the Encyclopedia. A comprehensive description of a material may also require quantities that are not explicitly present in the raw data. Thus, they have to be derived from other, known quantities. This concerns, for instance, the information whether a material is a bulk system and the corresponding space group. Both are determined from the atomic structure provided in a calculation s input file in the Repository. Derived quantities will be obtained in the Conversion Layer or in the Encyclopedia Preprocessing. As the requirements to a data-providing service from the Encyclopedia are very specific, particularly regarding response times to crucial queries, we decided to use an additional database system optimized for the Encyclopedia. It contains the above-mentioned Encyclopedia-relevant data. In a similar manner other work packages, mainly WP1 and WP4, use specialized, possibly temporary database systems, which are not regarded being part of the NOMAD database. In summary, the code-independent NOMAD Database consists of the calculation-oriented Archive and the materials-oriented Encyclopedia Database, both providing open interfaces for data-access. At any point, the Encyclopedia functionality defines boundary conditions for the development of the Encyclopedia Database, concerning the following questions: How many calculations are expected to be available? Which quantities have to be available; how often are they accessed; what response time is needed? What kinds of relationships exist amongst different quantities and calculations? What will be the most complex queries? In all these respects, the Encyclopedia will evolve during the project lifetime (and most likely even much further), and it will be supplemented with: classification in terms of materials classes and families, incorporation of new calculation methodologies (e.g. new density functionals, new force fields, approaches to excitations and charge-carrier transport, ), analytics tools, and specific (industrial or scientific) use cases. Many aspects of possible future developments cannot be fully defined at the moment, and depend on the outcome of other work. Thus the Encyclopedia will be developed in several steps, the first one being defined by deliverable D2.2, that is a first Encyclopedia Demonstrator, aiming at a basic material characterization: D2.1 Code-independent Database 7

8 The Demonstrator covers bulk materials, describing their atomistic structure, most relevant electronic properties, and details of the underlying calculations (computed by density-functional theory). The user may search for such materials according to the elements they contain, or a specific chemical formula. Besides the technical description of the data storage - the actual scope of this deliverable - we also provide a demonstration of a simple Encyclopedia API together with a visualization of the database content, the so-called Mini-GUI. This is a first step towards the tools to be used in the Demonstrator GUI. To this extent, the Encyclopedia Database has to be filled, which is the task of the Preprocessing module. As the preprocessing of all the data for the Demonstrator depends on complex algorithms (e.g. classification of materials by structural characteristics, electronic structure, material families and functions, etc.) that are still under development, only a minimal functionality for the data subject to the Mini-GUI is implemented. For a first implementation, as described here, we have chosen a simplified architecture concerning system (Figure 2) and software (Figure 3). 4 Results This section describes the status of the actual implementation of the NOMAD database related system and software infrastructure as of M System-infrastructure Figure 2: System infrastructure for running the packages presented in this deliverable. D2.1 Code-independent Database 8

9 The platform currently deployed by MPCDF-MPG (Figure 2) consists of various virtual machines (VMs) for development and production work. The Archive uses Docker 3 containers coordinated by kubernetes 4. The different parsing steps are executed in separate containers and the queue with the steps to be executed uses RabbitMQ 5. It is planned to distribute them on multiple VMs, which allows us to rapidly increase the number of workers, as well as easily run older versions of the conversion and normalization code. The Encyclopedia back-end (server-side-software), including database and preprocessing software, is installed on a single virtual machine. The front-end (client-side software) is provided by an nginx 6 web server, also running on the same VM. The source code and project related documentation are hosted centrally in a GitLab 7 installation provided by MPCDF-MPG. This platform is also used for certain communication and planning purposes. The technical details for reproducing this setup can be found on: Figure 3: Software infrastructure implemented for this deliverable. 4.2 Metadata The Metadata identifying, describing, and classifying the quantities stored in the NOMAD Database use various tools around a JSON 8 representation, also including descriptions of each single Metadata D2.1 Code-independent Database 9

10 entry. A RESTful 9 interface to access the Metadata description is available, and a human-friendly representation can be browsed at The concept has been extensively described in deliverable D1.1, and more information is available in the public wiki of the project ( Also the related git-repository itself is public, so any interested person can follow the developments. 4.3 NOMAD Archive The Archive stores the result of the conversion layer that converts the inputs and outputs of the calculations according to our Metadata model, and then performs a normalization step to compute the common properties. These are stored as files on a parallel filesystem using a JSON representation and soon also in HDF5 format, with every calculation corresponding to one file. The Metadata scheme allows one to identify the data stored in the Archive in a clear way. However, the content of these files depends also on the version of the program used to process them. Thus, the generated files are grouped in directories depending on the original files in the Repository and the parser used, including its version, such that files produced by different versions of our converters are stored in different places. This allows, for example, for comparing the result of parsing a certain quantity produced by parser version V1 with the one generated by version V2. We use git describe to have a human-readable, unique version for every part that processes a given file. The current Archive API consists of direct access to the JSON files and their description in the Metadata. A REST interface to access the data, similar to the existing one for the Metadata, is planned. 4.4 Preprocessing The Preprocessing links the data in the Archive to the Encyclopedia Database. In the current implementation also all derived quantities, that are not present in the Archive, are calculated in this step. Amongst the data presented in the Mini-GUI this applies to the chemical formula, calculated from the list of atoms, and the space group, which is determined from the atomic structure using the well-established library spglib Encyclopedia Database The Materials Encyclopedia translates between the calculation-oriented view in the Archive and the material-oriented view presented to the user. A material is characterized by its classification with respect to various categories, as well as by certain computed properties (quantities). This is reflected in the database schema depicted in Figure 4. The current schema has been designed with the focus on basic but realistic usages. As we are extracting calculated data and adapt it to a material-centric view, following considerations were included: Materials and calculations are the central entities, grouping the vast majority of the commonly-used data. 8 A human-readable format widely used on the web, see 9 REST (Representational State Transfer) is an architectural style heavily used in web-development 10 D2.1 Code-independent Database 10

11 Less-common or sparsely available data is extracted to separate tables (in the context of databases this is called normalization). Based on our analysis of the expected database queries, the overall amount of computation on the database level (e.g. number of JOINs) has been reduced. Even the database schema of the Demonstrator will evolve according to performance optimizations. Those are based on benchmarking actual queries on large datasets. From a physics perspective, the data can be divided into quantities concerning material classification, atomistic structure, methodology, calculation results, additional information, and grouping of calculations. How the database schema 11 (as in Figure 4) relates to this division is explained in Table 1. From the implementation point of view, the database has to be chosen according to the boundary conditions described in section 3.2 and fulfilling the following criteria: open-source to align with the principles of open science, as promoted by the project, stable (production-ready), scalable for future extensions, and able to handle terabytes of textual and binary data. Due to the expected performance benefits for our envisioned use cases, we decided for a relational database system. All these points reduce the possible options basically to MySQL 12 /MariaDB 13 and PostgreSQL 14. The latter has already been used for the NoMaD Repository. Regarding this, expertise with this database implementation, and the fact that future experiences can be shared, were valued as mutual benefit. Thus, it was decided in favor of PostgreSQL. Integral part of the software design is, that the actual database technology can be exchanged with minimal effort. With every large development step other database systems will be evaluated and integrated if advantageous on the long run. 11 Later a system for using the metadata definition (WP1) as fields names will be implemented D2.1 Code-independent Database 11

12 Table Fields Material classification material all fields in table Atomistic structure position all fields in table element all fields in table cell all fields in table Methodology calc level_of_theory code_name code_version basis_set_type basis_set_qq core_electron_treatment pseudopotential_description xc_treatment_basic xc_treatment_optional DOS_method k_point_related_quantities 15 all fields in table Calculation results calc conductivity has_direct_gap bandgap is_relaxed fermi_surface cohesive_energy energy_decomposition all fields in table DOS all fields in table k_point_related_quantities 15 all fields in table Additional information authors all fields in table files all fields in table Grouping of calculations calc_set all fields in table Table 1: Grouping of database fields according to physical interpretation. 15 The table k_points is a general storage for k-point related quantities. It contains the k-grid used for the self-consistent calculation as well as k-point resolved results, e.g. the bandstructure. D2.1 Code-independent Database 12

13 Figure 4: Encyclopedia Database schema currently implemented with respect to the Demonstrator (upcoming deliverable, due M12). D2.1 Code-independent Database 13

14 4.6 Encyclopedia API and Mini-GUI The Mini-GUI (Figure 5) is a tool covering very little data, just for demonstrating the data-workflow from the Encyclopedia Database to the user interface. It allows for querying materials that contain certain elements. These may be chosen from a periodic table of elements upon click. Selecting multiple elements means to look for materials containing all of them (Figure 6). After submitting a query, the result is displayed as simple list, making use of pagination. The Mini-GUI is implemented as rich web-client application, following the Single Page Application pattern. Most of the application code (JavaScript) and other resources (HTML, CSS, etc.) are retrieved at first page load. As the application needs new resources or data, it sends AJAX requests to get them, thereby avoiding page reloads. It gets its data in a very simple form of the Encyclopedia API, which provides an access point for GETrequests of the form: /materials/search?element=elements&page=page-number&limit=items-per-page In this request all parameters are needed, their meanings are: element: a comma-separated list of the chemical elements to search for e.g. element=n,h,c,ba page: the page number to show, 1 being the first page. limit: the number of entries shown on one page. The technologies used are: Python to build the back-end with the Flask 16 micro-framework for developing the RESTful API, Gunicorn 17 (Green Unicorn) as WSGI server to serve our main Python application; it is the recommended choice for Python web applications, nginx 6 as web server; it is designed to serve high-load web services and also includes handy features like reverse-proxy, load-balancing, basic authentication which we will use/need in the future, Supervisor 18 as process manager on the server; it provides a consistent interface through which long-running programs can be monitored and controlled, and Java Script, HTML, CSS to build the front-end (web-application) D2.1 Code-independent Database 14

15 Figure 5: Screenshot of the Mini-GUI working on a partial set of data. A search for silicon yields the list shown. D2.1 Code-independent Database 15

16 Figure 6: Similar to Figure 5, but now searching for the elements silicon and carbon, which reduces the result set to systems containing both. D2.1 Code-independent Database 16

17 4.7 Next steps and further evolution of the database In the following lists, the partner responsible for the respective task is mentioned in brackets. The next development step, the Demonstrator to be finished by October 2016, includes the following database-related improvements: implementation of a REST-API for the Archive [MPG-FHI], support of HDF5 as Archive file storage [MPG-FHI], definition of benchmarks followed by performance analysis of the Encyclopedia Database [HUB], further modularization of the full Encyclopedia software stack with the focus on running it in a container-based environment; these modules represent loosely coupled components, connected through well-defined interfaces (APIs) [HUB], and setup of development, test, and operational platforms on the new hardware that is being installed now at MPG-MPCDF, as well as on existing hardware at the other computing centers [MPG-MPCDF, LRZ, CSC, BSC]. Further ahead, the established system will be distributed, allowing parallelization on various levels. This is ongoing work following needs and previsions. On the most basic level, the parsing infrastructure, such parallelization has happened already. In other cases it will depend on the envisioned use-cases and the results of performance analysis gathered. [MPG-FHI, HUB, MPG- MPCDF, LRZ, CSC, BSC]. Apart from these technological advances, also, the Encyclopedia functionality will be extended. This affects the whole data processing from the Preprocessing module [MPG-MPSD], over the Encyclopedia Database [HUB], to the Encyclopedia GUI [BSC]. These extensions are required by the different types of methodology underlying the Repository data (molecular dynamics, excited state methods ) as well as further characterizations of materials. The classification scheme distinguishes for instance between: system type (atom, molecule, cluster, surface, adsorption system, polymer, interface, bulk, 2D, alloy, ), known structure type (zincblende, wurtzite, rocksalt, ), material family (perovskites, clathrates, skutterudits, ), and materials function (thermoelectric, supercoductor, ). [with input from FHI-MPG, KCL, HUB, CAM, UB, AALTO, DTU] Another conceptual extension is the development of a Materials Ontology, which implies implementing a database system and interfaces (API and GUI) for the related data [BSC and HUB]. D2.1 Code-independent Database 17

18 5 Impact The Archive-API completes the infrastructure of the Conversion Layer (WP1). As the Archive already contains massive data, this provides the base for their developments of Big-data Analytics tools (WP4) (initial deployment of a toolkit with deliverable D4.1). For the Materials Encyclopedia (WP2) all software-layers up to the GUI can now be developed and tested, targeting deliverable D2.2. This further includes related visualization tools (WP3). The work on data standards contributes also to future dissemination actions (WP7), like the conference on Open Database Integration for Materials Design, October 2016 in Leiden. This will be reported in deliverable D Conclusion This document reports the state of the storage for code-independent data in the NOMAD Laboratory as of M6. It describes the already established infrastructure that can host the output of the parsing effort and provides the base for the next developments. The following aspects are covered: system infrastructure (platform), metadata concept for describing the data, NOMAD Archive containing the parser output, basic data processing, Encyclopedia database structure and implementation, Mini-GUI demonstrating basic query-functionality, relation of the NOMAD Database to other parts of the project, and outlook to the next steps of the database development. D2.1 Code-independent Database 18