A Metadata Framework for Scientific Data Integration. Li Jianhui Computer Network Information Center Chinese Academy of Sciences

Transcription

1 A Metadata Framework for Scientific Data Integration Li Jianhui Computer Network Information Center Chinese Academy of Sciences

2 Outline SDB Background Metadata Requirements Analysis Issues Considered Framework Architecture Current Implementation

3 SDB Background SDB-Scientific Database A project lasted for almost 20 years From 1986 to now, sponsored by State Planning Commission Chinese Academy of Sciences and Natural Science Foundation of China successively A key project in the 10 th Five-year Program of CAS ( )

4 SDB current status SDB Background A collection of large-scale, multi-discipline, distributed scientific databases 300+ scientific databases Data Volume: 8.2TB Content cover Geo-science, Bio-Science, chemistry, Physics, astronomy etc. 45 institutes of CAS joining 45 websites for more than 200 on-line databases

5 SDB Portal

6 Metadata Requirements Analysis Distribution(45 Institute) Domain (Geo-science, Bio-science, Chemistry, Astronomy..) Data Type (Structure Data, Semi-Structure data, Un-structure data) Context(DB = {Distribution, Domain, Data type}

7 Metadata Requirements Real World Conceptualization The Same Beings Conceptualization Conceptual World C1 O C2 O Model World M11 O M12 O M21 O M22 O Database /data World Distribution DB111 DB112 DB121 DB122 DB211 DB212 DB221 DB222

8 System Heterogeneity Digital media repository management system Database management system DBMS, data model heterogeneity Concurrency control, recovery heterogeneity Operating System File system, naming, type heterogeneity Operations, transaction support heterogeneity Hardware System Instruction set, data coding heterogeneity Autonomy Design Autonomy The conceptualization or semantic interpretation of data The representation and the naming of the data elements The data being managed Constraints used to manage the data The functionality of the system The association and sharing with other systems The implementation Communication Autonomy Execution Autonomy Association Autonomy Heterogeneity Information heterogeneity Semantic Heterogeneity Structural, Representation/Schematic Heterogeneity Syntactic, Format Heterogeneity

9 Metadata Requirements User/app. User/app. User/app. User/app. Accessing Interface 1 Accessing Interface 2 Accessing Interface N Middleware DB1 DB2 DBN DB1 DB2 DBN Flat File Multimedia

10 Metadata Requirements Technical Level Technical View Query formulate & Understand Find & Locate Access Interpret & Understand Integrate Information Level Information View

11 Metadata Requirements Tools&App Data Grid Middleware Data Management Database Schema Management Multimedia Processing Accessing Control Uniform Data Accessing Module Information Service Module Database Portal Metadata Management Others Security&Authority Module Metadata Format Metadata Repository

12 Issues Considered Scientific database lifecycle Administrator& Organizer Searching& Retrieval Utilization User Organization Scientific Database Clear out Value-added App/Service Creation& Collection Creator& Provider

13 Issues Considered Different Views and Different needs General data Integration Domain data Integration Domain data Integration Institute data Integration Institute data Integration Institute data Integration DB1 DB2 DBN DB1 DB1 DB2 DB1 DB2 DBN

14 Issues Considered Different Type Metadata Metadata Content Independent Metadata Content dependent Metadata Content Descriptive Metadata Content based Metadata Domain Independent Metadata Domain Specific Metadata

15 Architecture Rule Set Media-specific Metadata SDB Core Metadata discipline Specific Metadata SDB Metadata Hierarchy Schemes Namespaces Mapping/ Crosswalks Register Management Tool Application Profile RDF/XML MD Instances

16 Semantic Rule Rule Set Root Module Module Component Component Component Component Element Element Element Root-Module-Component-Element

17 Element Semantic Chinese Name English Name Identifier Definition Type Domain Option Max Occurrence Note Semantic Rule

18 Syntax Rule Follow XML/RDF Schema syntax Define some special attribute for element 9 semantic attributes Plus such as default value showable Searchable etc. Defined a Namespace

19 Syntax Rule A Schema Example <?xml version="1.0" encoding="utf-8"?> <!-- edited with XMLSPY v5 U ( by et8 (et8) --> <xs:schema targetnamespace=" xmlns:xs=" xmlns=" // elementformdefault="qualified" attributeformdefault="unqualified"> <xs:element name="person" type="persontype"/> <xs:complextype name="persontype"> <xs:sequence> <xs:element name="name"> <xs:complextype> <xs:complexcontent> <xs:extension base="nametype"> <xs:attribute name="chinesename" type="xs:string" fixed=" "/> <xs:attribute name="englishname" type="xs:string" fixed="name"/> <xs:attribute name="showable" type="showabletype"/> <xs:attribute name="searchable" type="searchabletype"/> </xs:extension> </xs:complexcontent> </xs:complextype> </xs:element>. </xs:schema >

20 Application profile developing rule Strictly Following the semantic and syntax rule Inheriting all mandatory elements of the base schema Adding new module/component/element according special necessaries Clipping optional module/component/element Narrowing value domain of elements Changing a optional module/component/element to a mandatory one

21 Current Implementation SDB Core Metadata Schema Granularity A scientific database/dataset Metadata Content General Attribute&information for a database/dataset discipline independent Media independent Usage A common core for SDB Metadata A Bridge for different metadata schemas interoperability A base for developing SDB metadata application profiles

22 SDBCM Structure SDBCM Description Info Distribution Info Metadata Ref Service Info Structure Info Main Module 0.. Coverage Info Contact Info Reference Module

23 SDBCM Structure SDBCM Description Info Distribution Info Metadata Ref Ref./Mapping Ref./Mapping DC FGDC/ ISO19115 Service Info Structure Info Coverage Info 0.. Ref./Mapping Data Grid Requirements EML Contact Info

24 DatasetDescriptionInfo Title DatasetID TitleType ChineseTitle EnglishTitle Description Info Subject Description Purpose Type SizeType Size Source Provider Contributor UpdateFrequency DatasetDate SubjectType ControlledWord 0.. Keywords 0.. RecordNumber MemorySize DatasetDateType CreationDate LastModified ControlledWordType SubjectWord Thesaurus Language URL Relation 0.. Coverage RelationType RelatedDatasetTitle Relationship RelatedURL

25 ServiceDescriptionInfoType ResourceURI GeneralServiceInfoType IP Port GeneralServiceInfo Protocol DatabaseID ServiceDescription Service Info UserAuthentication ServiceAdmin ServiceConfig UserAuthenticationType UserName Password CertificateAuthority 0.. ServerConfigType ServerType OS DBMS DBMSVersion SearchEntry Service Info Structure

26 StructureDescriptionInfoType EntityType Entity EntityType EntityIdentifier EntityDefinition AttributeType AttributeIdentifier AttributeDefinition StorageType Structure Info Attribute 1.. Length Unit Precision Default ConstraintType PrimaryKey UniqueKey 0.. CheckConstraint Constraint ForeignKey 0.. ForeignKeyType ForeignKeyValue EntityReference Cardinality JointCondition NotNullConstraint Structure Info

27 Current Implementation Domain specific Metadata Schema Metadata Schema for Atmospheric Data Metadata Schema for Long-term Ecology Research Data Ecological Research Metadata- ERM

28 From SDBCM to SDB-ERM SDBCM ISO Inherited Elements Mandatory Elements Revised Elements Extension CSDGM SDB-ERM CERN FLED EML

29 Current Implementation Universal Metadata Management Tool A web-based metadata authoring/editing/managing/searching tool Functions exporting metadata schema Dynamically generating metadata authoring and searching interface supporting customized author&management interface supporting searching by one metadata schema Supporting different schemas mapping and searching cross metadata schemas

30 Current Implementation Universal Metadata Management Tool install & configure MD schema Process (Java bean) MDIS (LDAP) interim XML XML engine page User universal, extensible customizable

31

32

33

34

35 Current Implementation Training and Promotion A Training held in Sept in Qingdao

36 Summaries Metadata is an novel and effective way to integrate scientific database and is also very important for the Scientific Data Grid An open and extensive metadata framework is necessary in a multi-discipline distributed and heterogeneous database environment Good Tools can help metadata implementation

37 Thank you and welcome Comments