The Virtual Solar-Terrestrial Observatory ++

Transcription

1 The Virtual Solar-Terrestrial Observatory ++ Peter Fox* Deborah McGuinness$#, Luca Cinquini%, Patrick West*, Jose Garcia*, Tony Darnell*, James Benedict$, Don Middleton%, Stan Solomon* *HAO/ESSL/NCAR $ McGuinness Associates # Knowledge Systems and AI Lab, Stanford Univ. % SCD/CISL/NCAR 1

2 Outline Semantic Web for Large-Scale Integration of Scientific Data Terminology and general introduction Past projects Guiding principles Details of the VSTO Lessons learned/best practices/progress ORION/CI musings Postscript: education is not left behind 2

3 Terminology Workshop: A Virtual Observatory (VO) is a suite of software applications on a set of computers that allows users to uniformly find, access, and use resources (data, software, document, and image products and services using these) from a collection of distributed product repositories and service providers. A VO is a service that unites services and/or multiple repositories. VxOs - x is one discipline 3 NB: VO also refers to Virtual Organization

4 What should a VO do? Make standard scientific research much more efficient. Even the principal investigator (PI) teams should want to use them. Must improve on existing services (mission and PI sites, etc.). VOs will not replace these, but will use them in new ways. Enable new, global problems to be solved. Rapidly gain integrated views from the solar origin to the terrestrial effects of an event. Find data related to any particular observation. (Ultimately) answer higher-order queries such as Show me the data from cases where a large coronal mass ejection observed by the SolarOrbiting Heliospheric Observatory was also observed in situ. (sciencespeak) or What happens when the Sun disrupts the Earth s 4 environment (general public)

5 Virtual Observatories Conceptual examples: In-situ: Virtual measurements Related measurements Remote sensing: Virtual, integrative measurements Data integration 5 Both usage patterns lead to additional data management challenges at the source and for users; now managing virtual datasets

6 Importance of (interface) stds VO2 VO3 VO1 Semantic mediation layer DB1 DB2 DB3 DBn 6

7 Issues for Virtual Observatories Providing for multiple VOs: consider federating/aggregating rather than one-on-one Branding and attribution (where did this data come from and who gets the credit, is it the correct version, is this an authorative source?) Provenance/derivation (propagating key information as it passes through a variety of services, copies of processing algorithms, ) Data quality, preservation, stewardship, rescue Funding for participation - how to leverage existing efforts Interoperability at a variety of levels (~3) 7

8 What is an Ontology: A branch of study concerned with the nature and relations of being, or things which exist. A formal machine-operational specification of a conceptualization. Semantic Web: an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation, Catalog/ ID Thesauri narrower term relation Terms/ glossary Informal is-a Frames General Formal is-a (properties) Logical constraints Formal instance Disjointness, Value Inverse, partrestrs. of *based on AAAI 99 Ontologies panel McGuinness, Welty, Ushold, Gruninger, Lehmann 8

9 Languages and tools Semantic Web Standards OWL OWL-S Web Ontology Language (W3C) Messaging/services (Submitted W3C note) Semantic Web In-progress standards SWSL ODM Service description (W3C) Ontology Definition Metamodel (OMG) Tools for Semantic Web Editors: Protégé, SWOOP, Medius, Cerebra Construct, SWeDE Reasoners: Pellet, Racer, Cerebra Server, Medius KBS, Search: SWOOGLE swoogle.umbc.edu Other: Jena, SeSAME, Eclipse, KOWARI Collaboration: planetont.org Additional Emerging Semantic Standards for Earth Science SWEET, VSTO, MMI, 9

10 Discipline-based Projects Coupled Energetics and Dynamics of Atmospheric Regions (CEDAR) Mauna Loa Solar Observatory (MLSO) Earth System Grid (ESG) Center for Integrated Space-weather Modeling (CISM) 10

11 CEDARWEB Data query, selection and retrieval interface, with integrated tools, e.g. ability to plot (preview) data before retrieving it participants 600 users 320 instruments 1370 datasets 11

12 CEDARWEB 3.1 Ability to quickly plot data to assess suitability, quality, and produce a quick copy with some customization for a preliminary study. 12

13 Observations of the solar atmosphere Near real-time data from Hawaii from a variety of solar instruments, as a valuable source for space weather, solar variability and basic solar physics 120 users 300,000 datasets 10TB + 13

14 The Earth System Grid DATA storage SECURITY services M ETADATA services LBNL gridftp server/client TRANSPORT services ANALYSIS & VIZ services HRM M ONITORING services FRAM EWORK services ANL DISK Auth metadata NCAR M ysql RLS GSI CAS client TOM CAT SLAM ON daemon NCL opendapg client NERSC HPSS AXIS CAS server GRAM LAS server gridftp server/client HRM NCAR MSS LLNL GSI opendapg server ORNL TOM CAT DISK SLAM ON daemon CDAT opendapg client M ysql gridftp server/client Xindice HRM GSI DISK THREDDS catalogs RLS CAS client M yproxy client gridftp server/client M yproxy server ORNL HPSS opendapg server HRM DISK ISI M ysql RLS GSI CAS client M CS M ysql Xindice GSI 14 OGSA-DAIS M ysql GSI RLS

15 The data grid example - data driven science Earth System Grid (ESG) serving coupled climate system model data to a registered community of ~ 3000 (July) 220 TB, 25 TB delivered in 2005 Data grid based on OPeNDAP-g, subsetting, aggregation, bulk file transfers Since Dec. 2004, the ESG/IPCC clone portal has 28TB published (66,000 files) 650 users/projects, with > 428,000 files downloaded, ~100TB (~200GB/day) > 250 research papers Gearing up for 5th assessment:

16 Center for Integrated Space-weather Modeling (STC led by BU) Goal: To create a physics-based numerical simulation model that describes the space environment from the Sun to the Earth. THE USES OF SPACE WEATHER MODELING A scientific tool for increased understanding of the complex space environment. A specification and forecast tool for space weather prediction. An educational tool for teaching about the space environment. 16

17 Principles used in VSTO Users, users, users Use cases, use case, use cases Same framework for all aspects of data and information flow Rapid development of intelligent lightweight framework and rely on services to do heavy-lifting Job does not end when the user gets the data (still working on this) 17

18 User needs In discussions with data providers and users, the needs are clear: ``Fast access to `portable' data, in a way that works with the tools we have; information must be easy to access, retrieve and work with.' Few clicks, get what I want, whose tools? MY tools Too often users (and data providers) have to deal with the organizational structure of the data sets which varies significantly --- data may be stored at one site in a small number of large files while similar data may be stored at another site in a large number of relatively smaller files. There is an equally large problem with the range of metadata descriptions for the data. Users often only want subsets of the data and struggle with getting it efficiently. One user expresses it as: ``(Please) solve the interface problem.' Encapsulate more 18

19 Integrative use-cases: Find data which represents the state of the neutral atmosphere anywhere above 100km and toward the arctic circle (above 45N) at any time of high geomagnetic activity. Translate this into a complete query for data. Was all the needed information recorded? Information needs to be inferred (and integrated) from the use-case What is returned: Data from instruments, indices and models. 19

20 Why we were led to semantics When we integrate, we integrate concepts, terms In the past we would ask, guess, research a lot, or give up It s pretty much about meaning Semantics can really help find, access, integrate, use, explain, trust What if you - could not only use your data and tools but remote colleague s data and tools? - understood their assumptions, constraints, etc and could evaluate applicability? - knew whose research currently (or in the future) would benefit from your results? - knew whose results were consistent (or inconsistent) with yours? 20

21 What s new in the VSTO? Datasets alone are not sufficient to build a virtual observatory: VSTO integrates tools, models, and data VSTO addresses the interface problem, effectively and scalably VSTO addresses the interdisciplinary metadata and ontology problem - bridging terminology and use of data across disciplines VSTO leverages the development of schema that adequately describe the syntax (name of a variable, its type, dimensions, etc. or the procedure name and argument list, etc.), semantics (what the variable physically is, its units, etc.) and pragmatics (or what the procedure does and returns, etc.) of the datasets and tools. VSTO provides a basis for a framework for building and distributing advanced data assimilation tools 21

22 VSTO - semantics and ontologies in an operational environment: vsto.hao.ucar.edu,

23 23

24 24

25 Provenance 25

26 26

27 27

28 28 Exploring the ontology

29 29

30 30

31 31

32 VSTO Progress Semantic framework developed and built with a small team in a relatively short time Production portal released, includes security, etc. with full community migration (and so far endorsement) VSTO ontology version 0.4, (vsto.owl) Web Services encapsulation of semantic interfaces being documented More use-cases to drive the completion of 32 the ontologies - filling out the instrument ontology

33 Lessons learned/ best practices A little semantics goes a LONG way, and a little more goes even further Semantics is ready for our prime-time Interoperability: the few things we have to agree upon so that we need NOT agree on anything else (EC, 2005) Data management Communities 33 People Software

34 Modern Data Frameworks NOT just for outflow!! middleware WAS middleware NOW 34

35 MUST BE HERE Mostly here Sometimes here 35

36 Community aspects Identifying, and then presenting to, your community Participating in cross-disciplinary CI efforts (often out of band ) and meetings MUST give talks at science conferences MUST document and publish papers Providers and users are peers Vetting of ontology - diverse community required Surprising endorsement of semantic approach once we told them three times 36

37 People Importance of the team (beware of legacy people) - balance and meetings, 10X smaller than ITR, 10X less people shorter timescale - what is the sweet-spot? Chief architect and PI role - It s good to the the KING (Mel Brooks, HOTWPI) Software lead is a key role - make decisions Project management + small teams (2-3) Data scientist as a profession is re-emerging, with a new and different skill-set (informatics) 37

38 Software We built and trashed three prototypes in very short timeframes Balanced design and requirements should drive technology choices (not the other way around) Performance-oriented, sustainable software is fundamental (can handle scalability) Framework is independent of classes and individuals in ontology Security (all aspects) is a key functional requirement 38

39 ORION/CI musings Must be a peer in ORION (not IT ) Satisfy more than only the immediate (CSO, RGO, GSO) interoperability needs - i.e. focus on existing archives Provenance and annotation (aka metadata tagging, social networks, ) is important and linked to the data management plan/architecture Scalability and diversity - VO technology can help with semantic mediation Roles (human and autonomous) - semantics and VO can bring people and services together in new ways 39

40 Questions? Contact 40

41 Education - semantic integration Sun-Earth Connections A 9th grade teacher is preparing a lesson plan aimed at getting students to learn more about the northern lights, addressing NSES content standards in earth science. The teacher wants the students to learn the scientific terminology, where the phenomena occurs and retrieve some data or graphics for a recent occurrence. The goal of the lesson plan is the engage students, using authentic data from the aurora, as part of an inquiry-based program. The aim of this use-case is to translate the teacher s requirements into a search for, and to find a specific dataset appropriate to the education task from a Virtual Observatory. Data would need to be provided in a usable format, as well as in a graphically accessible way. At present, in order for a teacher to access VO data, they would need to know the appropriate scientific vocabulary, the types of data available, spatial locations and directional operating modes of instruments (also models and indices) to be able to locate, retrieve and use the data from them. This use-case will demonstrate how ontologies, and semantically enabled interfaces can bridge the current gap in 41 terminology and significantly reduce the level of detail that a person has to know about the data.

42 Vocabulary and beyond Teacher (hasgradelevel=9) - primary actor Lesson plan - intermediate product which connects actors Students - secondary actor Learn - process outcome Northern lights - identified phenomenon in non-science vocabulary NSES content standards (controlled vocabulary), ISO std. and ontology Earth science (application domain) Scientific terminology (controlled vocabulary) Where phenomena occurs (spatial, coordinates) Retrieve data (outcome, service) Retrieve graphics (outcome, service) Recent occurrence (time) Authentic data (validation, verification, domain and range) Aurora (phenomenon) Inquiry-based program -> develop ontology, i.e. classes and subclass relations, properties, inherit existing ontologies, rapid prototype 42

43 43

44 44

45 45

46 46

47 Data: Diversity, Integration, Size, Data policies are still highly variable or non-existent - how can data be managed to solve challenging scientific problem, societal problems without the continued need for a scientist to know every details of complex data management systems Not just large (well organized, long-lived, well-funded) projects/programs want to make their data available What does a large-scale, integrated, scientific data repository look like today? s i t n a e m t e a g d a n e a c r m u o a t s a as d l a em n o l s Most data still createdrin a manner to simplify generation, not access or b e g o r r p r p use o ;. s > r e Oof data; files, directories, metadata, e Leadstto very diverse organization t g V g < i a y s, etc. b m m r e Source/origin o z ; i management is driven by meta-mechanisms for t, S bintegration, n ig minteroperability e (but still need performance) s e a n avirtual g Observatories Grids ma Data Data assimilation Increasing realization: need management for all forms of data 47