Simplify EO data exploitation for information-based services

Transcription

1 RESEARch And SERvicE SuppORt Simplify EO data exploitation for information-based services European Space Agency

2 As part of ESA s Earth Observation Ground Segment Department, RSS has the mission to provide tools and services that can help the EO community in exploiting the EO data, the researchers in developing applications and the service providers in generating value added information. RSS also supports and promotes ground segment harmonisation activities, is responsible for the identification and collection of ground segment technology needs, interfaces with industry for the definition of new technology activities, and manages innovative technology projects. The Join & Share area makes available: A Blog dedicated to describe and explain best practices in service creation and prototyping Forums to find new partners, also among the research Institutes, to implement new, time to market and effective services EO technology projects wikies Forums for technical discussions on the RSS environments supporting the implementation and provision of data exploitation and informationbased services (G-POD, SSE, TOOLBOX, KEO). Listening to the needs and expectations of the EO users and all partners is part of the every day work of RSS. A direct consequence of this approach is a constant cooperation with the researchers and industries to explore new means of data access and utilisation with the goal of bringing together EO science, applications and services. To promote collaboration, the RSS web portal provides also a social network area, called Join&share, that offers to the EO actors the opportunity to share ideas, projects and skills in the Earth science research and services. From Satellite data to value added information, RSS offers specific solutions answering to the EO community needs

3 RSS provides: e-collaboration Environments Research, develop and make available environments for distributed collaboration, systematic or ad-hoc extraction of information, service chaining and provision Applications and Services Support application development and service provision, also as result of cooperation among scientist, value adders and service providers information Based Services Support the development and provision of services based on the information extracted from the real time or archived EO data test-beds and Reference Systems Support the development and provision of test-beds, reference systems and data-sets

4 ESA 2010 RSS join&share area: wiki.services.eoportal.org

5 GRiD PROcESSinG On DEMAnD Image processed by G-pod The ESA G-POD entered into operation in 2004 as a strong and powerful infrastructure dedicated to assist the generation of scientific value added products. The platform supports all the earth science requirements, most of all in terms of providing near real time processing of historical and multisource EO mission data. Thanks to G-POD scientists can create high-level products in a shorter time and with less costs and efforts. How? Just browsing the G-POD Web Portal: a flexible, secure, generic and distributed platform where users can configure, define, execute and monitor services through simple visualisation interfaces. G-POD Major Milestones G-POD Transfer to Operation Sept 2006 G-POD Call for Proposals Present G-POD Cat-1 AO open (continuous call) G-POD is used to demonstrate the potential of Cloud Computing in Earth Observation G-POD UK-PAC with (A)ATSR data and services G-POD Cloud ERS 2 Wave and Wind-Scatt Level 0 reprocessing campaigns G-POD 2.0 open to public 2004 JUN 2006 JUL 2007 OCT 2009 APR 2010 MAY 2010 ESA 2010 JUN

6 GRiD PROcESSinG On DEMAnD G-pod Web portal how to create a new G-POD service 1 Proposal: Scientist submits Application Description 2 Design: Scientist designs processor modules and related data flows. G-POD Team designs the application (encapsulation) 3 implementation: Scientist develops/adapts processor modules. G-POD Team integrates processor in a G-POD service. Scientist analyses results 4 Exploitation: Scientist keeps Intellectual Property. G-POD Team exploits service G-pod resources Trend: 10% yearly increment ESRin Site Over 250 Computing Nodes Over 200 TB On-line Storage UK-PAC Site 112 Computing Nodes 300 TB On-line Storage Network Gbit LAN, Mbps to GARR OS: Scientific Linux 5.2 Software Resources on-line IDL, Matlab BEAT, BEAM, BEST, CQFD, NEST, BRAT Middleware Globus Toolkit G-pod interoperability In addition to end-user services, G-pod fully exposes all its services via: SOAP Web-services and OpenSearch; Third Party application can, therefore, embed G-Pod services using service orchestration. For specific projects, G-Pod can also support interoperation and standard GRID-level interface; offering access to data resources and processing capabilities using protocols like GRAM (Job control) and GridFTP (data transfer). total on-line data: ~200 tb

7 SERvicE SuPPORt EnviROnMEnt Experiment excellence in the SSE Platform The definition and prototyping of services is made easy within the Service Support Environment (SSE), a neutrally managed, open and distributed platform. SSE offers an exclusive opportunity to integrate a wide range of heterogeneous EO and geospatial information services. SSE works to: Orchestrate synchronous and asynchronous Web Services for online and offline processes Provide an overarching infrastructure, neutrally managed by ESA Integrate the access to EO data within service prototyping and delivery Empower service providers through a business process management platform Minimise service providers upfront investments Allow the use of Open standards to facilitate adoption and evolution Facilitate the integration of satellite data and geospatial information within the processing and exploitation chains Through the SSE platform each service and data provider gets an enhanced possibility of prototyping and a notable reduction of the development risk. the control of the services remains with the owners and developers which, at the same time, retain the intellectual Property Rights and can benefit from a considerable cut on their investments in Information Technology tools for service deployment. ESA

8 SERvicE SuPPORt EnviROnMEnt SSE web portal Four simple steps to experiment and create new excellence 1 Register services on the SSE Portal to start development and testing 2 Orchestrate services 3 When ready, publish services either directly or through the Toolbox 4 Monitor service execution through a dedicated Console the SSE technology approach SSE is based on different technologies that have been integrated into a unique and powerful environment. To orchestrate the supply chain of EO and GEOSPATIAL INFORMATION, SSE provides the best available technologies and standards: SSE is based on the Service Oriented Architecture (SOA) and simplifies the integration of Web Services through the toolbox environment (accessible as a free download from the SSE portal). the service chaining is supported by an orchestration engine that provides a Standard workflow definition language. the Service output can be published in different ways exploiting a wide set of geospatial information tools and services either on the provider or on the SSE side, including an interface to the ESA DDS (Data Dissemination System) for data delivery via satellite. This approach allows the service provider to offer the functionalities either as individual Web Services or aggregated at a single interface and: Foster the development of reusable services Avoid the duplication of frequently used processing software Facilitate the combination of services into business processes.

9 OPEn AccESS OntOLOGy / terminology FOR the GMES SPAcE component DAtA AccESS With the aim to extend the use of EO applications for every day life, The Research&Service Support and Ground Segment Technology office has designed and implemented an innovative tool intended to link available EO products to semantic terms familiar to specific application domains. The tool provides a semantic-based link between application and related, available EO resources, through a shared, multi-domain thesaurus (made of a Graph and a Vocabulary, containing also synonyms and related terms) and a Taxonomy, specific to the GMES Space Component Data Access (GSCDA). This search tool is mainly a knowledge-based system, permitting a semantic access to the available EO resources. Thanks to this approach, the user can start the search inside the common Thesaurus, using the terms specific to his application domain, and, in a few clicks, identify the useful EO product(s) among those made available within the GSCDA database (classified by sensor, mission, product type and data sets). ESA

10 OPEn AccESS OntOLOGy / terminology FOR the GMES SPAcE component DAtA AccESS Ontology implemented of the GSCDA web pages how to search a term After accessing the semantic client on the GSCDA web portal, the user can search the Thesaurus using a free text option or a 2D navigator. In the free text option the first result the user gets is a hierarchical structure of the relevant terms, like in following example: Together with this semantic approach, at the bottom of the same web page, the user can find a list of related EO products (if available). When no product is displayed, the user has to refine his search, for example by selecting a term from those proposed by the system. In the 2D navigation option the user moves graphically along the visual Thesaurus. The semantic search is implemented through two Web Services, Semantic Search and Fast Domain Navigator, and three web applications. The multi-domain graph and the GSCDA taxonomy are represented in OWL (Web Ontology Language), a standard language to implement ontology knowledge-bases for handling via standard and open-source tools (e.g.: the Protégé graphical editor) and for possible analysis by logic reasoners.

11 heterogeneous MiSSiOn AccESSiBiLity In 2005 the Ground Segment Coordination Body started a collaborative project, among the European and Canadian space agencies, aimed at standardising the ground segment interfaces of the satellite missions, national and GMES Sentinels ones, for easier access to earth observation (EO) data. The goal of HMA is to define the interoperability concept and provide standards for coordinated data access enabling the interactions with services or Value Adders and EO Contributing Missions. The focus is oriented on the definition of the interfaces among the EO data Users and the different ground segments (and among the ground segments) in a well specified end to end architecture. Starting from the existing standards, when available, HMA, working side by side with the EC INSPIRE working group, has defined the discovery, catalogue ordering feasibility analysis and identity management interface standards allowing collaboration among the space agencies. HMA implements an open and distributed solution to support interoperability that allows services integration among heterogeneous infrastructures of the Ground Segment Partners. In order to manage the evolution and changes to the defined HMA standards, ESA, together with other relevant EO mission and data owners (national agencies, European institutions) has set-up the HMA Architecture Working Group (AWG, Through the Open Geospatial Consortium (OGC) the HMA working group discusses the standards to be acquired. OGC standards support interoperable solutions that "geo-enable" the Web, location-based services, and mainstream information technology, therefore enlarging the market potential for EO related applications. ESA

12 heterogeneous MiSSiOn AccESSiBiLity HMA wiki pages Operational use of hma HMA uses as a persistent test-bed, publicly available, the technology and infrastructure of the Service Support Environment ( Through the SSE platform HMA allows verifying the conformance of implementation versus the standards. The test-bed provides capabilities to validate against reference datasets and to test new operational scenarios, being an open, permanent infrastructure in which organisations or external projects (e.g. EC projects, OGC s OWS and GEOSS Interoperability Programs, etc.) can integrate their (compliant) services. The HMA standards are under configuration control and can be retrieved from the HMA Web Site (see below) at the Standards The HMA information models used to describe collections, datasets and Earth Observation products rely on well established International Organisation for Standardization (ISO) and Open GeoSpatial Consortium OGC standards: EO collection metadata: ISO Geographic Information - Metadata EO product metadata: OGC s GML Application Schema for EO Products collection and service discovery: OGC s Cataloguing of ISO Metadata using the ebrim profile of CS-W catalogue Service: OGC s Catalogue Services Specification 2.0 Extension Package for ebrim Application Profile: Earth Observation Products Order: OGC s Ordering Services for Earth Observation Products Feasibility Analysis: OGC s Sensor Planning Service Application Profile for EO Sensors Online Data Access: OGC s WMS EO Extension identity (user) Management: OGC s User Management Interfaces for Earth Observation Services

13 KnOWLEDGE-cEntRED EARth OBSERvAtiOn (KEO) SyStEM Processing chain handling Interactive KEO use Image Information Mining is one of the key issue to empower the user (e.g.: researcher, service provider, decision maker) to identify relevant information from Earth Observation (EO) data by interactive or batch processing of images. As a step in the above direction, ESA started the project Knowledge-centred Earth Observation (KEO), to support more automated extraction of information from EO images. The KEO system aims at expanding the use of Earth Observation (EO) data by: Encouraging the use of a common scientific cooperative environment Supporting automation of identification and extraction of user relevant information Providing a large set of EO data processing tools (bridge the Data / Information gap) Easing the access to the information extracted from EO data In summary, KEO allows an expert to identify available components suitable for his application, to create new components mainly aimed at the extraction of information from images, to chain all these components and to run the complete chain in order to extract information from the data. KEO functionalities All KEO system functionalities for EO data processing are provided through a unique Graphical User Interface, the KAOS Client Application. Processing Components can be either Software Modules or Feature Extraction Processors (FEPs). The Software Modules, deployed on KEO or user machines, can be: Written in Java Written in any programming language but wrapped by a Command Line Interface (CLI) Provided as Web Services (WS) The Software Modules can be executed within the CPE only if embedded into FEPs. A FEP can include one or more Software Modules and / or combinations of other FEPs. The CPE core is a FEP Engine, which activates FEPs via centralised or remote FEP Actuators according to the way in which they were chained using KAOS. ESA

14 KnOWLEDGE-cEntRED EARth OBSERvAtiOn (KEO) SyStEM The CPE provides a large number of Processing Components (currently about 300) for: Calibration and Classification of single images Objects / Features Detection from single images Signal Processing Inter-equalisation and co-registration of time series of images Change Detection and Hot Spot Monitoring Basic processing (format conversion, segmentation, etc.) KEO sub-systems KEO is a complex system made of a few sub-systems: The Knowledge-based Information Mining (KIM) sub-system, for interactive exploration of collections of EO images through a Probabilistic Information Mining concept The Component-based Processing Environment (CPE), for distributed processing and graphic programming via Processing Components (e.g.: for automating information extraction) The KEO Application Operating on Services (KAOS), the user client of above sub-systems KiM KIM is based on Primitive Features like: Spectral signature Texture information Geometric parameters Discrete Cosine Transform Semantic pre-classification After the interactive training, KIM permits to explore image collections and export the extracted information (features and maps) also for reuse in OGC Web Servers cpe The distributed CPE permits to: Create and semantically identify internal or external Processing Components Use Processing Components provided by the system or added by the user Graphically chain Processing Components into more complex Processing Chains Work with EO data coming from different sources

15 MuLti-SEnSOR EvOLutiOn AnALySiS: MEA AATSR image and related swath and re-projected classification map Dynamic phenomena may have relevant effects on environment and therefore on mankind and can leave traces on land cover changes, detectable from EO data. In order to boost automation in the identification of features with dynamic behaviour, ESA has just implemented a prototype of a multi-temporal analysis solution, called MEA. Its aim is to permit the definition and use of Evolution Models, which describe how to detect a specific feature from related land cover changes over time. In order to reach this objective, it is necessary to create a common classification system and an Earth Fixed Geo-spatial reference, also applicable across different missions. MEA was studied to help the user to: Define time Evolution Models of features Apply Evolution Models across 13 years of (A)ATSR data Obtain in a few seconds the results of related spatio-temporal and semantic queries. In addition MEA will be used to test the applicability of Evolution Models across different sensors with different resolutions: (A)ATSR at about 1Km AVNIR 2 at about 10 m. The SOIL MAPPER is a suitable solution for optical sensors, since it provides fast, automatic classification of single pixels into classes like: Vegetation Bare soil / Built-up Snow / Ice Clouds Water / Shadows Thanks to this classification SOIL MAPPER assigns also a semantic meaning to each pixel. ESA 2010

16 MuLti-SEnSOR EvOLutiOn AnALySiS: MEA To allow a multi-temporal analysis across sensors with different resolutions, MEA is based on a common grid on the Earth made of latitude/ longitude aligned Tiles grouping always 64*64 grid elements of variable resolutions. The user, when applying an Evolution Model in his time and region of interest, obtains one or more tiles over the Earth map, with pixels tagged with a colour to indicate how they match the search parameters of the Evolution Model. Currently the Tiles dimensions are selected as follows: ATSR: 0.25 x 0.25 (4 Tiles x degree), with pixel size ~ 434 m at Equator AVNIR 2: 0.23 x 0.23 (256 Tiles x degree), with pixel size ~ 5 m at Equator The implementation for 13 years of (A)ATSR data, even using only data over land and with minimal cloud coverage, results in over 300 million Tiles, posing huge issues for applying the Evolution Models through searches in the space, time and semantic dimensions within a few seconds of response time. When the images are remapped on the same Earth fixed grid, MEA permits an expert to define Evolution Models (changes in land cover classes) for periodic features or for transitions, as well as the tolerances over searched classes and time transitions. The searches are performed over the area of interest comparing changes in the single pixels, over time. In the next image, for example, the transition from snow to not-snow is tagged with the following colours: Green: perfect match Yellow: within tolerance Red: no such change Black: no data