Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 1 Processing and dissemination of satellite remote sensing data in an heterogeneous environment. Armin Burger, Paul Hasenohr armin.burger@jrc.it, paul.hasenohr@jrc.it Community Image Data portal project http://cid.jrc.it
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 2 Presentation structure Project s objectives A few words about the Institution Step 1: from a tape repository to a digital repository Step 2: extending this digital repository by building services around it Conclusion
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 3 European Commission (EC) The Commission is independent of national governments. Its job is to represent and uphold the interests of the European Union (EU) as a whole. It drafts proposals for new European laws, which it presents to the European Parliament and the Council. It is also the EU s executive arm in other words, it is responsible for implementing the decisions of Parliament and the Council.
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 4 Joint Research Centre (JRC) The European Commission s Research-Based Policy Support Centre The JRC provides customer-driven scientific and technical support to Community policy making.
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 5 Use of satellite imagery at the JRC Twenty projects using Satellite Remote Sensing (SRS) data Annual expenditure on SRS data > 7 M 100 000 s of Low Resolution images More than 12 000 High Resolution images Over 500 000 km 2 of VHR images Overall, more than 55 TB of SRS data Scattered storage and no central catalogue
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 6 First setup of an online archive Initiated on the archive of one project end 2003. MARS-PAC project is the largest civil purchaser of satellite imagery in the EU. Largest archive of HR and VHR data in the JRC Goal: To be able to search for images from a web browser, to visualize them and get a link to the files.
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 7 Characteristics of this first setup SRS data loading application Works with raw HR and VHR data Extracts metadata and stores them in a spatial DB (PostgreSQL/PostGIS) SRS data pre-processing Conversion of the data to TIFF format and georeferencing. Histogram stretching and conversion to ECW SRS data dissemination Search and browse imagery (ECW) Web mapping (Mapserver) Download of raw or georeferenced data
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 8 Outcome of this first setup Successful as initial objectives were reached Several limitations were identified: disk space not easily extendable restricted to raw HR and VHR no fine grained authentication possible (users and IPRs) no WMS or WCS service severe restrictions in usability by other projects at JRC Management wish to build on it and provide a site-wide service Expected cost savings in hardware and staff
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 9 Survey of potential users Initiative welcomed!!! Community Image Data portal project started!
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 10 CID portal project objectives Provide a centralized storage facility with backup and a data upload application. Provide a data acquisition service. Provide a service with IT infrastructure in High Availability mode. Build a central catalogue of all SRS data available at JRC with a CSW service and a web interface to the catalogue. Provide secure access according to IPRs and user credentials via direct file access at high speed (NFS v4, CIFS) OGC protocols (WMS, WCS) other (web mapping, KML...) Provide some SRS data processing services: [semi-]automated orthorectification
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 11 CID portal Staff and Budget Budget to initiate the project: Hardware: 200 K Software development: 130 K Orthorectification software Modules for authentication and catalogue interface Annual resources: IT consultancy and small developments: 50 K 1.5 engineers and 0.7 IT technician.
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 12 CID portal overview JRC SRS data Data Data Owner acquisition Provider JRC staff SRS data loading AUTH. service AUTH. service NFS4 CIFS DATA storage METADATA database SRS data processing CSW User WMS KML WCS Actor CID data serving service CID spec. service
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 13 CID services overview AUTH: Authentication and Authorization service SRS data acquisition: web application for SRS data ordering SRS data loading: website, web service and desktop interface SRS data processing: metadata extraction, registration in a central catalogue and orthorectification for HR and VHR data. CID data serving: direct file access and OGC services
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 14 Open source software and CID The preferred option at CID because: Knowledge available within CID staff Almost all functionalities required are available in FOSS Some functionalities available only in FOSS! Good support from the user community No cost to acquire the piece of software
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 15 Base infrastructure OS: Xen with Debian 4 (80%) and Windows 2003 server Base services: Bind, NTP, MIT Kerberos, LVS Monitoring and statistics: Zenoss, Awstats, JMeter
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 16 Authentication and Authorization Service AUTH. service Extremely important service: Enforcement of Intellectual Property Rights restrictions Sensitivity of some datasets (e.g. crisis areas) Technologies used: OpenLDAP with Kerberos NFS v4 and CIFS with ACLs Central Authentication Service (CAS) PostgreSQL with PostGIS Custom authorization application interfacing with OGC services (PHP/Drupal framework)
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 17 SRS data acquisition service SRS data loading Web based application interfacing with 6 SRS data providers for 9 missions. Technologies used: DotNetNuke framework
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 18 SRS data processing ortho node 1 JBoss cluster ortho node 2 ortho node x Digital Elevation Model SRS data processing raw image rectification process final ortho image
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 19 CID Data serving service CID data serving service Web mapping [p.mapper, CAS] Catalogue Web interface [PHP/Mapscript] CSW [GeoNetwork] WMS, WCS [PHP/Mapscript, CAS] Direct file access [NFS, CIFS] Download [PureFTPd] Official CID website for reference information [typo3]
Victoria, BC, Canada on 25 September 2007 FOSS4G2007 Conference 20 In the coming months... By end 2007: All backend and base services up and running in HA Processing service placed in its production environment Completion of the web portal Q1 / 2008: OGC services implemented SRS data loading service implemented SRS data acquisition service moved to the new environment First import of data and intense testing Q2 / 2008: Import of other datasets Public access to the CID portal (http://cid.jrc.it)