TELEIOS FP Deliverable 3.1. KDD concepts and methods proposal: report & design recommendations

Transcription

1 TELEIOS FP Deliverable 3.1 KDD concepts and methods proposal: report & design recommendations Corneliu Octavian Dumitru, Daniela Espinoza Molina, Shiyong Cui, Jagmal Singh, Marco Quartulli 1, Mihai Datcu Status: Final Scheduled Delivery Date:31/08/2011 and Consortium members 31/08/ Vicomtech on behalf of DLR

2 Executive Summary This deliverable presents a first draft of the concepts, and their demonstration and evaluation for Knowledge Discovery in Databases (KDD). The Knowledge Discovery components are the data sources composed of Earth Observation images, metadata and GIS information, the Data Model Generation involving the content extraction and context analysis of the data sources, Query, Data mining and Knowledge discovery functions implementing the information exploration and queries based on example, Interpretation and image understanding, supporting the rapid mapping applications, and Visual Data Mining allowing visual exploration of the database content and interacting with User through a Human Machine Communication. First, we present the Earth Observation (EO) data sources by concentrating in TerraSAR-X images and the XML annotation file with the metadata. After a review of the different TerraSAR-X products, we present the Data Model Generation for EO data sources. Here, we start with an overview of the components to take into account during the data model generation, later we explain in detail the content analysis, which is based on feature extraction methods. We also describe the relevant metadata to be extracted from the xml annotation file, which will support the RDF queries. We propose a conceptual design for TerraSAR-X data model and relation storage for the descriptor database, which will be implemented in MonetDB. We describe two examples of a system that implements the concept of query by example and the semantic definition by learning methods. The first one corresponds to a Search Engine whose main core is a Support Vector Machine (SVM) classifier. This tool relies on 1) feature extraction methods providing the most relevant descriptors of the images, 2) SVM as classifier grouping the image descriptors into generic classes, and 3) relevance feedback interacting with the end user. The second one follows a parameter-free approach and corresponds to Compression Based Image Retrieval. This tool uses compression techniques based on dictionary extraction, which is considered as descriptor of the image content, and the Fast Compression Distance, which is a similarity metric for retrieving the relevant images. This metric is directly implemented in MonetDB. This new approach was tested with TerraSAR-X images and opens a new perspective of simple and fast implementation for the query based on image content as example. Finally, we evaluate the feature extraction methods using several thousand patches from three types of TerraSAR-X products. The best results have been obtained with Gabor filters. Moreover, we define 35 semantic classes, which are described in an ontology frame for further use in RDF queries. D3.1 KDD concepts and methods proposal: report & design recommendations i

3 Document Information Contract Number FP Acronym TELEIOS Full title Project URL EU Project officer Virtual Observatory Infrastructure for Earth Observation Data Francesco Barbato Deliverable Number D3.1 Name KDD concepts and methods proposal: Report & design recommendations Work package Number WP3 Date of delivery Contractual M12(Sept 2011) Actual 31 August 2011 Status draft final Nature Distribution Type Authoring Partner QA Partner Contact Person Prototype Report Public Restricted Consortium DLR ACS Prof. Mihai Datcu Phone Fax D3.1 KDD concepts and methods proposal: report & design recommendations ii

4 Project Information This document is part of a research project funded by the IST Programme of the Commission of the European Communities as project number FP The Beneficiaries in this project are: Partner Acronym Contact National and Kapodistrian University of Athens Department of Informatics and Telecommunications (Coordinator) Fraunhofer Institute for Computer Graphics Research German Aerospace Center The Remote Sensing Technology Institute Photogrammetry and Image Analysis Department Image Analysis Team NKUA Fraunhofer DLR Prof. Manolis Koubarakis National and Kapodistrian University of Athens Dept. of Informatics and Telecommunications Panepistimiopolis, Ilissia, GR Athens, Greece koubarak@di.uoa.gr Tel: , Fax: MSc. Thorsten Reitz Fraunhofer Institute for Computer Graphics Research Fraunhofer Strasse 5, D Darmstadt, Germany thorsten.reitz@igd.fraunhofer.de Tel: , Fax: Prof. Mihai Datcu German Aerospace Center The Remote Sensing Technology Institute Oberpfaffenhofen, D Wessling, Germany mihai.datcu@dlr.de Tel: , Fax: Stichting Centrum voor Wiskunde en Informatica Database Architecture Group National Observatory of Athens Institute for Space Applications and Remote Sensing Advanced Computer Systems A.C.S S.p.A CWI NOA ACS Prof. Martin Kersten Stichting Centrum voor Wiskunde en Informatica P.O. Box 94097, NL-1090 GB Amsterdam, Netherlands martin.kersten@cwi.nl Tel: , Fax: Dr. Charalambos Kontoes National Observatory of Athens Institute for Space Applications and Remote Sensing Vas. Pavlou & I. Metaxa, Penteli, GR Athens, Greece kontoes@space.noa.gr Tel: , Fax: Mr. Ugo Di Giammatteo Advanced Computer Systems A.C.S S.p.A Via Della Bufalotta 378, RM Rome, Italy udig@acsys.it Tel: , Fax: D3.1 KDD concepts and methods proposal: report & design recommendations iii

5 Table of Contents 1. Introduction Knowledge Discovery components Contribution of this deliverable Structure of this deliverable Data Sources Earth Observation TerraSAR-X Product description TerraSAR-X image data The GeoTiff format for TerraSAR-X detected data The special multi-resolution TerraSAR-X product Selected TerraSAR-X datasets for this deliverable TerraSAR-X metadata file Summary Data Model Generation Extraction of TerraSAR-X image content Patch generation Patch identification Patch generation for detected data Patch generation for Image Time Series Feature Extraction Methods Feature identification Feature extraction methods based on texture and spectral analysis Compression method based on dictionary extraction Feature extraction for Image Time Series Quick-look generation for detected data and image time series TerraSAR-X XML file content D3.1 KDD concepts and methods proposal: report & design recommendations iv

6 TerraSAR-X XML metadata used for querying TerraSAR-X XML metadata used for patch generation Summary A conceptual Model and Relational storage Schema for the descriptor Database Conceptual design of TerraSAR-X data model Database Schema for image descriptors Description of the tables Image Xmlproduct Patch Label Features_glcm Features_qmfs Features_nstf Features_gafs Dictionary Summary Query by example and active learning methods for implementing knowledge discovery functions Search engine based on Support Vector Machine and relevance feedback Description of Search Engine based on SVM and relevance feedback Evaluation method for TerraSAR-X detected data Results of TerraSAR-X detected data classification Discussion of the performance results Evaluation method for Image Time Series Compression-based Image Retrieval System using similarity metrics Concept and Logical architecture of CBIR based on FCD D3.1 KDD concepts and methods proposal: report & design recommendations 5

7 Implementation and operation of CBIR based on FCD Experimental results of CBIR based on FCD Experimental results using TerraSAR-X images Summary Conclusions Applicable and Reference Documents Applicable Documents Reference Documents Other References Acronyms and Abbreviations Appendix Dataset structure for TerraSAR-X detected data Metadata product extracted from the TerraSAR-X XML file Precision Recall results for TerraSAR-X detected data D3.1 KDD concepts and methods proposal: report & design recommendations 6

8 1. Introduction The WP3 of TELEIOS deals with knowledge discovery from Earth-Observation (EO) images, related geospatial data sources and their associated metadata, mapping the extracted low-level data descriptors into semantic classes and symbolic representations, and providing an interactive method for efficient image information mining. WP3 addresses three important tasks: Task 3.1: Knowledge discovery from EO images and related Geographic Information Systems (GIS). Task 3.2: Semi-supervised learning methods for spatio-temporal and contextual pattern discovery Task 3.3: Human Machine Interaction (HMI) techniques for image information mining Task 3.1 focuses on the design and implementation of methods for the extraction of relevant descriptors (features) of EO images, specifically TerraSAR-X images, physical integration (fusion) and combined usage of raster images and vector data in synergy with existing metadata. The extracted content is captured and accessed using the new models for temporal and spatial extensions of RDF considered in WP2 and Knowledge discovery. Task 3.2 concentrates on investigating new semi-supervised learning methods to cope with heterogeneous spatial-temporal image data and to take into account contextual information. The methods combine labeled and unlabeled data during training to improve the performance of classification and the generation of categories (number of classes). The methods are applied jointly to raster, vector and text data, for the definition of semantic categories. Task 3.3 focuses on designing and elaborating HMI techniques with the users in the loop to optimize the visual interaction with huge volumes of data of heterogeneous nature. Since human perception is limited in communication capacity, the HMI paradigm is supported by special methods that increase the information being transmitted. We will elaborate dictionaries and visual ontology support to explain the image content: as for example translation of the vocabulary used in GIS sources, i.e. signs and symbols into an image vocabulary, at semantic descriptive level. These tasks may be better understood in the context of Figure 1, which introduces the framework of Knowledge Discovery in WP3 that is explained in the following section. D3.1 KDD concepts and methods proposal: report & design recommendations 7

9 1.1. Knowledge Discovery components Query, Data Mining & Knowledge Discovery Data Sources Data Model Generation DBMS Visual Data Mining Users Interpretation Understanding Figure 1: Components of Knowledge Discovery in a Database. TELEIOS intends to implement a communication channel between the EO Data Sources and the User (Operator) who receives the content of the Data Sources coded in an understandable format associated with semantics (see Figure 1). The Data Sources are TerraSAR-X images and their associated metadata (i.e. acquisition time, incidence angles, etc), and auxiliary data in vector format coming from GIS sources that complement the information about the images, for instance, park boundaries, city boundaries or land uses represented as polygons. The Data Model Generation focuses on a content and context analysis of the different data sources. The image content analysis provides different feature extraction methods, which are dealing with the specificities of TerraSAR-X images in order to represent the relevant and correct information contained in the images known as descriptors. The image descriptors are complemented with image metadata (text information) and GIS data (vector data). It is important to note that the efficiency of the query data mining and knowledge discovery depends on the robustness and accuracy of the image descriptors. The Data Model will be stored into a Database Management System (DBMS), which can act as the core of the Knowledge Discovery in Databases (KDD) and to support the Data Mining functions and RDF queries (WP2). The Operator (User) requires visual information that is intuitive, contrary to raw images such as TerraSAR-X images that feature information such as forests, water bodies, etc, as different grey levels. However, combining the image content with semantics, text descriptions, etc, the operator can better understand the content of the image and D3.1 KDD concepts and methods proposal: report & design recommendations 8

10 perform queries over collections of images easily. Thus, the semantics and ontologies support the image content. For instance, observing annotations of forest regions in Brazil or flooding areas in Nepal is more understandable than observing grey levels in a TerraSAR-X image. The joint use of the three types of information (image content, text, and vector data) will allow the operator performing queries as follows: 1) Queries based on the image content that can be seen as Visual Data Mining. 2) Queries based on web semantics and ontologies. The TELEIOS component Query, Data Mining and Knowledge Discovery requires integrating in 1) image processing and pattern recognition for understanding and extracting useful patterns from a single image, 2) spatial data mining for identifying implicit spatial relationships between images, and 3) content based retrieval of images from the archive based on their semantic and visual content. These types of techniques are used to discover knowledge from the EO data sources. Therefore, knowledge discovery from EO images is supported by concepts, tools, algorithms and applications for extracting and grouping relevant image descriptors, combining them with text and vector information, and analyzing the content and context in spatio-temporal relationships. The Data Model Generation will support these components. The Interpretation and Understanding component, with the interaction of the Operator, attempts to extract information from the Database Manager System in order to produce topological, spatial and temporal analysis by combining the relevant information stored into the Database. The Visual Data Mining component allows interactive exploration and analysis of very large, high complexity, and non-visual data sets stored into the database. It provides to the operator an intuitive tool for Data Mining by presenting a graphical interface, where the selection of different images in 2-D or 3D space is achieved through visualization techniques, data reduction methods, and similarity metrics to group the images. Thus, Visual Data Mining links the DBMS with the real user applications. Actually, Visual Data Mining relies on powerful Human-Machine Communication (HMC or HMI) and Graphical User Interfaces (GUI) with functionalities such as browsing, querying, zooming, mining, etc, which enable us to search in the EO database, to adapt to operator conjectures and to retrieve the results. The TELEIOS modules implementing the Query, Data Mining & Knowledge Discovery, Visual Data Mining, and Interpretation and Understanding modules process the Earth-Observation Data Model in interaction with the User, update it, and generate as output the information required by the user in a format adapted to the application context. As a conclusion, TELEIOS modules can help the user to deal with large image collections by accessing and extracting automatically their content (Data Model Generation), allowing querying (by means of image content and semantics) and mining relevant information (Query Data Mining and Knowledge Discovery), inferring knowledge about patterns hidden in the images (Interpretation and Understanding), and visualizing the results (Visual Data Mining). D3.1 KDD concepts and methods proposal: report & design recommendations 9

11 In the framework of WP3, the Task 3.1 is covered by the Data Model Generation and analysis of the Data sources. Task 3.2 is linked to Query Data Mining & Knowledge Discovery and Task 3.3 is related to Visual Data Mining, and Interpretation and Image Understanding. The different KDD components are related to the TELEIOS architecture presented in [RI-28] as shown in Figure 2. Here, the Ingestion Facility performs the metadata and feature extraction, and semantic ontology definition from the data sources, which correspond to the Data Model Generation process regarding Task 3.1 in WP3. The next level, DMBS refers to a common repository where all the information (descriptors, catalogue, etc) will be stored. The Services Processing Facility (SPF) level involves the data mining, knowledge discovery and visual mining by using a Human Machine Interaction (MHI) related with Task 3.3. Figure 2: TELEIOS functional second level decomposition diagram. Taken from ACS D1.2.1 [RI-28]. D3.1 KDD concepts and methods proposal: report & design recommendations 10

12 1.2. Contribution of this deliverable This deliverable mainly concentrates on Task 3.1 and Task 3.2 and makes the following contributions: We present the Knowledge discovery framework in Earth-Observation (EO) data sources, which is founded on the understanding of data sources -the image content, metadata and GIS-, the extraction of the right data sources descriptors, the identification of the associated metadata and GIS sources and the translation in the design of Data Model Generation for supporting RDF queries and KDD. We assess and validate the quality of the image descriptors and the feature extraction methods in order to exploit at best the high resolution of the TerraSAR-X images. The evaluated methods we used are Gray-Level Cooccurrence Matrix (GLCM), Nonlinear Short Time Fourier Transform (NSFT), Gabor Filters (GAFS), Quadrature Mirror Filters (QMFS) using TerraSAR-X detected data; and Local Pattern Histogram (LPHM) in the case of Image time series. In this point, we carried out our evaluation with a new approach, which is based on patches. In previous approaches, these methods were used as pixelbased methods. However, in our approach the whole image is tiled into several patches and the methods are applied to them, meaning a speed increase for data mining. We present the results of some experimental work we carried out using two TerraSAR-X basic products: Venice and Toulouse, Multi-look Ground range Detected (MGD) with Spatially and Radiometrically Enhanced [RD-1], image time series corresponding to Geo-coded Ellipsoid Corrected (GEC) TerraSAR-X basic product over Vâlcea; and multi-resolution product (1 to 8 meters of resolution) in order to derive the preliminary conclusions of this work. The experimental work was performed by means of a deep evaluation of feature extraction methods using different resolutions, patch sizes and image products. This evolution leads us to provide recommendations about the usage of the methods according to the type of TerraSAR-X product and the optimal patch size, than can be further implemented in real tools for knowledge discovery and automatic information extraction. In the context of Task 3.2, we introduce a first component for Human-Machine Interaction (HMI) in the framework of query data mining and knowledge discovery in large SAR imagery by presenting two software implementations: the first one corresponds to a Search Engine, which main core is the Support Vector Machine (SVM) classifier. This tool relies on 1) feature extraction methods providing the most relevant descriptors of the images, 2) SVM as classifier grouping the image descriptors, and 3) relevance feedback interacting with the end user. The second one corresponds to Content Based Image Retrieval (CBIR) based on the Fast Compression Distance, which is implemented using compression based techniques in order to describe the image D3.1 KDD concepts and methods proposal: report & design recommendations 11

13 content and distance measures in order to discover similar image content in the database Structure of this deliverable Chapter 1 of this document contains the tasks of WP3, presents the Knowledge Discovery concept connected with TELEIOS architecture, shows the contribution of this deliverable, and also contains the present section that gives an overview on the document structure. Chapter 2 defines a part of the data sources to be used in TELEIOS project and technically presents in detail the EO TerraSAR-X products (the EO TerraSAR-X image and EO TerraSAR-X metadata XML file) that were used in our experiments. Chapter 3 gives the details about the Data Model Generation (DMG) and its components: the TSX image content extraction and the TSX XML metadata file content. This chapter also sets the modality of tiling the TSX images and processing the features, and provides an overview on the content of the XML file. Chapter 4 provides a description of the conceptual design of the database based on an entity-relationship model as well as a detailed description of the database schema and the tables it comprises. Chapter 5 contains a description of a content based image retrieval system based on: compression and/or feature extraction techniques as the first example of Knowledge Discovery in Databases user applications. Chapter 6 presents the conclusion of this deliverable. Chapters 7 and 8 are dedicated to the applicable and reference documents and the list of acronyms and abbreviations. Chapter 9 presents the dataset structure used for tilling and extracting the features for detected data, the TSX metadata extracted for tiling the images, and the results (as tables) obtained for detected data. D3.1 KDD concepts and methods proposal: report & design recommendations 12

14 2. Data Sources The Data Sources are Earth Observation (EO) images with their associated metadata (header files) and vector information coming from a GIS (Geographic Information System). In this chapter, we discuss two of the three data sources to be used in TELEIOS project, namely EO TerraSAR-X (TSX) image and EO TerraSAR-X XML file as metadata. The third data source, GIS data, will be included during the second year of the project. First, we present in detail the EO TerraSAR-X product with its structure, later we mention the different types of TerraSAR-X images and the associated metadata Earth Observation TerraSAR X Product description TerraSAR-X is the German radar satellite launched on June It operates in the X- band and is a side-looking Synthetic Aperture Radar (SAR) based on active phased array antenna technology. It does supply high quality radar data for purposes of scientific observation of the Earth [RD-1]. The term Level 1b product stands for TerraSAR-X basic products, which are the operational products offered by the TerraSAR-X payload ground segment (PGS) to commercial and scientific customers. These products can be ordered through and will be delivered by the PGS user services at DLR. They are generated by the TerraSAR Multi Mode SAR Processor (TMSP) [RD-2]. The huge variety of Level 1b product [RD-2] types for TerraSAR-X (complex, detected, geocoded, etc) requires product annotation in an extensible and dynamic format. The XML format has been selected for this purpose (structure shown in Table 2). The Level 1b products are usually packed in a delivery package. It is supplemented by additional information and either archived into a.tar file or put onto a medium. Figure 3 gives an example of the directory structure inside such a delivery shell and Table 1 presents an overview of the relevant notations [RD-2] of the directory product name. D3.1 KDD concepts and methods proposal: report & design recommendations 13

15 Table 1: Filename constituents [RD-2] of the TerraSAR-X product. The TerraSAR-X product annotation component contains all the basic information on the delivered product as uniform as possible for all product types. Details can be found in [RD-1]. In here, it is important to mention that in the scope of this deliverable only the image data (TerraSAR-X image) and the annotation file (XML file) are considered. In the following, we start by presenting the different types of TerraSAR-X images in subsection Later, the annotation file containing the image metadata is presented in subsection D3.1 KDD concepts and methods proposal: report & design recommendations 14

16 Figure 3: Directory structure and files of the TerraSAR-X Level 1b product [RD-2] D3.1 KDD concepts and methods proposal: report & design recommendations 15

17 TerraSAR X image data In the context of TerraSAR-X payload ground segment, the SAR raw data are processed to basic products by the TerraSAR-X TMSP, which generates a set of images depending on the geometric and radiometric resolution as well as on the geometric projection and data representation. Figure 4 summarizes the TerraSAR-X basic products. The geometric resolution reduction offers two possible configurations: Spatially Enhanced Products (SE) or Radiometrically Enhanced Products (RE). The Spatially Enhanced (SE) product [RD-1] is designed for the highest possible ground resolution. Depending on imaging mode, polarization and incidence angle the larger resolution value of azimuth or ground range determines the pixel size. The Radiometrically Enhanced (RE) product [RD-1] is optimized with respect to radiometry. The range and azimuth resolution are intentionally decreased to significantly reduce speckle 2 by averaging approximately 6 (5 to 7) looks to obtain a radiometric resolution of about 1.5 db. The Signal-to-noise ratio (SNR) that generally decreases with larger incidence angles is also considered assuming a backscatter of 6 db at 20 and -12 db at 50. Because of the lower resolution, the required pixel spacing can be reduced and the product data size decreases significantly. Besides the different geometric resolutions, TerraSAR-X also allows a selection among different geometrical projections and data representations as follows: Single look Slant range (complex-valued) (SSC) Multi-look Ground range Detected (MGD) Geo-coded Ellipsoid Corrected (GEC) Enhanced Ellipsoid Corrected (EEC) The Single look Slant range (SSC) product [RD-1] is the basic single look complex product of the focused radar signal. The data are represented as complex numbers and the pixels are spaced equidistant in azimuth and in slant range. The SSC product is intended for scientific applications that require the full bandwidth and the phase information, e.g. SAR interferometry and interferometric polarimetry. The Multi-look Ground range Detected (MGD) product [RD-1] is a detected multi-look product with reduced speckle and equal resolution cells on ground. The image coordinates are oriented along flight direction and along ground range. The pixel spacing is equidistant in azimuth and in ground range. A simple polynomial slant to ground projection is performed in range using a WGS84 ellipsoid and an average, constant terrain height parameter. The advantage of this product is that no image rotation to a map coordinate system is performed and interpolation artefacts are thus 2 Speckle is considered as noise-like effect that affect the SAR images D3.1 KDD concepts and methods proposal: report & design recommendations 16

18 avoided. Consequently, the pixel localization accuracy is lower than in geo-coded products. For MGD product [RD-1], a coarse grid of coordinates is annotated in the product. The grid coordinates are calculated using a coarse Digital Elevation Model (DEM), while the projection of the image data is performed using an ellipsoid with one elevation determined for the scene. The Geo-coded Ellipsoid Corrected (GEC) product is a multi-look detected product projected and re-sampled to the WGS84 reference ellipsoid assuming one average terrain height. Available grid formats are UTM (Universal Transverse Mercator) and UPS (Universal Polar Stereograhic). As the ellipsoid correction does not consider a DEM, the pixel location accuracy varies due to the terrain. For other types of relief, the terrain induced SAR specific distortions will not be corrected and significant differences can appear in particular for strong relief and steep incidence angles. The geometric projection is map geometry with ellipsoidal corrections only (no terrain correction performed) [RD-1]. The Enhanced Ellipsoid Corrected (EEC) product is a multi look detected product. It is projected and re-sampled to the WGS84 reference ellipsoid. The image distortions caused by varying terrain height are corrected using an external DEM. Available grid formats will be either UTM or UPS. The pixel localization in these products is highly accurate. However, the accuracy still depends on the type of terrain as well as the quality and resolution of the DEM and on the incidence angle. The geometric projection is map geometry with terrain correction, using a digital elevation model (DEM). MGD and Geo-coded products (GEC and EEC) are TerraSAR-X detected products while SSC is a TerraSAR-X complex product. In the case of detected products, the image data consists of one or more polarimetric channel files in separate binary data matrices, which are stored as individual GeoTIFF files. In complex products (SSCs), the individual bursts of each ScanSAR beam are stored together in one individual binary file for each beam [RD-1]. D3.1 KDD concepts and methods proposal: report & design recommendations 17

19 Level 1b product SSC MGD GEC EEC RE SE RE SE RE SE Special multi-resolution TSX product Figure 4: TerraSAR-X basic products and special multi-resolution products [R0-1] The GeoTiff format for TerraSAR X detected data The individual polarization layers of the image data of projected products are given as separate files in the GeoTIFF file format in unsigned 16 bit representation and a subset of commonly used tags [RD-2]. GeoTIFF is an extension of the TIFF (Tagged Image File Format) standard which defines additional tags concerning map projection information [RO-2]. Large files which would exceed the 4GB limit are compressed using the standard TIFF packbits algorithm. The GeoTIFF format version 1, key revision 1.0 as specified in [RI-2] with a very limited number of tags and keys is used for the detected and projected image data. The projection tags and GeoTIFF keys set by the TMSP. GeoTIFFs main information, the transformation of the raster coordinate system to the target model coordinate system, is given by a 4 x 4 transformation matrix which can be evaluated by every standard GeoTIFF reader. The result is referenced to WGS84. UTM zones and UPS projection are annotated [RD-3]. Figure 5 and Figure 6 show the projection of the image on Google Earth and the composite quick-look of Venice (Italy) and Toulouse (France) available in the TSX package delivery of MGD product. Likewise Figure 7 displays Vâlcea (Romania) area in the case of GEC product. D3.1 KDD concepts and methods proposal: report & design recommendations 18

20 Figure 5: Overlay on Google earth, location of the test site, and quick-look - Venice. D3.1 KDD concepts and methods proposal: report & design recommendations 19

21 Figure 6: Overlay on Google earth, location of the test site, and quick-look - Toulouse. D3.1 KDD concepts and methods proposal: report & design recommendations 20

22 Figure 7: Overlay on Google earth, location of the test site, and quick-look - Vâlcea. D3.1 KDD concepts and methods proposal: report & design recommendations 21

23 The special multi resolution TerraSAR X product We called special multi-resolution product a set of images with different levels of resolution that were created from a MGD with SE standard product. It is important to mention that the idea behind this special processing is that the MGD-SE product is sampled at pixel spacing (rowspacing units / columnspacing units) equal to resolution (groundrangeresolution) in order to have uncorrelated speckle [RI-24]. The generation process is described as follows: The generation of a MGD product is performed in the following way: oversampling and detection followed by low pass filtering with inherent ground range projection and resampling. Input to the MGD generation is a single look slant range complex image being focused by the multi-mode SAR correlator module of the TMSP. The transition from coherent complex data to incoherent intensity data comes along with a doubled extent of the azimuth and range spectra, respectively. Thus, prior to detection the sampling of complex data is adequately increased in order to avoid aliasing of the detected data. The multi-looking step within the TMSP is implemented as a timedomain convolution of incoherent image intensity values and a low pass filter kernel. This is an alternative realization compared to the classical approach where individual looks are created as sub-bands in the spectral domain and transformed back to time domain followed by incoherent look summation. The time domain multi-look filter serves as well as interpolation kernel required for the SSC to MGD projection and resampling. One particular range and one particular azimuth filter are used for processing of a given scene, thus the number of looks is constant, while the ground range resolution varies to a small extent with range. In order to support at least two different classes of applications, one based on high resolution coming along with almost only one look, and the other one based on a higher number of radiometric looks. In the case of the SE variant the filter parameter selection aims at the best possible quadratic (ground-) resolution for a given acquisition. In the RE case the goal is to obtain a higher equivalent number of looks (ENL). In case of the SE variant the best possible quadratic resolution is limited by the ground range resolution for step incidence angles below 32. For the very step incidence angles excessive azimuth bandwidth is turned into a slightly increased ENL. For incidence angles flatter then 32 the resolution and ENL are almost constant in the order of 1.1m and 1.0 to 1.2 ENL. In case of the RE variant the resolution is limited to 4m to 3m. Thus the achievable ENL is in the order of 5 (at 20 ) and 8 (at 45 ) [RD-1] Selected TerraSAR X datasets for this deliverable In this deliverable we considered to work with TerraSAR-X Detected data (DET), Image Time Series (ITS), and special multi-resolution product. D3.1 KDD concepts and methods proposal: report & design recommendations 22

24 In the case of detected products, we selected two MGD TerraSAR-X basic products [RO-1], first one over Venice and the second one over Toulouse, with RE and SE, marked with green colour in Figure 4. In the case of Image Time Series, we used GEC standard product with RE, marked with light orange colour in Figure 4. In the case of special Multi-resolution, we chose a MGD product with SE. The Multi-resolution product was created using a special processing chain in order to have products with different level of resolutions. In our case, we selected 1, 2, and 4 meters of resolution. In the case of detected products, we organize the test datasets as follows: Dataset 1: MGD products with SE; Dataset 2: MGD products with RE; Dataset 3: MGD product with SE at 1m resolution Dataset 4: MGD product with SE at 2m resolution Dataset 5: MGD product with SE at 4m resolution Dataset 1 and Dataset 2 correspond to TerraSAR-X basic products while Dataset 3, 4 and 5 are multi-resolution products (described in the previous section). In the case of image time series, we selected 12 GEC TerraSAR-X images with RE covering Vâlcea county in Romania, named here Dataset 6. These datasets will be used for experimental work among this deliverable as well as for evaluating the feature extraction methods, which will be presented in subsection of the Chapter TerraSAR X metadata file TerraSAR-X annotation file contains a complete description of the Level 1b product components and it is considered as metadata source. Data types, valid entries and allowed attributes (e.g. units) are defined in detail for each element in the following description of the XSD schema files (the files themselves are also available to the user). Since XML is ASCII based and readable by common tools (e.g. a web browser or simple text editors) and not a binary format, the indicated data types (strings, integers, doubles, etc) for most of the annotations are the intrinsic default types [RD-2]. A XML file which is included in the delivered product packages [RD-2] can have a sample sequences as follows: <productinfo> <missioninfo> <mission>tsx-1</mission> D3.1 KDD concepts and methods proposal: report & design recommendations 23

25 </missioninfo> <acquisitioninfo> </acquisitioninfo> </productinfo> <platform> <orbit> <statevec num="95" qualind="1" maneuver="no"> </statevec> <statevec > </statevec> </orbit> </platform> A more detailed schema of the XML file is presented in Table 2. D3.1 KDD concepts and methods proposal: report & design recommendations 24

26 Table 2: Overview of main segments and hierarchical structure of the main product annotation file [RD-2]. D3.1 KDD concepts and methods proposal: report & design recommendations 25

27 2.2. Summary In this chapter, we presented the Earth Observation Data Sources that are used in TELEIOS project. We have described in detail each type of the TerraSAR-X product with emphasis in MGD and GEC with either RE or SE. The raw image data source will be transformed in a compact representation by extracting the useful descriptors in form of features as well as the most relevant metadata that will be selected during the Data Model Generation (DMG) that will be presented in Chapter 3. The GIS data source will be incorporated in the next year of TELEIOS project. D3.1 KDD concepts and methods proposal: report & design recommendations 26

28 3. Data Model Generation In this chapter we present the Data Model Generation (DMG) with its components. We start by describing Figure 8, which depicts a block diagram of the Data Model Generation. Here, we observe that it is composed of Data Sources, Content Analysis, and Context Analysis. Data Sources Content Analysis Context Analysis compressed image EO images Image Content Metadata Metadata Content Context Analysis Data Model Generation DBMS GIS GIS Content Figure 8: The Data Model Generation involves content and context analyses of the different data sources in order to provide proper image descriptors, which will be stored into a database. The Data Sources were presented in Chapter 2. In this chapter, we focus on the Content Analysis and finally the Context Analysis will be carried out in the forthcoming deliverables. The Content Analysis of the Earth Observation image data source takes as input the TerraSAR-X images and operation parameters, and generates as output their content descriptors in the form of vectors, which can later be used either for classification purposes or data mining and knowledge discovery. In the case of metadata, the input is the TerraSAR-X xml file (called from now EO TSX -XML file or shortly XML file) and the output is a set of semantic descriptors. The Content analysis of the GIS data source will be covered in next year of the project. D3.1 KDD concepts and methods proposal: report & design recommendations 27

29 It is worth noting that the content analysis is based on image descriptors, which are texture features obtained by feature extraction methods, spectral characteristics of the image, dictionary elements obtained by compression techniques, and metadata. In the following sections, we start with presenting the image content extraction in section 3.1. In this section, we explain the tiling method that is used in order to have multi-resolution and scale analysis of the considered TerraSAR-X products in section Later, we introduce the feature extraction methods based on texture characterization, spectral descriptors, compression techniques, and local pattern histogram applied to the selected dataset in section In section we present the quick look generation. Finally, we describe the TSX-XML file content in section Extraction of TerraSAR X image content During the last decades, imaging satellites have acquired huge quantities of data. TerraSAR-X sensors have delivered several millions of scenes that have been systematically collected, processed, and stored. The image size, the information content, and the resolution are continuously increasing therefore the current available systems must be updated in order to successfully respond the new requirements. The first major idea to be taken into consideration in the development of the system is the determination of relevant content that will be used when the images are classified or retrieved. In this deliverable we propose a solution in order to exploit advantageously high resolution satellite images and extracting the information from the content in order to increase the classification performances. Presently (with few exceptions), all the methods proposed in literature to extract the relevant information from the images are applied as pixel-based methods using a small analysing window. This approach is suitable for low resolution but is not productive for high resolution images; and for this reason a method based on patches is proposed to be used. The experiments reported in literature ([RI-19] - [RI-22]) show that is possible to detect a number of classes: in general, one obtains less than 10 classes for low resolution images and more than 30 classes for high resolution images, depending on the bounds of classes which are defined by a user Patch generation For high resolution images ( 1 meter), pixel-based methods do not capture the contextual information (complex structures are usually a mixture of regions and cover D3.1 KDD concepts and methods proposal: report & design recommendations 28

30 many pixels; different distributions of the same objects can have different semantic meanings), and the global features describing the overall properties of images are not accurate enough. Therefore, the general approach adopted in this project is to divide the TerraSAR-X image into a number of overlapping or not sub-images (by tiling the image into patches) and to compute the feature extraction associated to these patches. There are only few works in this direction [RI-19]-[RI-22] where the images are tiled into patches with different sizes. In Shyu et al. [RI-19] the patch size is 256 x 256 meters in order to ensure that the extracted information (features) capture the local characteristics within a patch rather the global features across the entire image. In Popescu et al. [RI-20], the TerraSAR-X High Resolution Spotlight images with a resolution of about 1 meter and the size of x meters were tiled into patches of 200 x 200 meters in order to characterize the large and relatively small structures available in the urban scene: Las Vegas (USA), Venice (IT), Gizah (EG), and Gauting (DE). From about 7000 patches the proposed method allowed to extract about 30 different classes. In contrast to [RI-19], in Birjandi and Datcu [RI-21] the original image is tiled into patches of 16 x 16 pixels or 128 x 128 pixels. The results of the classification (city, forest, and sea) were better in the second case when the image is tiled into patches of 128 x 28 pixels. The same authors propose in [RI-22] a patch contextual approach for High Resolution satellite images (with the resolution of 0.6 meter and the image size of x pixels) where patch size is 200 x 200 pixels. In the next subsections, we will explain how the TSX image will be tiled into patches and how these patches are converted into local features that act as highly compact content descriptors. The features derived from the patches to be analyzed shall be stored in a common database that permits rapid searches within it. The basic processing steps to be performed for detected data (DET) can be summarized as follows: select input product to be analyzed, extract the information necessary to tile the image into patches (possibly using information extracted form the EO TSX-XML file), tile each GeoTIFF image into patches (different scale and resolution), convert each patch into a feature vector, store the feature vectors into a common database. D3.1 KDD concepts and methods proposal: report & design recommendations 29

31 Patch identification Patch identification code definition should take into account the following issues [RI-1]: The data type will be necessary in order to differentiate different type of data: DET for detected data ITS for image time series data We refer here as source to the original product. A complete TSX filename is as follows: TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T The image name. A complete name of the image with the extension will be needed: IMAGE_HH_SRA_spot_047.tif. The patch size is another parameter that should be preserved in case you want to go back to the original image. This parameter is useful for detected data (DET) when you calculate the size of the patches at different resolutions/products. Position of the patch within the source has to be preserved, in order to back trace it, as well as for geo-referencing. The patch position should be defined by row and column block of the patch within the source. Underline _ is to be understood as field separator. The format of each patch is GeoTIFF (for DET and ITS). <Data type>_<tsx product name>_<image name>_<patch size>_<patch row block nu mber>_<patch column block number>.<format> In the following we present an example: DET_TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T165908_ IMAGE_HH_SRA_spot_047.tif_400_9_14.tif D3.1 KDD concepts and methods proposal: report & design recommendations 30

32 Patch generation for detected data A multi-resolution and scale analysis was performed as shown Figure 9 and Figure 10 respectively, using TerraSAR-X detected data (DET). This analysis performs a pyramid of different resolutions of each sub-scene to ensure that the various textures are identified at the related scale. In order to correctly tile the images into patches we need to extract from the TSX XML file the following information: rowspacing, columnspacing; numberofrows, numberofcolumns; refrow, refcolumn. When the patch extraction (tiling) algorithm is applied we need to consider only the real number of columns which is obtained by taking the difference between the number of columns (numberofcolumns) and the reference columns (refcolumn). For the rows, the number of rows is identically with the reference rows and no operation is needed. An example how are tiled the real patches is presented in Figure 11 (where n and m depends by the image and patch size). We observe that the two images previously displayed in Figure 5 and Figure 6 (extracted from the dataset 4, resolution 2m) can be seen in the right side (red dotted line) a vertical black letterboxing bar that need to be removed before tiling the image into patches. This can be easily done taking into account only the real number of columns/rows. This remark is valid for all the MGD products that we have used for the investigation. D3.1 KDD concepts and methods proposal: report & design recommendations 31

33 special MGD-SE 8 m special MGD-SE 4 m MGD-RE ~2.9 m special MGD-SE 2 m special MGD-SE MGD-SE 1 m 0m Figure 9: The multi resolution pyramid corresponding to each product. The TerraSAR- X MGD-SE standard product, marked with red, has 1m resolution. TerraSAR-X MGD- RE, marked with lilac, has 2.9m resolution. Multi-resolution product marked with orange, has 1, 2, 4 and 8 m of resolution. Figure 10: The scale analysis. In the next three figures (Figure 12, Figure 13 and Figure 14) one patch by product is displayed. The patch was randomly chosen from the set of existing patches (the D3.1 KDD concepts and methods proposal: report & design recommendations 32

34 coordinates in the image-product are: row_blocks = 9 and col_blocks = 14 (patch 9,14 )) and covering the same area on the ground. row Patch 0,0 Patch 0,1 Patch 0,n //////////////////// Patch 1,0 column Patch m,0 //////////////////// Patch m,n //////////////////// //////////////////// Figure 11: How to tile the MGD product-image into patches (Patch row_blocks, col_blocks ). The size of the patch depends on the product. For example choosing/fixing the size of the MGD-SE product the other patches (for the MGD-RE and special multi-resolution MGD-SE products) are computed recursively in order to have the same area on the ground. In the following we present an example Example In order to have the correct patch-size, we need to extract from the EO TSX XML file the number of columns and rows for each product. In the following, we give the appropriate XML file parameters for each product: MGD-SE the number of columns is parameter n_cols_se and the number of rows is parameter n_rows _SE; MGD-RE the number of columns is parameter n_cols_re and the number of rows is parameter n_rows_re special MGD-SE (for all four resolutions) the number of columns is parameter n_cols_mr_1 and the number of rows is parameter n_rows_mr_1 (for 1m D3.1 KDD concepts and methods proposal: report & design recommendations 33

35 resolution). The same two variables are extracted for 2m, 4m and 8m resolution. In the case of Venice site by fixing the patch size of the standard MGD-SE product, it is possible to compute the patch size for the rest of the products/resolutions as follows: MGD-SE pixels_se =400 (fixed a priori) MGD-RE pixels_re = pixels_se*(n_cols_re / n_cols_se) = 160 special Multi-resolution MGD_SE pixels_mr_1=pixels_se*(n_cols_mr_1/n_cols_se)=200 pixels_mr_2=pixels_se*(n_cols_mr_2/n_cols_se)=80 pixels_mr_4=pixels_se*(n_cols_mr_4/n_cols_se)=50 pixels_mr_8=pixels_se*(n_cols_mr_8/n_cols_se)=25 Figure 12: Example of the Patch (9,14) with size equal to 400 pixels corresponding to TerraSAR-X MGD-SE product over Venice. D3.1 KDD concepts and methods proposal: report & design recommendations 34

36 Figure 13: Example of the Patch (9,14) with size equal to 160 pixels corresponding to TerraSAR-X MGD-SE product over Venice (a) (b) (c) (d) Figure 14: Examples of TerraSAR-X special multi-resolution MGD-SE product patch (9,14) over Venice at (a)1m with size = 200, (b) 2m with size = 100, (c) 4m with size = 50, and (d) 8m with size = 2. An example of how the tiling of the patches looks when overlapped over the original image is presented in Figure 15. The right side of figure shows a zoom of one of the patches. D3.1 KDD concepts and methods proposal: report & design recommendations 35

37 patch size: 50 Figure 15: The grid used to tile the special multi-resolution MGD-SE product (Venice image at 4 m resolution) into patches and an example of the cut patch. The algorithms to tile the images into patches and to generate a multi-band GeoTIFF file for each patch covering a specific area (each band of the file represent one product and/or resolution, the details are further presented) are modules implemented in IDL. Considering previous example, the structure of these multi-band GeoTIFF files is: band 1 MGD-SE product (e.g., 400 x 400), band 2 MGD-RE product (e.g., 160 x 160), band 3 special MGD-SE product at 1m resolution (e.g., 200 x 200), band 4 special MGD-SE product at 2m resolution (e.g., 100 x 100), band 5 special MGD-SE product at 4m resolution (e.g., 50 x 50), band 6 special MGD-SE product at 8m resolution (e.g., 25 x 25). It is important to observe that the required run time mainly for tiling depends on the selected product and patch size. A typical run time for the pyramid resolution-product is on the order of several minutes. D3.1 KDD concepts and methods proposal: report & design recommendations 36

38 For instance, a Dell Latitude Windows XP notebook equipped with a 3.06 GHz dual CPU processor and 3.45 Gbytes of RAM (at 3.06 GHz) required about 1 minute generating 1144 patches from the 5284 x 9160 High Resolution TerraSAR-X image (Toulouse site). The product type is: special MGD-SE product at 1m resolution with patch size of 200 x 200meters. As important consideration, we note that when the images are tiled the format of the original image needs to be conserved for the patches. For example in the case of Venice image this format is unsigned int Patch generation for Image Time Series At this site, 12 images covering Vâlcea county in Romania are available. The images are standard GEC-RE product and have a ground resolution of 2.9m and were acquired every 11 days since August 5, 2010 with incidence angle around and average height about 384 m. In order to correctly tile the GEC product by cutting the interest area from the original image (see Figure 16), we need to extract from the EO TSX XML file the following information: rowspacing, columnspacing; numberofrows, numberofcolumns; <timeutc> <incidenceangle> <sceneaverageheight> <groundrangeresolution> <callfactor>. As the image is projected to the WGS84 reference ellipsoid rotated, a part covered by a rectangle is selected with 3400 x 4100 pixels. Each image (from the 12 available for time series) is tiled into patches with size 100 x 100 pixels. Figure 17 displays two patches extracted from the selected area. The patches were randomly chosen from the existing patches and the coordinates in the extracted image from the standard GEC-RE product are: row_blocks = 11 and col_blocks = 40 (patch 11,40 ) and row_blocks = 20 and col_blocks = 30 (patch 20,30 ). The algorithm for patch extraction (tiling the image) is a module implemented in IDL. D3.1 KDD concepts and methods proposal: report & design recommendations 37

39 //////////// ///////////////// ///////////////// ///////////////// ///////////////// ///////////////// ///////////////// ///////////////// //// //////// ////////////////// ////////////////////////// /////////////////////////////////// //////////////////////////////////////////// ///////////////////////////////////////////////////// /////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////// Patch 0,0 Patch 0,1... Patch 0,n... Patch m,0 Patch m,1 Patch m,n Figure 16: How to tile the GEC product-image into patches (Patch row_blocks, col_blocks ).... /////////////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////// ////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////// ////////////////////////////////////////////////////// ////////////////////////////////////////////// //////////////////////////////// /////////////////////// ///////////// ///// /////////////// ///// /////////////// ////////// /////////////// /////////////// /////////////// /////////////// Figure 17: Examples of Vâlcea image patches corresponding to TerraSAR-X GEC-RE product. (Left) patch 11,40 with size = 100, and (right) patch 20,30 with size 100 pixels. The required run time depends on the patch size and the number of images available for the ITS. For instance, a Dell Latitude Windows XP notebook equipped with a 3.06 GHz dual CPU processor and 3.45 Gbytes of RAM (at 3.06 GHz) required about 1 minute / image to generate 984 patches with the size of 100 x 100 pixels (the extracted image from the Vâlcea site has 3400 x 4100pixels). D3.1 KDD concepts and methods proposal: report & design recommendations 38

40 Feature Extraction Methods Image descriptors are important for the characterization of image structures and content. In order to characterize the TerraSAR-X images we propose the following algorithms (shortly presented in the next subsections): The Grey-Level Co-occurrence Matrix (GLCM), the Non-linear Short time Fourier Transform (NSFT), the Gabor Filters (GAFS), the Quadrature Mirror Filters (QMFS), and LZW compression techniques for dictionary extraction, which will be used with TerraSAR-X, detected data. Moreover, the Local Pattern Histogram (LPHM) will be used with Image Time Series. The GLCM, GAFS, QMFS provide texture features as image descriptors. NSFT gives spectral characteristics as image descriptors. LZW brings the dictionary element as image descriptor. All the algorithms listed before accept as input GeoTIFF data (byte, unsigned integer, and float). In the next subsection the method for feature identification is presented. The procedure can be applied for both types of data (DET and ITS) Feature identification The feature identification code definition should take into account the following issues: The data type. In this case will be: PAR feature extracted parameters. We refer here as source to the original product. A complete TSX filename is as follows: TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T TSX1_SAR GEC_RE HS_S_SRA_ T043539_ T The image name. A complete name of the image with the extension will be necessary: IMAGE_HH_SRA_spot_047.tif IMAGE_VV_SRA_spot_042.tif The patch size is another parameter that should be preserved. D3.1 KDD concepts and methods proposal: report & design recommendations 39

41 Position of the patch within the source has to be preserved like in the case of patch identification (row and column block of the patch). The algorithm used to extract the feature vector. Fist column of Table 3 presents the algorithms according to the type of data. The input parameters required for each algorithm. This number of input parameters is variable and is in relation with each algorithm. They are presented in second column of Table 3. Algorithm GLCM: Grey-Level Cooccurrence Matrix NSFT- Nonlinear Short Time Fourier Transform GAFS Gabor Filters Input parameter Alg_input_parameters_n Orientation scalegaussian Orientation Type of data Detected Detected Detected QMFS Quadrature Mirror Filters nnblevels Detected LPHM Local Pattern Histogram nnbbin widthfirstbin incrratebin, threshfirst Image time series incthresh Table 3: Feature extraction algorithm with their required input parameters and the data type. Underline _ is to be understood as field separator. The format of each feature is GeoTIFF and TXT or DAT (for both DET and ITS). <Data type>_<tsx product name>_<image name>_<patch size>_<patch row block nu mber>_<patch column block number>_<algorithm name>_<alg_input_parameters_1> _<Alg_input_parameters_2>_..._<Alg_input_parameters_n>.<format> D3.1 KDD concepts and methods proposal: report & design recommendations 40

42 As for instance in the case of detected data using GAFS algorithm: PAR_TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T165908_ IMAGE_HH_SRA_spot_047.tif_400_9_14_GAFS_2_2.txt In the case of image time series using LPHM algorithm: PAR_TSX1_SAR GEC_RE HS_S_SRA_ T043539_ T043540_I MAGE_VV_SRA_spot_042.tif_100_11_40_LPHM_5_3_2_5_2.txt We remark that for example each PAR file can contain the number of features extracted using one of the previous algorithms and the feature values (see Table 4). The values need to be in a float format as ASCI format (saved as.txt ) or unformatted (saved as.dat ). For this task an IDL procedure was implemented and can be used when it is needed. No. of parameters Value Param 1 Value Param 2... Value Param n Table 4: Example of the feature vector for each patch / algorithm Feature extraction methods based on texture and spectral analysis Texture is an apparently paradoxical notion. On the one hand, it is commonly used in the early processing of visual information, especially for practical classification purposes. On the other hand, no one has succeeded in producing a commonly accepted definition of texture. The resolution of this paradox, we feel, will depend on a richer, more developed model for early visual information processing, a central aspect of which will be representational systems at many different levels of abstraction. These levels will most probably include actual intensities at the bottom and will progress through edge and orientation descriptors to surface, and perhaps volumetric descriptors. Given these multi-level structures, it seems clear that they should be included in the definition of, and in the computation of, texture descriptors. [RI-23] Texture feature extraction methods have been utilised in a variety of applications as medical image processing, remote sensing images, automated inspection, and document processing and they are having a very important place. For feature extraction we propose the following approach: Fist we apply the different feature extraction methods as Grey-Level Co-occurrence Matrix, Gabor filtering, Quadrature Mirror Filters, and Non-linear Short time Fourier Transform; then we select D3.1 KDD concepts and methods proposal: report & design recommendations 41

43 the features that are the best description of the data content (see this in the evaluation of the features based on the precision-recall measure in subsection 5.1.2). In the case of GLCM algorithm, when the GeoTIFF images are tiled before saving the generated patches these are converted to 8 bits (by using the logarithm to scale from 16 to 8 bits) in order to properly apply the algorithm. Grey-Level Co-occurrence Matrix The Grey-Level Co-occurrence Matrix (GLCM) is a local tabulation of how often different combinations of pixel brightness values (gray levels) occur in an image [RI-3], [RI-4]. The GLCM is also called the Gray Tone Spatial Dependency Matrix. The GLCM is created from a gray scale image by selecting either horizontal (0 ), vertical (90 ), or diagonal (45 or 135 ) orientation. The size of GLCM depends on the number of gray values available in the image. For example, in [RI-3] they obtain for an input image of 8 bits, i.e. 256 values, a GLCM of 256 x 256 elements. In our case, we scale the radiometric range of the input images to 16 steps and obtain a GLCM size of 16 x 16 elements. The algorithm is an implementation of the GLCM texture parameters of [RI-3] and the results computed from the GLCM are: Mean Energy Dissimilarity Variance Correlation Cluster shade Entropy Homogeneity Cluster prominence Contrast Autocorrelation Maximum probability The GLCM algorithm consists of a single module implemented in Java. The output result consist of 12 GLCM parameters for each patch (where the total number of patches are nr_block_row * nr_block_col). Non-linear short time Fourier transform Non-linear short time Fourier transform analysis is based on the principle of stationarity of short time signals. This method of TSX image feature extraction and complex image information retrieval was proposed in [RI-7] and [RI-20]. The proposed method extracts six non-linear features. The first two features are based on statistical properties of the spectrum and the next four features are motivated from timbre features used for music genre classification [RI-8]. D3.1 KDD concepts and methods proposal: report & design recommendations 42

44 The results computed are: Mean (of the coefficients) Variance (of the coefficients) Spectral centroid in range Spectral centroid in azimuth Spectral flux in range Spectral flux in azimuth The non-linear short time Fourier transform algorithm consists of a single module implemented in Java using the Java Advanced Imaging library. The output result consists of 6 parameters for each patch. Gabor filters A Gabor filter is a linear filter used in image processing [RI-5]. Frequency and orientation representations of a Gabor filter are similar to those of the human visual system, and it has been found to be particularly appropriate for texture representation and discrimination. In the spatial domain, a 2D Gabor filter is a Gaussian kernel function modulated by a sinusoidal plane wave [RI-6]. The Gabor filters are self-similar - all filters can be generated from one mother wavelet by dilation and rotation. A two dimensional Gabor function g ( x, y) and its Fourier transform G ( u, v) can be written as [RI-6]: 1 1 x 2 g( x, y) exp 2 x y 2 2 x y 2 2jWx, (1) 2 y G( u, v) exp 1 2 ( u W ) 2 2 u v 2 2 v where u 1 and v 2 x 1 2 y. (2) The self-similar Gabor wavelets are obtained through the generating function: g ( x, y) a m g( x ', y ' mn ), m 0,... S 1; n 0,..., K 1; a 1 x' a m ( x cos y sin) and y' a m ( xsin y cos), n, (3) K where S is the number of scales and K is the number of orientations. The variables in the above equations are defined as follows: 1 U S1 a h ; Ul ( a 1) U h u ; ( a 1) 2ln (2ln 2) 2 tan 2ln u u v U h 2ln 2 ; W U 2 2 h. (4) K U h U h D3.1 KDD concepts and methods proposal: report & design recommendations 43

45 The Gabor algorithm provides the necessary routines for the user to apply Gabor filters to images. This implementation of the Gabor filter convolves an image with a lattice of possibly overlapping banks of Gabor filters at different scale (scale), orientation (θ), and frequency (U l and U h ). The scale is the scale of the Gaussian used to compute the Gabor wavelet. The Gabor algorithm controls the generation of output texture parameters. The texture parameter results computed from the Gabor filter are the follows: Mean and Variance for different Scale (scale) Orientation (θ) The Gabor filters algorithm consists of a single module implemented in C with a Java input/output interface. The output result consists of 2 * scalegaussian * orientation texture parameters for each patch (e.g., if scalegaussian = 2, orientation = 2 then 8 parameters/patch). Quadrature mirror filters Quadrature mirror filter banks are multirate (i.e. with variable sampling rate throughout the system) digital filter banks, introduced by Croisier, Esteban and Galand in [RI-9]. During the last two decades since the inception of quadrature mirror filter banks, they have been extensively used in speech signal processing, image processing and digital transmultiplexers [RI-10]. Quadrature mirror filter banks are used to split a discretetime signal into a number of bands in the frequency domain to process each sub-band in independent manner [RI-10]. As proposed in [RI-11], statistical features obtained from the filtered images using quadrature mirror filter banks in synergy with some other features can be used for image (satellite image) indexing. The number of features which can be obtained from the presented algorithm depends upon the level selected for the quadrature mirror filter sub-band decomposition like a wavelet. Features are nothing but the mean and variance of the four filtered and subsampled images (low pass, horizontal, vertical and diagonal) in the quadrature mirror filter sub-band pyramid. There are many techniques available to design quadrature mirror filter banks. We have chosen the QMF banks designed by Eero P. Simoncelli and Edward H. Adelson at the Vision Science Group, The Media Laboratory, Massachusetts Institute of Technology. The results computed from the quadrature mirror filter are: Low pass sub-band D3.1 KDD concepts and methods proposal: report & design recommendations 44

46 Mean & Variance for : Horizontal sub-band Vertical sub-band Diagonal sub-band The algorithm consists of a single module implemented in C with a Java input/output interface. The output result consists of ( nnblevels *3 1) * 2 parameters for each patch (e.g. if nnblevels = 1 then 8 parameters/patch). Results of the feature extraction methods for Detected data The results generated by the feature extraction algorithms described before can be transferred as input to a clustering and/or classification algorithm (e.g., a k-means algorithm, or a fuzzy c-means clustering algorithm, or a support vector machine). Below, we present the feature extracted considering the four methods presented before (for non-normalized and normalized features). For normalisation of the features, different methods can be considered and described in detail in [RI-12]. For our task the Z-score normalisation method was selected and used further in section 5.1. As an example, we chose the special multi-resolution MGD-SE product at 1m resolution. The input parameters are: the TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T , the IMAGE_HH_SRA_spot_047.tif image, the patch size 200, the row block = 0 and column block = 0, The feature vector (the number of features depending by the selected input parameters of the algorithm) and their corresponding values (in ASCII format) are displayed in the following tables for each algorithm. Here, the first row represents non-normalised values and the second row displays the normalized values. An example of GLCM algorithm results with orientation equals 1. No. of param Mean Variance Entropy Contrast Energy Correlation E E-14 D3.1 KDD concepts and methods proposal: report & design recommendations 45

47 Homogeneity Autocorrelation Dissimilarity Cluster shade Cluster Prominence Maximum probability E E Examples of NSFT algorithm results. No. of param Mean Variance Spectral centroid in range Spectral centroid in azimuth Spectral flux in range Spectral flux in azimuth E E E E E Examples of GAFS algorithm results with scalegaussian = 2 and orientation = 2 No. of param 8 Mean scalegaussian 1 scalegaussian 2 orientation 1 orientation Variance scalegaussian 1 scalegaussian 2 orientation 1 orientation An example of QMFS algorithm results with nnblevels equals 1. No. of param Low pass sub-band Horizontal sub-band Mean Vertical sub-band Diagonal sub-band Low pass sub-band Horizontal sub-band Variance Vertical sub-band Diagonal sub-band D3.1 KDD concepts and methods proposal: report & design recommendations 46

48 Figure 18 and Figure 19 show some feature extracted using NSFT and GLCM feature extraction methods, respectively. Both methods were applied to the whole Venice image. a) b) Figure 18: The NSFT method is applied to the whole Venice image. The original image (a) and the mean (b) are displayed. a) b) c) d) Figure 19: The GLCM method is applied to the whole Venice image. The original image (a) and three features: mean (b), entropy (c), and correlation (d). D3.1 KDD concepts and methods proposal: report & design recommendations 47

49 Compression method based on dictionary extraction Fast Compression Distance The Fast Compression Distance (FCD) is a similarity measure based on data compression and following a parameter-free approach. It was proposed in [RI-15]. This approach combines the robustness of Normalized Compression Distance (NCD) [RI-18] with the speed of Patter Recognition based on Data Compression (PRDC) [RI-17] by establishing a link between both concepts. The NCD measures the similarity between two images based on their compression level. The NCD can be explicitly computed between any two strings or files x and y and it represents how different they are. The PRDC was introduced by [RI-17] as a classification methodology for general data. The main idea of PRDC is to extract dictionaries by applying a compressor (i.e. LZW algorithm [RI-16]), directly from each object represented by a string. The data have to be previously encoded into a string. The definition of FCD is given by D( x) D( x), D( y) FCD( x, y), (5) D( x) is the number of patters which are found in both dictionaries. Figure 20 displays a graphical representation of this concept. where D(x) and D(y) are the sizes of the respective dictionaries and D( x), D( y) Figure 20: Graphical representation of the intersection between two dictionaries D(x) and D(y), respectively extracted from two objects x and y through compression with the LZW algorithm. The values of FCD range between 0 and 1, where the best case is 0 occurring when the images are identical. D3.1 KDD concepts and methods proposal: report & design recommendations 48

50 We can use FCD as similarity metric in order to search similar images, concept that can further be applied in retrieval process. It can be achieved in two steps: First, the extraction of the dictionaries as image descriptors and the second the computation of the similarity measure (equation 5). Extraction of the dictionaries: A dictionary D(x) is extracted using the compression algorithm LZW [RI-16], which is a lossless dictionary based compression method. This kind of compression algorithms scan a file for patterns (sequences) of data that occur more than once. These sequences are stored in a dictionary. The algorithm consists on several python scripts and the result is the dictionary element. The dictionaries are computed for each patch and stored into the database. Figure 23 presents an example of dictionary elements for one patch. Similarity measure: This is computed following the equation 5 between two dictionaries. Results of the dictionary extraction for Detected data Figure 21 shows an example of the dictionary extraction, which is the result of LZW compression algorithm; this is saved in a text file. Later, it is stored in a database in order to allow computing the FCD. Figure 21: Dictionary extraction example. The dictionary element is stored into a database table Feature extraction for Image Time Series Local pattern histogram The method [RI-13] is applied for each patch. First, it thresholds the images using a series of thresholds, leading to three kinds of patterns and then the histograms of the D3.1 KDD concepts and methods proposal: report & design recommendations 49

51 local pattern size are computed and concatenated. As this is only applicable for images, it is adapted to time series data by computing the histogram of local patterns in spatiotemporal space. The procedure is demonstrated in Figure 22. Supposes the patch size is 5 x 5 and images at five different times are available, resulting in a 5 x 5 x 5 spatio-temporal space. A series of thresholds ti are selected to threshold the patches by the interval gc ±ti (gc is the value of the center pixel). Each threshold will result in three kinds of local patterns, representing local volumes in spatio-temporal space. Three kinds of patterns in spatiotemporal space are indicated by colours (red, green, and blue), corresponding to bright, homogeneous and dark targets in RSX image time series. The histogram of the three kind local patterns Hr, Hg and Hb in spatio-temporal space are computed with increasing bin size. The three histogram are concatenated as a feature vector H = (Hr;Hg;Hb). An example of features is presented in Figure 23. Intuitively, these features are able to discriminate different classes, such as road, building, grass land and forest [RI-14]. Figure 22: Demonstration of Local pattern histogram feature extraction method used for image time series. The input parameters of the algorithm are: the number of bin in each histogram, the width of the first bin, the increase rate of the bin width, the first threshold, and the increase rate of the threshold. D3.1 KDD concepts and methods proposal: report & design recommendations 50

52 The dataset for ITS is represented by a scene of 3400 x 4100 pixel acquired at 12 different times. The entire image is divided into 24 x 41 patches, which means the patch size is 100 x 100 pixels. The typical run times for generating the feature vector for each patch is around 2 seconds/patch, which means 33 minutes using Dell Latitude Windows XP notebook equipped with a 3.06 GHz dual CPU processor and 3.45 GB of RAM (at 3.06 GHz). The feature extraction algorithm consists of a single module implemented in Matlab. a) b) c) d) e) f) Figure 23: Example of feature component extracted with LPHM: a) Test patch, b) - f) feature components Quick look generation for detected data and image time series Quick-look identification code definition should take into account the same issues as patch identification code presented in subsection Two differences can be noted 1) the data type which in this case is QLK for quicklook and 2) the format is in this case JPG for DET and GIF for ITS. D3.1 KDD concepts and methods proposal: report & design recommendations 51

53 <Data type>_<tsx product name>_<image name>_<patch size>_<patch row block nu mber>_<patch column block number>.<format> An example of detected data is presented below QLK_TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T165908_ IMAGE_HH_SRA_spot_047.tif_400_9_14.jpg In the detected data case the quick-look image is a JPG file and is created for each specific part of the site at different resolutions-products, as is shown in Figure 24. Figure 24: Examples of quick-looks (200 x 200 pixels) for detected data using special multi-resolution MGD-SE product at 1m. In the other case for Image Time Series the quick-look image is a GIF file and is created for each specific location to indicate the dynamic evolution of the scene as shown Figure 25 using the 12 patches acquired at different times. Figure 26 shows 3 time series of patches with changes and no-changes for a specific area using 8 from 12 available patches. Figure 25: Examples of quick-looks (100 x 100 pixels) in the case of Image Time Series (GEC-RE product). D3.1 KDD concepts and methods proposal: report & design recommendations 52

54 Changes Changes D3.1 KDD concepts and methods proposal: report & design recommendations 53

55 No changes No changes Figure 26: Examples of changes and no-changes for a specific area. D3.1 KDD concepts and methods proposal: report & design recommendations 54

56 3.2. TerraSAR X XML file content The TerraSAR-X XML file provides information used for querying the image metadata as well as information for tiling the image content. Examples of metadata queries can be found in [RI-29]. In the following, we describe the main aspects of this file TerraSAR X XML metadata used for querying An overview of main segments and hierarchical structure of EO TSX XML file [RD-2] has been presented in Table 2 of section 2.1. In the next paragraphs the most important elements of this table are presented [RD-2] and selected as EO TSX XML metadata content needed for the Data Model Generation. These elements are the following: productcomponents: 1. annotation: Pointer to the annotation file (xml file). path: Localisation of the xml file. filename: Name of the xml file (i.e. TSX1_SAR MGD_RE HS_S_SRA_ T _ T xml). 2. imagedata: Information about the TerraSAR-X image productinfo path: Localisation of the GeoTIFF file. filename: Name of the file (i.e. IMAGE_HH_SRA_spot_042.tif). 1. missioninfo: Mission and orbit parameters at start of scene mission: name of the mission.(i.e. TSX-1). orbitphase: Orbit phase. The possible values are: -1 prelaunch phase, 0 launch phase, 1 nominal Orbit. orbitcycle: Cycle number (i.e. 15). absorbit: absolute orbit number at start of scene relorbit: relative orbit number 131 D3.1 KDD concepts and methods proposal: report & design recommendations 55

57 numorbitsincycle: nominal number of orbits per cycle depends on phase currently 167. orbitdirection: ascending / descending flag. 2. acquisitioninfo: SAR sensor configuration and instrument modes during acquisition sensor: Identifier of the sensor (i.e. SAR). imagingmode: From the many technical possibilities four imaging modes have been designed to support a variety of applications. The following imaging modes are defined for the generation of basic products: StripMap mode (SM) in single or dual polarization, High Resolution Spotlight mode (HS) in single or dual polarization, Spotlight mode (SL) in single or dual polarization, ScanSAR mode (SC) in single polarization lookdirection: look direction of the satellite sensor, which can be left or right. antennareceiveconfiguration: single-receive antenna (SRA) polarisationmode: Polarisation mode of the antenna. This can be single, dual, twin, quad. pollayer: polarization layer list (i.e. HH, VV, HV, VH). 3. productvariantinfo: Product type and variant description producttype: Refers to Product Identification Scheme, which is used to indentify and classified the different basic products for TerraSAR by using a mnemonic scheme described in the following. The product identifier is split into 4 sub-identifiers and the global product name is composed as: <projection>_<resolution class>_ <imaging mode>_<polarization mode> (e.g. MGD_SE_SM_S for a spatially enhanced single polarization StripMap product in multi look ground range projection). productvariant: This specify the type of geometrical projection and data representation. The possible values are: Single Look Slant Range, Complex representation (SSC), Multi Look Ground Range, Detected representation (MGD), Geocoded Ellipsoid Corrected, detected representation (GEC) and Enhanced Ellipsoid Corrected, detected representation (EEC). D3.1 KDD concepts and methods proposal: report & design recommendations 56

58 projection: Type of projection (i.e. slant range, ground range, map). resolutionvariant: The TerraSAR-X products can be Spatially Enhanced Products (SE) or Radiometrically Enhanced Products (RE). radiometriccorrection: Refers to the calibration of the image. It can be absolutely calibrated, relative calibrated, not calibrated. 4. imagedatainfo: Image layer format pixelvalueid: complex amplitude and phase, radar brightness (beta nought), sigma nought, etc. imagedatatype: detected or complex. imagedataformat: GeoTIFF for geocoded images, COSAR for SSC products. numberoflayers: number of polarizations + DRA channels + elevation beams (ScanSAR). imagedatadepth: bits per pixel (16bit detected or 2x16bit complex). imageraster: Main description about the image data: numberofrows: Total of rows in the image. numberofcolumns: Total of columns in the image. rowspacing: Spacing of samples within a row from common raster [s or m]. columnspacing: Spacing within a column (e.g. azimuth sampling). groundrangeresolution: Resolution in range. azimuthresolution: Resolution in azimuth. azimuthlooks: effective number of looks (ENL). rangelooks: number of looks taken in range 5. sceneinfo: Time and scene location information D3.1 KDD concepts and methods proposal: report & design recommendations 57

59 sceneid: Orbit and timing information. This field allows to uniquelly indentify the TerraSAR-X product (i.e. C43_N2_D_HS_spot_042_R_ T05:25: Z). scene Star TimeUTC: time stamps of first image row (all processed azimuth times should be Doppler zero times (e.g T20:29: Z in CCSDS ASCII time format). scene Stop TimeUTC: time stamps of last image row. scenecentercoord: Information about the center of the scene expressed in geographic coordinates: refrow: Position in image row. Range sample position for SSCs. Annotated image sample positions for complex products has only informative purposes. All localisation is based on timing information refcolumn: Position in image column. lat: Geographical latitude positive towards north. lon: Geographical longitude positive towards east. azimuthtimeutc: geo coordinates are derived from this timing information using the corrections annotated in the geo reference annotation component. rangetime: geo coordinates are derived from this timing information using the corrections annotated in the geo reference annotation component. incidenceangle: Incidence angle is the angle between the vertical to the terrain and the line going from the antenna to the object TerraSAR X XML metadata used for patch generation The TSX XML metadata extracted and used for tiling the detected data is displayed as follows (as a screenshot): Product name Image name D3.1 KDD concepts and methods proposal: report & design recommendations 58

60 Number of rows / columns Row/column spacing units Azimuth resolution Ground range resolution Latitude and longitude Reference number of rows / columns The TSX XML metadata extracted for each product standard and special multiresolution product is presented in detail in Table App 1 - Table App 3 in section 9.2 of the Appendix. The useful TSX XML metadata used for tiling and identification of the acquisition time of the image need for the time series is presented further (as a screenshot): Product name Image name D3.1 KDD concepts and methods proposal: report & design recommendations 59

61 Number of rows / columns Acquisition time Incidence angle Average height Calibration factor The TSX XML metadata extracted in the case of Image Time Series is displayed in detail in Table App 4 of section 9.2 in the Appendix Summary In this chapter we focused on the Data Model Generation that creates a quasi-complete description of the Data Sources in a reduced format. The functions of all components are managed by an extension module the will be presented in next chapter. In this chapter, the content analysis of the image was carried out using different feature extraction methods, which will be stored into a relation database, to be presented in Chapter 4. This study of the different feature extraction methods gives us empirical evidence about the performance and quality of the methods, which is very important for deciding which of these methods will be used in following implementations of data mining and knowledge discovery. Moreover, since the accuracy in the classifications and performance of image mining and retrieval systems strongly depends on the primitive features (descriptors) used, a precise assessment of the quality of the extracted features is necessary. These considerations will be presented in detail in Chapter 5, where the complete evaluation will be carried out. D3.1 KDD concepts and methods proposal: report & design recommendations 60

62 4. A conceptual Model and Relational storage Schema for the descriptor Database In previous chapter, we presented the Data Model Generation, which is based on the data sources (image and its metadata), the content analysis involving the tiling of the image, the extraction of descriptors (features) form the image, and relevant metadata. In this chapter, we provide a description of the conceptual design of the TerraSAR-X data based on an entity-relationship model (ERM) in section 4.1 as well as a detailed description of the relational storage schema and the tables it comprises in section 4.2. MonetDB [RI-26] is used as Database Management System Conceptual design of TerraSAR X data model The conceptual model to support the Data Model Generation of TerraSAR-X product is presented in Figure 27 considering as data sources the image and it metadata (xml file). The main entities in this model are: Earth-observation product, Image, xml Product info, Patch, Feature and Label, which are described as follows: The EO TerraSAR-X product is composed of the 1) TerraSAR-X image, 2) the XML annotation file, which contains a complete description (metadata) of the image and its generation steps, and 3) GIS elements, which will be covered in following deliveries. The image is tiled into several patches in order to apply different feature extraction methods in each patch. The patches can have different size depending on the resolution. The patches are converted into feature vectors, which are complete descriptors of the image content. Each patch has semantic labels describing the main information contained in the patch. D3.1 KDD concepts and methods proposal: report & design recommendations 61

63 Figure 27: Entity-relationship diagram for the model generation of TerraSAR-X Earth- Observation product 4.2. Database Schema for image descriptors The database schema is presented in Figure 28 and the main tables are described in the following D3.1 KDD concepts and methods proposal: report & design recommendations 62

64 Figure 28: Database schema diagram for the Data Model Generation of TerraSAR-X Earth Observation product. D3.1 KDD concepts and methods proposal: report & design recommendations 63

65 Description of the tables In the following, we present a description of the main tables in the database schema used for storing the tiling process and the feature extraction results Image This table contains the information about the TerraSAR-X image. The schema of the table is given below Attribute Type Description image_id (PK) INT Id of the table filename VARCHAR(100) Name of the image file numberofrows INT Number of rows numberofcols INT Number of columns rowspacing FLOAT Spacing of samples within a row rowspacingunit VARCHAR(2) Units of row spacing (i.e m) columnspacing FLOAT Spacing within a column columnspacingunits VARCHAR(2) Unit of column spacing (i.e m) resolution INT Image resolution latitude FLOAT Latitude of the image longitude FLOAT Longitude of the image Xmlproduct This table contains the information about the xml file, which includes the metadata of Earth Observation product. The schema of the table is given below Attribute Type Description xmlproduct_id (PK) INT ID of the table image_id (FK) INT Id of the associate table mission VARCHAR(20) Name of the mission orbitphase INT Orbit phase orbitcycle INT Cycle number absorbit INT Absolute orbit number at start of scene relorbit INT Relative orbit number numorbitsincycle INT Nominal number of orbits per cycle orbitdirection VARCHAR(3) Ascending or descending flag sensor VARCHAR(20) Identifier of the sensor imagingmode VARCHAR(5) Imaging mode: SM, HS, SL, SC D3.1 KDD concepts and methods proposal: report & design recommendations 64

66 lookdirection VARCHAR(5) Look direction of the satellite sensor. polarisationmode VARCHAR(6) Polarization mode of the antenna. Dual, single, Twin or quad pollayer VARCHAR(2) Layer list HH, VV, e producttype VARCHAR(12) Product identification scheme productvariant VARCHAR(3) Type of geometrical projection and data representation projection VARCHAR(15) Type of projection resolutionvariant VARCHAR(2) Spatially (SE) or radiometrically enhanced (RE) radiometriccorrection VARCHAR(15) Calibration of the image pixelvalueid VARCHAR(20) complex amplitude and phase, radar brightness (beta nought), sigma nought, etc. imagedatatype VARCHAR(8) Image data type can be detect or complex imagedataformat VARCHAR(7) Format of the image, tiff, geotiff, etc. numberoflayers INT Number of image layers imagedatadepth INT Image data depth refers to bits per pixel. numberofrows INT Total of rows in the image numberofcolumns INT Total of column in the image groundrangeresolution DOUBLE Resolution in range azimuthresolution DOUBLE Resolution in azimuth azimuthlooks FLOAT Effective number of looks rangelooks FLOAT Number of looks in range sceneid VARCHAR(70) Orbit and timing information scenestartimeutc DATE Scene start time UTC refers time stamps of first image row scenestoptimeutc DATE Scene stop time UTC time stamps of last image row scenecentercoord_refrow FLOAT Row of the scene center scenecentercoord_refcolumn FLOAT Column of the scene center scenecentercoord_lat FLOAT Geographical latitude of the center of the scene scenecentercoord_lon FLOAT Geographical longitude of the center of the scene center_azimuthtimeutc DATE Time stamps of center image in azimuth center_rangetime FLOAT Time stamps of center image in range incidenceangle FLOAT Incidence angle productname VARCHAR(100) File name of EO product productpath VARCHAR(500) Physical path of the product scenecornercoord_lon FLOAT Geographical longitude of the corner of the scene scenecornercoord_lat FLOAT Geographical latitude of the corner of the scene Referenceprojection VARCHAR(100) System of geographic reference D3.1 KDD concepts and methods proposal: report & design recommendations 65

67 Patch This table stores the information of the image tiles. The schema of the table is given below Attribute Type Description patch_id (PK) NULL Id of the table patching_id (FK) INT Id of the patching table datatype VARCHAR(3) Type of image row FLOAT Row position on the main image cols FLOAT Column position on the main image quicklookpath VARCHAR Physical path of the image Label This table stores the semantic labels. This table links a patch with a semantic meaning. The schema of the table is given below Attribute Type Description label_id (PK) INT Id of the table patch_id (FK) INT Id of Patch table name VARCHAR(100) Semantic name description VARCHAR(250) Description of the label Features_glcm This table stores the primitive features obtained using Grey-Level Co-occurrence Matrix method. The schema of the table is given below Attribute Type Description features_glcm_id (PK) INT Id of the table patch_id (FK) INT Id of patch table mean FLOAT mean variance FLOAT Variance entropy FLOAT Entropy contrast FLOAT Contrast energy FLOAT Energy correlation FLOAT Correlation Homogeneity FLOAT Homogeneity autocorrelation FLOAT Autocorrelation Dissimilarity FLOAT Dissimilarity Clustershade FLOAT Cluster shade clusterprominence FLOAT Cluster prominence maximumprobability FLOAT Maximum probability D3.1 KDD concepts and methods proposal: report & design recommendations 66

68 Features_qmfs This table stores the primitive features provided by Quadrature mirror filters method. The schema of the table is given below Attribute Type Description features_qmfs_id (PK) INT Id of the table patch_id (FK) INT Id of patch table Mean FLOAT Mean Variance FLOAT Variance Lowpass FLOAT Low pass sub-band horizontal FLOAT Horizontal sub-band Vertical FLOAT Vertical sub-band Diagonal FLOAT Diagonal sub-band Features_nstf This table stores the primitive features obtained using Non-linear short Fourier transform method. The schema of the table is given below Attribute Type Description features_nsft_id (PK) INT Id of the table patch_id (FK) INT Id of patch table Mean FLOAT Mean of the coefficient Variance FLOAT Variance of the coefficient centroidrange FLOAT Spectral centroid in range centroidazimuth FLOAT Spectral centroid in range Fluxrange FLOAT Spectral flux in range fluxazimuth FLOAT Spectral flux in azimuth Features_gafs This table stores the primitive features provided by Gabor filters method. The schema of the table is given below Attribute Type Description features_gafs_id (PK) INT Id of the table patch_id (FK) INT Id of patch table Mean FLOAT Mean Variance FLOAT Variance Scale FLOAT Scale orientation FLOAT Orientation D3.1 KDD concepts and methods proposal: report & design recommendations 67

69 Dictionary This table stores the dictionary of the images obtained using LZW compression method. The schema of the table is given below Attribute Type Description dictionary_id (PK) INT Id of the table patch_id (FK) INT Id of the patch dict_elem INT Value of the dictionary element 4.3. Summary In this chapter we introduced a conceptual model for TerraSAR-X data considering as data sources the image and its metadata (xml file), and proposed a relational storage schema for the descriptor database. The content of this chapter is a starting point for next implementations; therefore, the proposed schema will be extended for supporting the data model generation including all data sources and data mining functions. Furthermore, on the subject of assessment some performance metrics (i.e. precision and recall) can be automatically computed since the descriptors and queries results are stored into the databases. In terms of performance and efficiency, which are requirements for query data mining and knowledge discovery, having a descriptor database and methods running on it may improve the results of the queries. As a first example, we will discuss in Chapter 5, the implementation of the Fast Compression Distance directly in Monetdb using SQL sentences. We expect that the other feature extraction or clustering methods can be implemented using database technology, allowing faster queries. Finally, this descriptor database will be used as one for the basic ingredients for the generation of an RDF semantic data-set that will provide the capability of performing semantic queries. D3.1 KDD concepts and methods proposal: report & design recommendations 68

70 5. Query by example and active learning methods for implementing knowledge discovery functions In the context of Knowledge Discovery in Databases (KDD) and Query Data Mining, the feature extraction process) is the starting point of the ingestion process. Given all the extracted descriptors that are available in the database, the next step is to provide methods and tools for clustering in order to find well-recognized classes, active learning in order to label the classes automatically, computing similarity metrics between image content, and retrieving the proper images. In this chapter, we describe two examples of a system that implements the concept of query by example and the semantic definition by learning methods. The first one corresponds to a Search Engine whose main core is a Support Vector Machine (SVM) classifier. This tool relies on 1) feature extraction methods providing the most relevant descriptors of the images, 2) SVM as classifier grouping the image descriptors into generic classes (without semantic), and 3) relevance feedback interacting with the end user. The second one corresponds to Content Based Image Retrieval (CBIR) based on the Fast Compression Distance, named here CBIR-FCD, which is implemented using compression based techniques to describe the image content, and distance measures to discover similar content in the database. CBIR-FCD has been tested with multimedia images giving satisfactory results [RI-15]. We extended this approach to TerraSAR-X images. Both systems implement the concept of query based on example using the image content. We start by presenting a description of the search engine based on SVM in section 5.1. Later, using this tool for evaluating the feature extraction methods in terms of precision and recall, we present some evaluation results using detected data and image time series in subsections and 5.1.3, respectively. Section 5.2 introduces the CBIR based on Fast Compression Distance, its main functions, logical architecture, and examples of operation and some evaluation results using TerraSAR-X images Search engine based on Support Vector Machine and relevance feedback Presently Earth Observation (EO) satellites acquire huge volumes of high resolution images, very much over-passing the capacity of the users to access the information content of the acquired data. In addition to the existing methods for EO, data and information extraction are needed new methods and tools to explore and help to discover the information hidden in large EO image repositories. D3.1 KDD concepts and methods proposal: report & design recommendations 69

71 In the next section a concept of search engine based on Support Vector Machine (SVM) classifier is presented. The proposed search engine supports users to search images of interest in a large repository, by visual ranking of automated suggested images, grouped in classes of relevance Description of Search Engine based on SVM and relevance feedback The system diagram is presented in Figure 30 and it is composed of: a TerraSAR-X satellite image database, a primitive feature extraction block, a feature database, a classification component (SVM), a class database, image patch index, and a Graphical User Interface (GUI) which allows the human -machine communication. The Relevance Feedback SVM software tool supports users to search images (patches) of interest in a large repository. The Graphical User Interface of this tool allows Human-Machine Interaction to rank the automatically suggested images which are expected to be grouped in the class of relevance. Visual supported ranking allows enhancing the quality of search results by giving positive and negative examples as right and left click respectively. This tool allows importing number of scenes into a new project which have been already prepared for ingestion into tool. Preparation of scenes refers to tiling the full scene into patches (in.tif format), generating quick-looks corresponding to each patch (in.jpg format), and computation of primitive features (in multi-band.tif format). A project contains all the information regarding the already imported scenes (see upper part of Figure 29). To start the information mining, the user first need to open an existing project by pressing <Open an existing project...> button and selecting the project identifier which always has an extension '.jsprj'. Once the project is loaded, user can give relevant examples for current search by pressing <Positive examples...> button. Negative examples can be given by pressing <Negative examples...> button. These two buttons allows to input the examples one-by-one. But user can also load the already prepared class training file which always has an extension '*.jstrain' (this option is available for the second version of this tool). After giving sufficient positive and negative examples user should press the <Train> button to start the search in the database. Pressing this button will open another window with three rows containing patches (see lower part of Figure 29). The results on these three rows are generated by Support Vector Machine running in the backend. First row refers to the patches belonging to the class of search. Second row contains patches ambiguous, closest to the negative side of decision surface on feature space in SVM. Last row contains patches ambiguous patches which lies closest to the positive side of decision surface. User can modify the decision surface by giving more positive or negative examples by pressing 'Left-mouse-click' and 'Right-mouse-click' respectively. Pressing again the <Train> button invokes the SVM to modify decision surface as per the new and older given examples. Once a user is satisfied patches related D3.1 KDD concepts and methods proposal: report & design recommendations 70

72 to relevant category can be extracted from the data-base. Pressing <Enter a class name> button allows users to define the class name of interest and after that pressing <Build data-set now> button will copy the retrieved patches in a particular folder connected with the class name given earlier. Figure 29: Graphical user interface of the Search engine based on Support Vector Machine and Relevance Feedback. D3.1 KDD concepts and methods proposal: report & design recommendations 71

73 Results / Examples Image Data Base Patch Quick Looks Image Patches Features Data Base SVM GUI Relevance Feedback GLCM Gabor QMF NL-STFT Primitive Feature Extraction Blocks Class Data Class Class Data Base Data Base Base Class Data Base Image Patch Indexing Index Validation Figure 30: Search engine based on Support Vector Machine and Relevance Feedback software tool diagram. D3.1 KDD concepts and methods proposal: report & design recommendations 72

74 Evaluation method for TerraSAR X detected data Following the previous idea presented in section of patch generation, we chose to tile the images into patches having on the ground 200 x 200 meters (this means for each product and/or resolution the patch size will be different) and after that to extract the important information as a set of features. The Support Vector Machine (SVM) was considered as classifier. In order to perform an objective evaluation of these features, we compute and compare these features for each MGD product (standard or special multi-resolution products). The dataset was previously presented in subsection This result in 5 experiments corresponding to each product/resolution (see the pyramid resolution in Figure 9 of subsection 3.1.1): standard MGD-SE product, standard MGD- RE product and three special MGD-SE product at 1m, 2m, and 4m resolution. We define 35 semantic classes and group the patches accordingly, using the SVM tool and the human expertise. In our approach for assigning the patches into classes, one patch was assigned only to one class based on the dominant content of the patch. From the generated classes, we select 30 classes, which have more than 10 patches in the class. The list of all 35 classes is presented in Table 5. Here, the 30 selected classes are marked with bold, and one patch example for each class is shown in Figure 31. Our interest is to compare these features for different products in order to identify the best feature (or features) and the product that will be used further in the TELEIOS project. In the next paragraphs the investigated methods proposed in subsection for TerraSAR-X images are compared. At the beginning, we applied all methods in order to have the feature vectors, which are detailed in the following In the case of GLCM, the result is a set of 48 features, 12 features for each orientation: 0, 45, 90, and 135. In the case of NSFT, the result is a set of 6 features. In the case of GAFS, the result is a set of 8 features using 2 scales and 2 orientations, and a set of 48 features using 4 scales and 6 orientations. In the case of QMFS, the result is a set of 8 features using the number of levels equal to 1, and a set of 14 features using number of levels equal to 2. For each feature extraction method, we tried to detect the classes among the 2700 patches of our database. For each class, we give 20% of the patches (of each class) for the training and we tray to detect the similar patches during 7-10 training iterations (this depends also on the class and method). The evaluations stop when the classified patches which are displayed by the Search Engine based on SVM and Relevance Feedback tool D3.1 KDD concepts and methods proposal: report & design recommendations 73

75 remain in a stable result. The procedure is repeated 2-3 times for the same class, giving the same positive and negative examples in the same order. For the quantitative assessment, we compared the classification results with the annotated database (for all 30 classes). Different measures are proposed in literature in order to evaluate the performance and we chose for our evaluation the Precision-Recall that will be computed for each class, feature, and product. The precision is defined as the fraction of the retrieved images which are relevant, while the recall is defined as the fraction of relevant images which have been retrieved: A B precision and A A B recall, B where A represent the retrieved images and B represent the relevant images. The mean of the precision-recall is computed as: m precision 1 N N i1 precision( i) and m recall 1 N N i1 recall( i), where i 1,... N represent the number of repeated iteration (in our case N = 2 or N = 3). Table 5: The list of classes generated from the two scenes (Venice and Toulouse) using Search engine based on Support Vector Machine. These classes are considered as annotated database for evaluation purposes. Class No. Semantics No. of patches No. of train patches 1 Bridge type Harbor River deposits Agriculture Distortions Mixt Vegetation and Water Vegetation Urban + Water Urban type Cemetery Water + vegetation Water + ambiguities Water Water + Boat Vegetation + Building Beach area Train lines type Grassland Forest Bridge type D3.1 KDD concepts and methods proposal: report & design recommendations 74

76 21 Water + Urban or Vegetation Road + Vegetation Structure roof Train lines type Urban type Grassland with rectangular shape Grassland with objects Building - shape Urban type Building - reflaction Vegetation + Urban Road + Building Tree + Building Parking Park with street 2 1 D3.1 KDD concepts and methods proposal: report & design recommendations 75

77 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 Class 10 Class 11 Class 12 Class 13 Class 14 Class15 Class 16 Class 17 Class 18 Class 19 Class 20 Class 21 Class 22 Class 23 Class 24 Class 25 Class 26 Class 27 Class 28 Class 29 Class 30 Class 31 Class 32 Class 33 Class 34 Class 35 Figure 31: Typical classes extracted from the images (Venice and Toulouse). D3.1 KDD concepts and methods proposal: report & design recommendations 76

78 Results of TerraSAR X detected data classification The precision-recall was computed for each class and are displayed in the next figures (as a graph) and detailed in Table App 5 - Table App 9 of section 9.3 in the Appendix We summarized the precision and recall results for each test dataset in the following: In the case of the standard MGD-SE product, the precision is shown in Figure 32 and the recall in Figure 33. In the case of the special multi-resolution MGD-SE product at 1m, the precision is shown in Figure 34 and the recall in Figure 35. In the case of the standard MGD-RE product, the precision is shown in Figure 36 and the recall in Figure 37. In the case of the special multi-resolution MGD-SE product at 2m, the precision is shown in Figure 38 and the recall in Figure 39. In the case of the special multi-resolution MGD-SE product at 4m, the precision is shown in Figure 40 and the recall in Figure 41. Precision - MGD-SE product 100% 90% 80% 70% Percent 60% 50% 40% 30% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS 2 20% 10% 0% Class Number Figure 32: The precision graph - comparison between all the investigated features for the MGD-SE product. D3.1 KDD concepts and methods proposal: report & design recommendations 77

79 Recall - MGD-SE product % 90.00% 80.00% Percent 70.00% 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 33: The recall graph - comparison between all the investigated features for the MGD-SE product. Precision - special MGD-SE product at 1m resolution % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 34: The precision graph - comparison between all the investigated features for the special MGD-SE product at 1m resolution. D3.1 KDD concepts and methods proposal: report & design recommendations 78

80 Recall - special MGD-SE product at 1m resolution % 90.00% 80.00% Percent 70.00% 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 35: The recall graph - comparison between all the investigated features for the special MGD-SE product at 1m resolution. Precision - MGD-RE product % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 36: The precision graph - comparison between all the investigated features for the standard MGD-RE product. D3.1 KDD concepts and methods proposal: report & design recommendations 79

81 Recall - MGD-RE product % 90.00% 80.00% Percent 70.00% 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 37: The recall graph - comparison between all the investigated features for the standard MGD-RE product. Precision-special MGD-SE product at 2m resolution % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 38: The precision graph - comparison between all the investigated features for the special MGD-SE product at 2m resolution. D3.1 KDD concepts and methods proposal: report & design recommendations 80

82 Recall - special MGD-SE product at 2m resolution % 90.00% 80.00% Percent 70.00% 60.00% 50.00% 40.00% 30.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS 1 QMFS % 10.00% 0.00% Class Number Figure 39: The recall graph - comparison between all the investigated features for the special MGD-SE product at 2m resolution. Precision - special MGD-SE product at 4m resolution % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS % 20.00% 10.00% 0.00% Class Number Figure 40: The precision graph - comparison between all the investigated features for the special MGD-SE product at 4m resolution. D3.1 KDD concepts and methods proposal: report & design recommendations 81

83 Recall - special MGD-SE product at 4m resolution % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% GLCM 1_2_3_4 NLFT GAFS 2_2 GAFS 4_6 QMFS % 20.00% 10.00% 0.00% Class Number Figure 41: The recall graph - comparison between all the investigated features for the special MGD-SE product at 4m resolution. In the next figure, Figure 42, the average of the recall is displayed for all considered products (values presented in detail in Table App 15 and Table App 16 of section 9.3 in the Appendix). Comparative results - the average of Recall for all the products % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% 30.00% MGD-SE product "special" MGD-SE product 1m "special" MGD-SE product 2m MGD-RE product "special" MGD-SE product 4m 20.00% 10.00% 0.00% Class Number Figure 42: The average recall graph - comparison between all the products for the 30 classes. D3.1 KDD concepts and methods proposal: report & design recommendations 82

84 Discussion of the performance results Table 6 shows a synthesis the best performance of the features (the best three features) in terms of precision-recall in the case of the standard and special multi-resolution MGD product. Table 6: The list of the best three feature extraction algorithms in terms of precisionrecall using the different TerraSAR-X products. Product name Precision Recall QMFS_ % GAFS_2_ % MGD-SE GLCM 87.22% GAFS_4_ % GAFS_4_ % QMFS_ % special MGD-SE at 1m resolution GLCM 91.03% GAFS_2_ % QMFS_ % GAFS_4_ % GAFS_4_ % NLFT 34.59% GAFS_4_ % GAFS_2_ % MGD-RE QMFS_ % GAFS_4_ % QMFS_ % NLFT 36.14% special MGD-SE at 2m resolution special MGD-SE at 4m resolution GAFS_4_ % GAFS_2_ % QMFS_ % GAFS_4_ % GLCM 88.34% QMFS_ % GLCM 97.41% GAFS_4_ % GAFS_4_ % GAFS_2_ % GAFS_2_ % QMFS_ % After the investigation and comparison between the features and products, we report the following observation: D3.1 KDD concepts and methods proposal: report & design recommendations 83

85 - The Gabor filters (GAFS) with different scales and orientation perform better than the other features especially when the recall is computed. The second performance in recall is attributed to QMFS. The QMFS has the advantage of being faster (in required run time for feature computation) than the GAFS. - One of the following algorithms can be taken into account: GAFS, QMFS or GLCM when the precision is key. - When the Gabor filters with 4 scales and 6 orientations are used as a feature vector for the Relevance Feedback SVM tool, after the first iteration (when the positive and negative examples are given by the user) the positive examples are retrieved very easily with highest probability contrary to the non-linear short time Fourier transform where the positive examples are very hard to be retrieved after many iterations. - In Table 7, we can see that the best recall (which is important for our evaluation) as average for all classes and features is obtained for the standard MGD-RE product (details for these values can be found in Appendix in Table App 5 - Table App 14). For precision at the first sight we can notice that special MGD-SE product at 4m resolution is better than the rest of the products. - Starting from the previous observation (regarding the precision), we try to improve the performances of the standard MGD-RE product in precision by sub-sampling the data. In Figure 43 and Figure 44 and respectively in Table App 15 and Table App 16 in section 9.3 of the Appendix are presented the comparative results (only for the best two features Gabor filter with 2 scales and 2 orientations and Gabor filter with 4 scales and 6 orientations) for standard MGD-RE product and the sub-sampled standard MGD-RE product. In Table 8, we can note that if the data is sub-sampled the precision is improved, but the recall is a bit smaller for this data. Taking into account that for the sub-sampled data the patch size is smaller this has direct implication on the feature computation time. Based on previous observations, we propose to use for TELEIOS project the standard MGD-RE product (having the best recall among all the products standard or special ) and as a feature extraction method the Gabor filters with different scales and orientations and/or the quadrature mirror filters with different number of levels. D3.1 KDD concepts and methods proposal: report & design recommendations 84

86 Table 7: The comparison of the precision - recall (as average for all classes and features) for all investigated products. Product name Total: all classes and features Precision Recall MGD-SE 80.72% 34.16% special MGD-SE at 1m resolution 84.61% 34.36% MGD-RE 82.27% 35.62% special MGD-SE at 2m resolution 81.26% 34.92% special MGD-SE at 4m resolution 89.46% 30.47% Table 8: The comparison of the precision - recall (as average for all classes and both versions of Gabor filters) for the standard MGD-RE product and the sub-sampled MGD-RE product. Product name Total: all classes and Gabor features Precision Recall MGD-RE 85.98% 39.64% sub-sampled MGD-RE 87.56% 38.41% D3.1 KDD concepts and methods proposal: report & design recommendations 85

87 Comparative Precision: MGD_RE vs sub-sampled MGD_RE % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% 30.00% 20.00% MGD_RE -- GAFS 2_2 sub-sampled MGD_RE -- GAFS_2_2 MGD_RE -- GAFS 4_6 sub-sampled MGD_RE -- GAFS_4_ % 0.00% Class Number Figure 43: The precision evaluation between the standard MGD-RE product and the sub-sampled MGD-RE product for two versions of Gabor filters. D3.1 KDD concepts and methods proposal: report & design recommendations 86

88 Comparative Recall: MGD_RE vs sub-sampled MGD_RE % 90.00% 80.00% 70.00% Percent 60.00% 50.00% 40.00% 30.00% 20.00% MGD_RE -- GAFS 2_2 sub-sampled MGD_RE -- GAFS_2_2 MGD_RE -- GAFS 4_6 sub-sampled MGD_RE -- GAFS_4_ % 0.00% Class Number Figure 44: The recall evaluation between the standard MGD-RE product and the sub-sampled MGD-RE product for two versions of Gabor filters. D3.1 KDD concepts and methods proposal: report & design recommendations 87

89 Evaluation method for Image Time Series The dataset using in this evaluation was previously presented in subsection This corresponds to 12 GEG-RE products over Vâlcea county in Romania. Four classes from the five identified in the image are shown in Figure 45. Each class has a specific evolution pattern. Therefore, the objective is to develop features to characterize and separate different evolution patterns. Forest Agriculture Grassland Urban Figure 45: Classes extracted from TerraSAR-X image over Vâlcea. A preliminary experiment for evolution classification was performed using the SVM. For the quantitative assessment, the confusion matrix is used to evaluate the accuracy and the results are shown in Table 9. Table 9: Confusion matrix using image time series for classification. D3.1 KDD concepts and methods proposal: report & design recommendations 88

90 5.2. Compression based Image Retrieval System using similarity metrics Content-Based Image Retrieval (CBIR) systems allow searching for an image in a large database by giving as query the image content. The term content in this context might refer to colours, shapes, textures, or any other information that can be derived from the image itself. In this context, the use of compression-based techniques in the image retrieval area is a parameter-free approach, which uses dictionaries directly extracted from the image data to be able to compress the image content. The dictionary is considered as descriptor of the image. This approach constitutes an interesting alternative to those classical methods based on setting and tuning of the parameters. The Fast Compression Distance (FCD) is a new compression-based similarity measure presented in [RI-25], which combines the speed of Patter Recognition based on Data Compression (PRDC) [RI-17] with the robustness of Normalized Compression Distance (NCD) [RI-18] achieving an increase of the speed on the large database queries. The use of FCD as similarity metrics between two images allows defining a CBIR, since the query image is compared with all the images stored into the database by comparing their dictionaries and retrieving the most similar images. This approach was presented in [RI-25] and has been used for retrieving multimedia images with satisfactory results. In this section, we extend this approach to TerraSAR-X images. In the next subsections, we present in detail the concept and logical architecture of CBIR based on a Fast Compression Distance, its main functions, its logical architecture, examples of operation and some evaluation results using TerraSAR-X Concept and Logical architecture of CBIR based on FCD A CBIR based on compression techniques can be conceptually described in Figure 46. In here, the main levels of the system are: D3.1 KDD concepts and methods proposal: report & design recommendations 89

91 Figure 46: Concept of the Content-based Image Retrieval system based on Fast Compression Distance. This concept can be seen as flowchart of the system. 1. Data level: This level corresponds to the image database, where several TerraSAR-X scenes are stored in order to be further processed. A scene is tiled into several patches following the process described in section Processing level The content of a patch is compressed by using dictionary based techniques. Later, the dictionary and the patch are stored into the database. In this level, we observe that when a query is performed, the query image is also compressed and later its dictionary will be compared using a FCD similarity metric. 3. Query level In the next level, the query is performed by using a similarity measure (FCD), which is computed by comparing dictionaries between the images. 4. Retrieval level The final level shows an ordered list of the retrieved images. This list is ranked according to the level of similarity. Logical architecture In order to support these levels, the logical architecture of the CBIR system based on compression techniques is shown in Figure 47. In here, the boxes with solid line represent components that have already been implemented, while the boxes with dashed line are the components that will be implemented. D3.1 KDD concepts and methods proposal: report & design recommendations 90

92 Figure 47: Logical architecture of the CBIR system based on compression methods. D3.1 KDD concepts and methods proposal: report & design recommendations 91

93 The logical architecture of the CBIR can be seen as a set of the following components (frameworks): Pre-processing framework Different pre-processing steps should be performed according to the type of image. For example, in the case of colour images is needed to convert from RGB (red, green, blue) colour model to HSV (hue, saturation, and value) colour space. In the case of TerraSAR-X is needed to convert from 16 bits to 8 bits. Analysis framework The analysis framework covers the different dictionary extraction methods and compression techniques. As for example in the case of multimedia data or image time series, the MPEG-7-VST [RI-27] is used, which is a standard for description and search of audio and visual content. In the case of TerraSAR-X images, the Lempel- Ziv-Welch (LZW) [RI-16] compression algorithm is used to extract the relevant dictionaries from the analyzed images and compress their content. Both techniques, MPEG-7-VST and TIFF-LZW are based on codebooks analysis. It is worth noting that more compression techniques can be implemented and added in the system. Indexing framework The image content can be indexed either based on the complexity measures or classes defined by the user. Query and ranking framework. The query is based on similarity measures, which are derived from dictionaries directly extracted from the data. Indeed, the similarity measures can be considered as the core of the retrieval system. In the current system implementation, the FCD is used as similarity metric. However, another metrics as for example the PRDC or a user customized similarity measure can be added. Here, it is important to note that both the indexing and the querying are stored and executed into Monetdb database. The interaction with the end-user is performed by using Graphical User Interface (GUI) based on web technologies. Comparison or evaluation framework The results of the query are evaluated by computing performance measures, which can be either single query measures or statistical measures as for example: precision and recall. This framework will provide a view about the performance of the system in retrieving images with similar content. D3.1 KDD concepts and methods proposal: report & design recommendations 92

94 Implementation and operation of CBIR based on FCD A scheme of the implementation is presented in Figure 48. From the implementation point of view, CBIR-FCD relies on the processing chain, which starts by tiling the images into patches according to process described in section 3.1.1, computing the dictionaries using LZW [RI-16] compression method, and storing them into the database. As results this algorithm provides a text file containing the dictionary elements, which later will be inserted into the database. LZW algorithm is implemented in C. A python script reads the dictionary text file and later using SQL sentences inserts it into a table in database. MonetDB [RI-26] is used as Database Management System (DBMS). Image tiling Display available patches Dictionary extraction Insert into DB Similarity metric Select query image Figure 48: Flowchart of the CBIR-FCD system implementation. Since the patches and their dictionaries are available in the database, then, a GUI implemented in python and django adapter is used to display the available patches. In here, at the beginning, the python script randomly performs queries to the database showing a list of images. Later, the end-user can search for relevant/similar images using their content as example. The user selects the query image by clicking on it and this request is processed by a script, which recovers the patch id and performs the SQL query. In order to retrieve and display the resulting images, a compression-based similarity metric, the FCD (described in section ) is computed. FCD algorithm is implemented in SQL sentences Experimental results of CBIR based on FCD The CBIR-FCD relies on web interface as shown Figure 49 and Figure 50. The examples presented below use optical images (IKONOS sub-scene over Rome) in order to make the visibility of the results easier; however the experimental part was performed using TerraSAR-X images. Here, two possible queries can be performed. D3.1 KDD concepts and methods proposal: report & design recommendations 93

95 Query-based on image content as example. Query-based on semantic labels. In the case of query-based on image content, the end-user selects an image, which is passed as content example for the query. Later the system internally computes the FCD (see equation 5) between the query patch and the other patches stored in the database and display the results ordered according to the short distance. Figure 49 presents an example. In here, the user selects an image containing houses and the retrieved images are shown below. From the results, we can observe most of the retrieved images are correct since they contain roofs. In the case of query-based on semantic label, the end-user introduces a semantic label and the system queries the database with this label. The result is a list of patches associated with the semantic label. An example is shown in Figure 50. Here, the user introduced houses as query label and the system retrieved all associated patches. It is important to note that the patches were previously manually annotated with a semantic label and these labels are stored into the database as part of the patch information. In the following, the results of different queries using the CBIR-FCD and TerraSAR-X images are described. Figure 49: Results of query based on image content. Houses in Rome were selected as example and the retrieved images are presented below. D3.1 KDD concepts and methods proposal: report & design recommendations 94

96 Figure 50: Results of query based on fixed labels. The label houses was introduced and the retrieved images are presented below Experimental results using TerraSAR X images Two TerraSAR-X scenes were used for querying the database based on the image content. The first image corresponds to Venice and the second one to Toulouse. Both images are MGD, spatially enhancement (MGD-SE) at 1m of resolution standard product, previously describe in subsection as Dataset 1. As pre-processing step, both images were converted to 8 bits (by using the logarithm to scale from 16 to 8 bits) in order to properly apply the algorithms. Later, both images were tiled into patched with 200x200 pixel size, then extracted their dictionary, and stored into the database. The test data set is summarized in Table 10 D3.1 KDD concepts and methods proposal: report & design recommendations 95

97 Table 10: Test data set description. Both scenes, Venice and Toulouse were tiled into patches with 200 pixel sizes. TSX MGD- SE product Total patches Patch size Semantic labels Venice x Toulouse x The patches were annotated with a semantic label by using the Search Engine based on SVM tool and user supervision previously presented in section 5.1. The semantic labels associated to the selected classes were previously described in Table 5. In the following, we present some examples of retrieving TerraSAR-X structures using both images. Table 11 displays the query images and the 20 top retrieved images. Some quality metrics (Precision and Recall) were computed from these results and they are summarized in Table 12. Query images Retrieved images Class9 Class6 Class7 Class36 D3.1 KDD concepts and methods proposal: report & design recommendations 96

98 Class20 Class31 Class28 Class32 Table 11: Results of the queries based on image content using CBIR-FCD as data mining tool. Table 12 shows the precision and recall for the classes and the query time in seconds needed for searching and retrieving the results. Table 12: Precision and recall of the semantic classes using query based on content and the query time. Class Precision (%) Recall (%) Query time (sec) Class1 5,36 5, Class2 10,71 10, Class3 5,36 5, Class4 7,14 6, Class5 8,93 8, Class6 0,00 0, Class7 1,79 1, Class8 8,93 8, Class9 41,07 39, Class10 0,00 0, D3.1 KDD concepts and methods proposal: report & design recommendations 97

99 Class11 12,50 12, Class12 39,29 37, Class13 3,57 3, Class14 0,00 0, Class15 23,21 22, Class16 0,00 0, Class17 0,00 0, Class18 0,00 0, Class19 0,00 0, Class20 0,00 0, Class21 8,93 8, Class22 1,79 1, Class23 5,00 3, Class24 5,36 5, Class25 5,00 3, Class26 0,00 0, Class27 5,00 1, Class28 10,00 7, Class29 44,64 44, Class30 1,79 1, Class31 0,00 0, Class32 30,00 23, Class33 5,00 1, Class34 0,00 0, Class35 7,14 7, The results show that CBIR based on FCD can not recognize 11 classes, since their precision is less than 1%. However, 16 classes were recognized with less than 10% of precision, 4 classes with less than 30% and 4 classes with less than 50%. At this point in the implementation of CBIR-FCD, the results are not satisfactory since their precision values are lower than 50%. However, this innovative technique has achieved promising results using multimedia images [RI-15, RI-25], for this reason more research will be done to adequate it to TerraSAR-X images and to improve the results. It also will be tested with multi-resolution images and different patch sizes Summary In this chapter, we presented two software implementations based on the concept of query by example. The Search Engine based on Support Vector Machine and relevance D3.1 KDD concepts and methods proposal: report & design recommendations 98

100 feedback was used for annotating image patches as well as for evaluating the relevant image descriptors. Based on previous experimental work, we propose to use for TELEIOS project the standard MGD-RE product (having the best recall among all the products standard or special multi-resolution) and as a feature extraction method the Gabor filters with different scales and orientations and/or the quadrature mirror filters with different number of levels. In the case of the second implementation, Content-based Image Retrieval based on Fast compression distance, unfortunately at this stage on the project, the query results are not satisfactory in terms of precision and recall (lower than 50%). However, it is important to observe that the time for querying and retrieving the results is very good, which is an important condition for query and data mining. Our conclusion is that the current implementation needs to be improved by developing a more suitable dictionary extraction or pre-processing methods for TerraSAR-X. D3.1 KDD concepts and methods proposal: report & design recommendations 99

101 6. Conclusions In this deliverable we presented the Knowledge discovery framework in Earth- Observation (EO). First we discussed the different components and the connection with the TELEIOS architecture. In particular, we described two data sources in detail in order to perform the content analysis in the next step. Then, we introduced the Data Model generation in terms of the content analysis of TerraSAR-X images and their metadata. The Data Model generation is related with the ingestion facility in TELEIOS architecture. The image content analysis was performed through methods for feature extraction using detected data, image time series, and multi- resolution products. We carried out a quality assessment of the image descriptors in order to ensure the performance and efficiency in data mining and knowledge discovery functions. As part of the evaluation, we also studied the behaviour of the image descriptors when different resolutions, scales and patch sizes are used. We proposed to work with image tiles for the feature extraction instead of pixels, and from the experimental results we conclude that while the traditional feature extraction methods are able to distinguish between 10 classes, with our new proposed approach the number of semantic classes has been increased to 30. We implemented two software tools (Search Engine, whose main core is the Support Vector Machine (SVM) and Content Based Image Retrieval (CBIR) based on the Fast Compression Distance) in order to apply the concept of query based on image content as example and to have a starting point for semantic discovery and automatic image annotation. In the forthcoming deliverables, we plan to complete the Data Model Generation presented in Chapter 3 and the database schema for descriptors presented in Chapter 4 by adding the GIS information and their relevant descriptors, and study the context analysis. Moreover, we expect to have a contribution related to visual data mining by improving the methods that are under development. D3.1 KDD concepts and methods proposal: report & design recommendations 100

102 7. Applicable and Reference Documents 7.1. Applicable Documents Document Title Internal Referenc e Katrin Molch et al., (2010). Naming Convention for Image Patches. [RI- 1] N. Ritter and M. Ruth, (1995) GeoTIFF Format Specification GeoTIFF, [RI- 2] Revision 1.0, Version Robert M Haralick, K Shanmugam, Its'hak Dinstein (1973). "Textural Features for Image Classification". IEEE Transactions on Systems, Man, and [RI- 3] Cybernetics, SMC-3 (6): The GLCM Tutorial [RI-4] H.G. Feichtinger, Th. Strohmer (1998). "Gabor Analysis and Algorithms", [RI-5] Birkhäuser. B.S. Manjunath, W.Y. Ma (1996). Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine [RI-6] Intelligence, 18 (8): A. Popescu, I. Gavat, M. Datcu (2008). "Complex SAR image characterization [RI-7] using space variant spectral analysis". IEEE Radar Conference, 1-4. T. Li, M. Ogihara (2006). Towards Intelligent Music Information Retrieval. [RI-8] IEEE Transactions on Multimedia, 8(3), A. Croisier, D. Esteban, C. Galand (1976). "Perfect channel splitting by use of [RI-9] interpolation / decimation / tree decomposition techniques". International Conf. on information Science and Systems, Patras, Greece: P.P Vaidyananthan (1987). Quadrature Mirror Filter Banks, M-Band [RI-10] Extensions and Perfect-Reconstruction Techniques. IEEE ASSP Magazine, 4 (3): M. Campedel, E. Moulines, M. Datcu (2005). Feature Selection for Satellite [RI-11] Image Indexing. ESA-EUSC: Image Information Mining Theory and Application to Earth Observation. N. Karthikeyani Visalakshi, K. Thangavel (2009). Impact of Normalization in Distributed K-Means Clustering. International Journal of Soft Computing, 4 [RI-12] (4): D. Dai, W. Yang, H. Sun (2011). Multilevel Local Pattern Histogram for SAR Image Classification, IEEE Geosciences Remote Sensing Letters, 8 (2): 225 [RI-13] 229. S. Cui, M. Datcu, (2011) Coarse to Fine Patches-based Multitemporal Analysis [RI-14] of Very High Resolution Satellite Images, Multitemp 2011, to be published. D. Cerra, M. Datcu, (2010). Image Retrieval using Compression-based Techniques, Proceedings of the International ITG Conference on Source and [RI-15] Channel Coding, Siegen, Germany. Welch, T. (1984). Technique for high-performance data compression, [RI-16] Computer 17(6): D3.1 KDD concepts and methods proposal: report & design recommendations 101

103 Watanabe, T., Sugawara, K., Sugihara, H. (2002). A new pattern representation scheme using data compression, IEEE Transactions on [RI-17] Pattern Analysis and Machine Intelligence 24(5): Li, M., Chen, X., Li, X., Ma, B., Vitanyi, P. M. B. (2004). The similarity [RI-18] metric, IEEE Transactions on Information Theory 50(12): Shyu, C.-R., Klaric, M., Scott, G., Barb, A., Davis, C., Palaniappan K., (2007). GeoIRIS: Geospatial Information Retrieval and Indexing System [RI-19] Content Mining, Semantics Modeling, and Complex Queries, IEEE Transactions on Geoscience and Remote Sensing 45 (4): Popescu, A., Gavat, I., Datcu, M., (2010), Image Patch Contextual Descriptors for Very High Resolution SAR data: A Short Time Fourier [RI-20] Transform Non-Linear Approach, (submitted to IEEE - letters). Birjandi, P., Datcu, M., (2009), ICA based visual words for describing under meter high resolution satellite images, Proceeding of. IGARSS [RI-21] 2009, Cape Town. Birjandi, P., Datcu, M., (2010), Patch Contextual Descriptors for Very High Resolution Satellite Images: A Topographic ICA Approach, (to be [RI-22] published). Zucker, S. W. and K. Kant, (1981), Multiple-level Representations for Texture Discrimination, Proceedings of the IEEE Conference on Pattern [RI-23] Recognition and Image Processing, Dallas, H. Breit, M. Eineder, T. Fritz, B. Schattler, M. Huber, and J. Mittermayer, (2006), TerraSAR-X Products and Product Performance Update, in [RI-24] IGARSS, pp D. Cerra (2010). Pattern-Oriented algorithmic complexity: Towards compression-based information retrieval, Ph.D. dissertation, Telecom [RI-25] ParisTech University, Paris, 2010 Monetdb, Copyright (c) , Available from: [RI-26] José M. Martínez, (2004), MPEG-7 Overview (version 10), [RI-27] ISO/IECJTC1/SC29/WG11 N6828 Palma de Mallorca, October 2004 Ugo Di Giammatteo, Sergio Perelli, Manuela Sagona (2011). The TELEIOS software architecture- Phase I and preliminary TELEIOS infrastructure, [RI-28] Deliverable D1.2.1 by ACS Michael Sioutis (2011), WP3 Knowledge Discovery Semantic [RI-29] Technology in the DLR Use Case,Technical report by NKUA 7.2. Reference Documents Internal Document Title Identifier Reference Basic Products Specification Document, Issue: 1.6 TX-GS-DD-3302 [RD-1] Level 1b Product Format Specification, Issue: 1.3 TX-GS-DD-3307 [RD-2] D3.1 KDD concepts and methods proposal: report & design recommendations 102

104 7.3. Other References Papers/Publications Internal Reference [RO-1] [RO-2] D3.1 KDD concepts and methods proposal: report & design recommendations 103

105 8. Acronyms and Abbreviations DEM Digital Elevation Model DLR German Aerospace Agency EEC Enhanced Ellipsoid Corrected ENL Equivalent Number of Looks ESA European Space Agency GEC Geocoded Ellipsoid Corrected H Horizontal Polarization HS High Resolution spotlight Mode L1b Level 1b MGD Multi-Look Ground-range Detected RE Radiometrically-Enhanced S Single Polarization SAR Synthetic Aperture Radar SC ScanSAR Mode SE Spatially-Enhanced SL spotlight Mode SM Stripmap Mode SNR Signal to Noise Ratio SRA Single Receive Antenna Configuration SSC Single Look Slant Range Complex TMSP TerraSAR-X Multi-Mode SAR Processor TN Technical Note TSX TerraSAR-X (X-band SAR) D3.1 KDD concepts and methods proposal: report & design recommendations 104

106 UPS Universal Polar Stereographic UTM Universal Transverse Mercator V Vertical Polarization XML - Extensible Markup Language Internal notations for this document (algorithms, patches, etc): DET DETected Data FCD Fast Compression Distance GAFS Gabor filters GLCM Grey-Level Co-occurrence Matrix ITS Image Time Series data LPHM Local Pattern Histogram NCD Normalized Compression Distance NSFT Non-linear Short Fourier Transform PAR for feature extracted parameters (PAR) PRDC Patter Recognition based on Data Compression QLK for quick look QMFS Quadrature Mirror Filters D3.1 KDD concepts and methods proposal: report & design recommendations 105

107 9. Appendix 9.1. Dataset structure for TerraSAR X detected data The TSX dataset is stored in three sub-folders depending by the product type (only the MGD products are considered for this section). A screenshot of the dataset content can be seen in Figure App.1. With the green colour are highlighted the useful information need for DMG. Dataset example for the standard MGD-SE product: Product:TSX1_SAR MGD_SE HS_S_SRA_ T165907_ T Product_Content: - ANNOTATION - AUXRASTER - DET - DET_all - IMAGEDATA - PAR - PAR_all - PREVIEW - QLK - SUPPORT - TSX1_SAR MGD_SE HS_S_SRA_....xml The DET_all folder contains two sub-folders, namely PATCHES_tiff (patches on 16 bits) and PATCHES_ima (patches converted to 8 bits). In the same time when the tile patches are generated, the quick-looks of these patches are also saved in QLK folder in a.jpg format. D3.1 KDD concepts and methods proposal: report & design recommendations 106

108 Another folder which will be created separately after the feature extraction methods are applied is the PAR_all folder. All the files of a specific algorithm are saved in separately folders. For example in the PAR_all folder we have: GLCM_orient_1 (which means Grey-Level Co-occurrence Matrix with orientation 1), NSFT (which means non-linear short time Fourier transform, GAFS_sc2_or2 (which means Gabor filters with scalegausian equal to 2 and orientations equal to 2), QMFS_NbLev_1 (which means quadrature mirror filters with number of levels equal to 1). Note: Two other folders ( DET and PAR ) are created specially for the Relevance Feedback SVM tool. The first folder contains the patches in GeoTIFF format (identically with the ones store in PATCHES_tiff folder) and a set of features extracted using the methods described in section Similar dataset structure like the one shown in Figure App.1 can be create for ITS Metadata product extracted from the TerraSAR X XML file The useful information extracted from the XML file of the standard and special products and used for processing is displayed in 4 tables (Table App 1 - Table App 4) Precision Recall results for TerraSAR X detected data In the Table App 5 - Table App 14 the precision-recall is displayed for all the MGD products and investigated features. The results are computed only for the classes with more then 10 patches the others are removed from the investigation. D3.1 KDD concepts and methods proposal: report & design recommendations 107

109 Note: With red colour is mark the best result obtained for each class, and with the blue colour is represented the average of the precision or recall for each class or feature. Finally, with orange colour is represented the global average of the precision or recall for entire product (this means for all investigated classes and features). D3.1 KDD concepts and methods proposal: report & design recommendations 108

110 Figure App 1: Screenshot of the content of the TerraSAR-X dataset for detected data. D3.1 KDD concepts and methods proposal: report & design recommendations 109