The novel tool of Cumulative Discriminant Analysis applied to IASI cloud detection

Transcription

1 Applied Spectroscopy The novel tool of Cumulative Discriminant Analysis applied to IASI cloud detection G. Masiello, C. Serio, S. Venafra, SI/UNIBAS, School of Engineering, University of Basilicata, Potenza, Italy U. Amato, I. De Feis IAC/CNR, Ist. Applicazione del Calcolo, Napoli, Italy L. Lavanant Centre de Meteorologie Spatiale, Lannion, France R. Stuhlmann, S. Tjemkes EUMETSAT, Darmstadt, Germany 1

2 Outline 1. Description of Test Statistics Thermal Contrast Shape 2. Methodology to combine statistics in a single cloud mask CDA N-Dimensions 3. Results from the analysis with two globally distributed IASI database D_DIFA1 and D_DIFA2 4. Conclusions and Future development 2

3 Cloud detection basics In the thermal infrared cloud detection relies on the fact that normally clouds are colder than the underlying surface (thermal contrast) sky spectra tend to be different in shape of those in cloudy conditions 1. Description 2. Methodology 3. Results 4. Conclusion 3

4 Test Statistic Method hs Based on shape similarity between a couple of spectra (Observed and Reference) T0 Based on the difference T2-T0, with T2, the Brightness 900 cm-1 Split window test based on CO2 791 cm-1, ΔTCO2=T4-T5, with T4 and T5 the brightness cm-1 and cm-1 Spatial Coherence test sh Based on a cluster of n n nearby pixels. Let {T0}i, i=1,..,nxn, The set of T0 corresponding to a given n n clusters, sh is the standard deviation of this set. 4. Conclusion ΔTCO2 3. Results Window slope test W Based on the Brightness Temperature, 833 cm-1 contrasted against a suitable threshold 2. Methodology Surface temperature tests χ2 Based on χ2 like variable defined on a couple of skin temperature values (Ts, TsR), with Ts, estimated from the spectrum and TsR a suitable reference 1. Description Cloud detection scheme: The baseline 4

5 T 0 : window brightness temperature, 833 cm -1 T CO2 split test (sensitive to surface pressure as well) Normalized thermal contrast χ 2 test Thermal Contrast Type of statistics h s test window slope test, W 0 K < BT(900)-BT(833) <1.5 K Spatial Coherence sh Shape & Morphology 1. Description 2. Methodology 3. Results 4. Conclusion 5

6 Thermal Contrast Alone Full SCA 2. Methodology All data 1. Description Udine exercise 3. Results 4. Conclusion Upper panels show the scatter plot of ECMWF Skin Temperature against the one derived from the spectra. Morphology Tests detect cirrus cloud contaminated spectra classified clear by Thermal Contrast tests alone. 6

7 Discriminant Analysis and more generally classification methods formally rely on the minimization of an objective (cost) function that penalizes misclassifications of the training dataset. Discriminant analysis I-type error is defined as the fraction of pixels being and retrieved as Cloudy (False negatives). II-type error the fraction of pixels being Cloudy and classified as (False positives). 1. Description 2. Methodology 3. Results 4. Conclusion 7

8 I and II type errors in terms of the cumulative density function I-type error is defined as the fraction of pixels being and retrieved as Cloudy (False negatives). II-type error the fraction of pixels being Cloudy and classified as (False positives). 1. Description 2. Methodology 3. Results 4. Conclusion 8

9 A novel approach: Cumulative Discriminant Analysis (CDA) CDA is based on the cumulative distribution function CDA is based on the following requirements: Minimization of both errors of the I and II type Unbiased nonparametric estimate Advantages Fully non parametric estimates of the cdf ECDF, Empirical Cumulative Density Function Consistent estimator (surely converges to the true cumulative density function) Uniform convergence over x 1. Description 2. Methodology 3. Results 4. Conclusion 9

10 Basic of the methodology Cost Function 1. Description 2. Methodology 3. Results 4. Conclusion 10

11 Basic of the methodology Two Dimensions 1. Description 2. Methodology 3. Results 4. Conclusion 11

12 Basic of the methodology: Three Dimensions 1. Description 2. Methodology 3. Results 4. Conclusion 12

13 Basic of the methodology: Four and more Dimensions 1. Description 2. Methodology 3. Results 4. Conclusion 13

14 DIFA Database for training, cross-check and validation We identified and developed two data sets of IASI spectra that are: Qualified for sky-type (clear/cloudy) through the CMS cloud mask, which is based on the co-location of the IASI footprint with AVHRR imagery. Complemented with skin temperature fields from the ECMWF analysis, co-located in space and in time with IASI footprints. D_DIFA1: IASI spectra (8 orbits) 12 h global acquisition on 18 November 2009 D_DIFA2: IASI spectra (9 orbits) 15 h global acquisition on 22 July Description 2. Methodology 3. Results 4. Conclusion 14

15 4. Conclusion D_DIFA2, 22/07/07 3. Results Land 2. Methodology D_DIFA1, 18/11/09 Sea 1. Description Cloud mask D_DIFA1 & D_DIFA2 (reference AVHRR cloud mask, Grey Cloudy, Blue ) 15

16 1. Description Tropical, land surface DIFA-SCA (D_DIFA1 + D_DIFA2) Cloudy Total (83.9 %) (16.1 %) (100 %) Cloudy (15.4 %) (84.6%) (100 %) 4. Conclusion CMS 3. Results 84.4 % 2. Methodology Total Score n11+n22 = n12+n21 16

17 1. Description Mid-Lat Summer, sea surface DIFA-SCA (D_DIFA1 + D_DIFA2) Cloudy Total (84.2 %) (15.9 %) (100 %) Cloudy (13.6 %) (86.4%) (100 %) 4. Conclusion CMS 3. Results 86.1 % 2. Methodology Total Score n11+n22 = n12+n21 17

18 1. Description Summary of results: D_DIFA1 2. Methodology 3. Results 4. Conclusion 18

19 Cloudy Total 7310 (87%) 1064 (13%) 8374 (100%) Cloudy 194 (13%) 1288 (87%) 1482 (100%) Land : tropics IASI-DIFA cloud mask SEVIRI Cloud mask (99%) (99%) (1%) (1%) (100%) Cloudy Cloudy (8%) (8%) (92%)(92%)(100%) Total score = 96.6% IASI-DIFA cloud mask Cloudy Total 5188 (95%) 289 (5%) 5477 (100%) Cloudy 22 (0.7%) (99.3%) (100%) 4. Conclusion SEVIRI SEVIRI IASI-DIFA IASI-DIFA Cloud Cloudma cloudcloud maskmask mask sk Total Cloudy Cloudy 3. Results Total score = 87% 2. Methodology SEVIRI Cloud mask 1. Description D_DIFA2 data set for AFRICA Validation with SEVIRI Cloud Mask Total score = 96.4% 19

20 Concluding remarks We have invented a new tool CDF and developed the methodology to combine more statistics in a single cloud mask. Static Application Data has been generated for land and sea surface Once compared with suitably filtered AVHRR cloud masks we have a total agreement, which is better than 85% for any kind of surface 1. Description 2. Methodology 3. Results 4. Conclusion 20

21 1. Description Future works for the Geostationary Platform 2. Methodology 3. Results Time Continuity 4. Conclusion 21