Classification (or thematic) accuracy assessment. Lecture 8 March 11, 2005

Transcription

1 Classification (or thematic) accuracy assessment Lecture 8 March 11, 2005

2 Why and how Remote sensing-derived thematic information are becoming increasingly important. Unfortunately, they contain errors. Errors come from 5 sources: Geometric error still there None of atmospheric correction is perfect Clusters incorrectly labeled after unsupervised classification Training sites incorrectly labeled before supervised classification None of classification method is perfect We should identify the sources of the error, minimize it, do accuracy assessment, create metadata before being used in scientific investigations and policy decisions. We usually need GIS layers to assist our classification.

3 training vs. ground reference Several ways to do error evaluation Based on training pixels (areas) The problem is that the locations of training sites are usually not random. They are biased by analyst s a priori knowledge of where certain LULC types exist in the scene. This will results in higher classification accuracies than the one below Based on ground reference pixels These sites are not used to train the classification algorithm and therefore represent unbiased reference information It is possible to collect some ground sites prior to the classification, perhaps at the same time as the training data But majority of test reference is often collected after classification. Landscape often change rapidly. Therefore, it is best to collect both the training and ground reference as close to the data of remote sensing data acquisition as possible. (for example, agriculture crops change fast)

4 Cont To visit all ground reference sites is sometimes difficult, or even inaccessible (extremely rugged terrain, private land owners, government agencies). In this case, to use high spatial resolution image (need almost the same time for agriculture, same month or season for forest, desert, and urban area) When compare the image pixel to ground reference, to minimize the geometric misregistration, we usually not only look at one pixel, but 3x3, 5x5 pixels for labeling the pixel according to the class that has highest frequency of occurrence. For example, there are 3 corn pixels and 6 soybean pixels in the 3 x 3 window, then we main label the pixel soybean for error matrix evaluation.

5 Sample size How many samples needed for assessing the classification accuracy is an important consideration Based on binomial probability theory Based on multinomial distribution 2 z ( p)( q) N = 2 E 2 2 (85((15) = = N is the sample size, p is the expected percent accuracy of the entire map, q=100-p, E is the allowable error, and Z=2 (σ) covering 95.4% of image. If your expected accuracy is 85% at an allowable error of 5%, the number of points for a reliable results is 203. Based on binomial probability theory

6 Based on multinomial distribution N = BΠ i (1 Π b 2 i i ) Π i is the proportion of a population in the ith class out of k classes that has the proportion closest to 50%, b i is the desired precision for this class, B is the upper (α/k) x 100th percentile of the chi square (χ 2 ) distribution with 1 degree of freedom, and k is the number of classes. For example, k=8, and we know Π i is closing 30% of the total population, we desire a level of 95% confidence and a precision (b i ) of 5%, 1- α/k = /8 = , so B= χ 2 (1, ) = BΠi (1 Π ) N = b 7.568(0.3)(1 0.3) i = = 2 i 636 samples So randomly 80 samples per class are required (8 x 80 = 640) If we have no idea about the proportion of any of the classes in the image, then we can use the worst-case multinomial distribtuion algorithm where we assume that one class occupies 50% of the study area: B N = = = 757 samples (0.05 ) b i So randomly 95 samples per class are required (8 x 95 = 760)

7 Sampling design Where we will locate the samples in the real world? Should be randomly without bias. Five common sampling designs Random sampling Systematic sampling Stratified random sampling Stratified systematic unaligned sampling, and Cluster sampling

8

9 Example: they took 407 samples (pixels) based on the stratified random sampling after classification. First made 5 files (each contain one class), using a random number generator to get points.

10 Bases of Error (Confusion) Matrix Producer (analyst) accuracy is a measure indicating the probability that the classifier has labeled an image pixel into Class A given that the ground truth is Class A. it is the probability of a reference pixel being correctly classified. Omission error represent pixels that belong to the ground truth class but that the classification technique has failed to classify them into the proper class. User accuracy is a measure indicating the probability that a pixel is Class A given that the classifier has labeled the pixel into Class A. it is the probability that a pixel classified on the map actually represents that category on the ground. Commission error represent pixels that belong to another class but are labeled as belonging to the class. Overall accuracy is total classification accuracy. Kappa coefficient (K hat ) is a discrete multivariate technique of use in accuracy assessment. K hat >80% represent strong agreement and good accuracy. 40%-80% is middle, <40% is poor.

11 Conditional K hat coefficient of agreement Kˆ i = N N xii x x x i+ i+ i+ x x + i + i ˆ = = K resid 75%

12 Fuzzification of the error matrix Hard classification and fuzzy classification Hard classification + fuzzy error matrix Fuzzy logic used only during the phase when ground reference information is obtained Fuzzy classification + fuzzy error matrix

13 USGS and NIMA land cover maps and accuracy USGS LULC map (1:250,000 and 1:100,000) is a 9 categories and 37 sub-categories based on manual interpretation of 1970 s and 1980 s aerial photography. The successor to LULC is the USGS's National Land Cover Data (NLCD). Unlike LULC, which originated as a vector data set in which the smallest features are about ten acres in size, NLCD is a raster data set with a spatial resolution of 30 meters (i.e., pixels represent about 900 square meters on the ground) derived from Landsat TM imagery of cloud-free scenes acquired during the spring and fall (when trees are mostly bare of leaves) of The steps involved in producing the NLCD include preprocessing, classification, and accuracy assessment, Selected scenes are geometrically and radiometrically corrected, then combined into sub-regional mosaics comprised of no more than 18 scenes. All mosaics are then projected to the same Albers Conic Equal Area projection (with standard parallels at 29.5 and 45.5 North Latitude, and central meridian at 96 West Longitude) based upon the NAD83 horizontal datum. An unsupervised classification algorithm is then applied to the preprocessed mosaics to generate 100 spectrally distinct pixel clusters. Using aerial photographs and other references, image analysts at USGS then assign each cluster to one of the classes The USGS has hired private sector vendors to assess the classification accuracy of the NLCD by checking randomly sampled pixels against manually interpreted aerial photographs. Results from the first four completed regions suggest that the likelihood that a given pixel is correctly classified ranges from only 38% to 62% The NLCD is distributed by state on CD-ROM, and is available at nominal cost from the USGS through its Land Cover Characterization Program. For the rest of the world, Earthsat has been contracted by the U.S. National Imagery and Mapping Agency (NIMA) to produce 30m Landsat GeoCover land-cover maps using unsupervised classification to classify a TM image of early 1990 s into 240 clusters, then combine them into 13 land-cover classes, all scenes within a UTM zone are mosaicked and edgematched. GeoCover LC is under independent accuracy assessment evaluation. To date, accuracies calculated through a polygon count methodology have averaged better than 70%

14 Fuzzy accuray assessment for GeoCover LC map

15 The deterministic K hat = 37%

16 An example of GIS layers used to assist classification

17 El Paso County Total 386 scenes of IKONOS

18 1. Base Map: Mosaicked IKONOS image 2. Green and White Polygons: El Paso s Agriculture Fields El El Paso Paso Lower Lower Valley Valley Irrigation Irrigation Area Area 3. Areas that Remain: Residents, Schools, Roads, Buildings, Irrig.. Networks Cd.. Juarez, Mexico Outside Irrigation Boundaries

19 Mask Building: IKONOS Image

20 Mask Building: Road Boundary

21 Mask Building: Small Base Irrigation Map Ditches

22 Mask Building: Large Irrigation Canals

23 Mask Building: Parcel Map

24 Mask Building: Combined Layers

25 Mask Building: Agriculture Areas Remain

26 Agriculture areas identified throughout the Irrigation District

27 Selecting ROIs Alfalfa Cotton Grass Fallow Chili

28 Background: ETM+, 7/15/01 Top image: IKONOS, Oct, 2000 Ground truth

29 Error Matrix Ground truth c l a s s i f i c a t i o n Omission error Grass Alfalfa Cotton Chili Fallow (corn) 16.7 % 4.5% 4.5% 54.4 % 0% total Grass Alfalfa Cotton Chili Fallow total Producer s accuracy 83.3 % 95.5 % 99.5 % Overall accuracy=1686/1768=95.4% 45.7 % 100% Kˆ = N k User s accuracy i = 1 N = 92.5 % 95.6% 4.4% 82.7% 95.0% 89.4% 100% 2 x ii k i k i ( x ( x i + Commission error 17.3% 5.0% 10.6% 0% ( = ( ) i + x x + i + i ) )...)

30 The mask used for classification using TM image TM, 30 meter

31 IKONOS, 1 meter

32 Classifying TM Imagery

33 Classify Active Crops

34 Classify Active Crops and Fallow Lands TM image Bands: 342 April 7, 1991

35 Classifying TM Imagery

36 Active Crops TM image Bands: 342 July 15, 2001

37 Need day Integrated RS/GIS classification technologies based on NDVI RS GIS

38 Classification Results In 10 Years ETM TM Image and Time Crops Fallow Total IKONOS, 2000 (Aug.-Oct.) 35,319 11,476 46, Jul 15 42,168 4,894 47,062 Apr 26 14,163 32,898 47,061 Sep 07 38,358 8,704 47,062 Jun 10 38,203 8,952 47, Sep 12 42,451 4,610 47, Aug 13 45,068 3,543 48,611 Jul 12 41,795 6,973 48,868 Mar 22 8,852 39,977 48, Jul 23 41,377 7,520 48, Apr 15 15,206 33,752 48, Mar 27 11,825 37,303 49, Jun 26 33,495 15,774 49,269 Apr 07 9,435 39,958 49,393

39 Accuracy check by using Confusion Matrix based on ground truth information

40 In the past 10 years, Urbanization has reduced agriculture lands Red area = 2,164 acres Upper Valley Lower Valley