Class Accuracy QGIS Plugin

Transcription

1 Class Accuracy QGIS Plugin Pete Bunting Aberystwyth University Earth Observation and Ecosystem Dynamics Group Department of Geography and Earth Sciences

2 2 This work (including scripts) is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit creativecommons.org/licenses/by/4.0/.

3 Contents 1 Introduction Installation Source Code Running the Class Accuracy Plugin Create Accuracy Points What data to use as reference Use Class Accuracy Plugin Select Vector Layer Select Columns Start Analysis Go to Feature Add Classes Assign Points to Class Ending Analysis Worked Example: Classifier Comparison Create a Classification Generate Accuracy Points Run the Class Accuracy Plugin Create Error Matrix Conclusions 17 3

4 Chapter 1 Introduction This is a brief introduction to the Class Accuracy plugin written for speeding up the process of accessing classification accuracy. This plugin complements the functions which are within the RSGISLib software (Bunting et al., 2014) to produce classification accuracy points and image classification. The Class Accuracy plugin aims to make it fast and as painless as possible to undertake an accuracy assessment with many points. While this is never a quick task it could be sped up a lot of existing tools, hence this plugin. If you have any questions/queries/comments/suggestions then please get in touch via the RSGISLib mailing list. 1.1 Installation To install the plugin 1. Download the latest version ( downloads/) 2. Extract the downloaded zip file into your QGIS plugins directory which under your home directory (~/.qgis2/python/plugins/) 3. Open QGIS and Select Plugins > 'Manage and Install Plugins... > Installed and tick on the ClassAccuracy plugin. You should now have the Class Accuracy plugin installed (Figure 1.1) 4

5 CHAPTER 1. INTRODUCTION 5 (a) Class Accuracy Plugin icon. (b) Class Accuracy plugin menu. Figure 1.1: Illustration of the Class Accuracy QGIS Plugins icon and menu though which the plugin in executed. 1.2 Source Code The source code is maintained on a bitbucket cloud repository: org/petebunting/classaccuracy

6 Chapter 2 Running the Class Accuracy Plugin 2.1 Create Accuracy Points The first step in assess the accuracy of your classification is to generate a set of points which you are to assign the correct class to and which are then compared to your classification. RSGISLib provides a number of functions which are useful for this analysis. # Create a random set of points rsgislib.classification.generaterandomaccuracypts(inputimage, outputshp, classimgcol, classimgveccol, classrefveccol, numpts, seed, force) # Create a set of random points stratified per class. rsgislib.classification.generatestratifiedrandomaccuracypts(inputimage, outputshp, classimgcol, classimgveccol, classrefveccol, numpts, seed, force, usepxllst) # Populate an existing set of points with the class from a classification rsgislib.classification.popclassinfoaccuracypts(inputimage, inputshp, classimgcol, classimgveccol, classrefveccol) For thinking about how many points you should generate refer to the literature in this field (e.g., Congalton, 1991; Foody, 2009, 2015). If you are unsure, Foody (2009) is the text to read but if you just to get on with it then I would recommend doing a stratified random sample with 100 points per class. The rsgislib.classification.popclassinfoaccuracypts function is also very useful as if you are comparing classifiers you can easily reuse the same points by just adding extra columns to the points you have already generated and labelled. 6

7 CHAPTER 2. RUNNING THE CLASS ACCURACY PLUGIN 7 In addition, you may also find the command line tool rsgiscalcaccmatrix.py useful as if you export your shapefiles to CSV files (using ogr2ogr -f CSV output.csv input.shp) then you can produce a formatted error matrix using latex or as a CSV file you can open an your preferred spreadsheet package. If you have had a number of sites for which you have undertaken an accuracy assessment this command also allows you to provide a list of CVS files as input and it will merge them into a single error matrix. 2.2 What data to use as reference This is not the topic of this tutorial but a few words. If possible, the data you use as reference (i.e., you are use to check you classification against) should be independent of your classification approach. For example, you ve undertaken a classification using Sentinel-2 so you validate with aerial photography. However, be careful to ensure the imagery you are using for validation is the same date etc. In many cases it simple isn t possible to have an independent validation datasets so this assessment must be undertaken using the imagery which was used for the classification. Remember, the OpenLayers plugin allows you to open the latest GoogleEarth and Bing Maps layers within the QGIS and these can be used, with some care, as your reference layers. 2.3 Use Class Accuracy Plugin The Class Accuracy dialog box is shown in Figure 2.1. To start the analysis you need to set the input parameters in sections 1. Select a Vector Layer and 2. Select Columns Select Vector Layer This drop down list will list all the vector layers you have open in QGIS. Just select your accuracy assessment points Select Columns From the vector layer you have selected, you now need to select the relevant columns within the attribute table: Classified Column this is the column which stated what the point was classified as within your classification result

8 CHAPTER 2. RUNNING THE CLASS ACCURACY PLUGIN 8 Figure 2.1: The Class Accuracy Plugin dialog box.

9 CHAPTER 2. RUNNING THE CLASS ACCURACY PLUGIN 9 Output Column at the start of the process this column will be the same as the Classified Column but as you go though the points and change them to the correct class this column will be updated. At the end, this column will be your reference column Processed Column this column is a binary column where initial all points will have a value of zero and as each is visited that value is changed to one. This column allows the tool to skip points which have already been processed. However, if you wish to visit all points and ignore this column (although it still needs to be defined) tick on the Visit Processed Points option below Start Analysis Next just press the Start button. This will zoom and pan to the first point in the shapefile. If the zoom level is not correct for the data you are using as validation then this can be edited at the bottom of the dialog 7. Change scale. Press update after changing the number. This same scale will be define for the whole session. The class to which this point was classified will be shown in section 5. Change class if incorrect, on the left hand side. If the point is incorrectly classified you can change it in the drop down (note. this list is all the classes in the classification). If it is correctly classified or you selected the correct class then press the Next button. This will move on to the next point and save the previous feature to the shapefile Go to Feature If you have previous worked on this shapefile but don t want to use the Processed Column, for example to revisit and check some points, you can jump to any point in the list of points using the 4. Go direct to a feature. Just end the feature ID (note, this starts at 1) and press Go To. As you press Next this will go through the points consecutively from this feature Add Classes It is sometime the case where you want to add an additional class which was not classified in your classification. You can do this using section 6. Add extra class to the list. Just type the name of the class you want and select Add. This class will now be available in the drop down list in section 5 of the dialog.

10 CHAPTER 2. RUNNING THE CLASS ACCURACY PLUGIN Assign Points to Class It can also be useful to be able to select a group of points and assign them to a class as a single step. This is possible using the QGIS selection tool use this to select all the points you want to assign. Then check the drop down box in select 5. is set to the class you want to assign to and then press the Assign button Ending Analysis You can press Finish at any point and it will make sure everything is saved to your shapefile and then end the processing. When you reach the last point in the file this happens automatically. You can pick up from where you finished using either the Processed Column or the Go To option. You can also use the Calc Error Matrix button at the bottom to produce a CSV file with the error matrix for those points.

11 Chapter 3 Worked Example: Classifier Comparison 3.1 Create a Classification So, obviously the first thing you need is a classification... This tutorial is not about how you create your classification so please see the other tutorials and documentation of the RSGISLib website (urlhttp:// However, for this example a basic pixel based classification is undertaken and a comparison made of using a Extra Trees random forests based classifier and a straight decision tree classifier. The script is show below and the Sentinel-2 image (Figure??, script and training data are provided in the Example directory. #/usr/bin/env python import rsgislib from rsgislib import imageutils from rsgislib import vectorutils from rsgislib.classification import classimgutils from rsgislib.imagecalc import calcindices from sklearn.ensemble import ExtraTreesClassifier from sklearn.tree import DecisionTreeClassifier sen2img = 'SEN2_ _lat53lon367_T30UVD_ORB080_utm30n_vmsk_rad_srefdem_sub.tif' sen2ndvi = 'SEN2_ _lat53lon367_T30UVD_ORB080_utm30n_vmsk_rad_srefdem_sub_ndvi.kea' sen2wbi = 'SEN2_ _lat53lon367_T30UVD_ORB080_utm30n_vmsk_rad_srefdem_sub_wbi.kea' validimgmsk = 'SEN2_ _lat53lon367_T30UVD_ORB080_utm30n_vmsk_rad_srefdem_sub_validmsk.kea' forestvec = './Training/Forest.shp' hardsurfacevec = './Training/HardSurface.shp' vegetationvec = './Training/Vegetation.shp' watervec = './Training/Water.shp' 11

12 CHAPTER 3. WORKED EXAMPLE: CLASSIFIER COMPARISON 12 forestimg = 'ForestTrain.kea' hardsurfaceimg = 'HardSurfaceTrain.kea' vegetationimg = 'VegetationTrain.kea' waterimg = 'WaterTrain.kea' forestimgsamp = 'ForestTrainSamp.kea' hardsurfaceimgsamp = 'HardSurfaceTrainSamp.kea' vegetationimgsamp = 'VegetationTrainSamp.kea' waterimgsamp = 'WaterTrainSamp.kea' forestpxltrainvals = 'ForestPxlTrainVals.h5' hardsurfacepxltrainvals = 'HardSurfacePxlTrainVals.h5' vegetationpxltrainvals = 'VegetationPxlTrainVals.h5' waterpxltrainvals = 'WaterPxlTrainVals.h5' # Calculate Indices calcindices.calcndvi(sen2img, 3, 7, sen2ndvi, stats=true, gdalformat='kea') calcindices.calcwbi(sen2img, 1, 7, sen2wbi, stats=true, gdalformat='kea') # Create Valid Mask imageutils.genvalidmask(sen2img, validimgmsk, 'KEA', 0.0) rsgislib.rastergis.populatestats(validimgmsk, True, True, True) # Rasterise the training samples (shapefiles) vectorutils.rasterise2image(forestvec, sen2img, forestimg, gdalformat='kea', burnval=1) vectorutils.rasterise2image(hardsurfacevec, sen2img, hardsurfaceimg, gdalformat='kea', burnval=2) vectorutils.rasterise2image(vegetationvec, sen2img, vegetationimg, gdalformat='kea', burnval=3) vectorutils.rasterise2image(watervec, sen2img, waterimg, gdalformat='kea', burnval=4) # The number of pixels to be sampled from each training class nsamples = 6000 # Sample the training did to identify the training pixels. imageutils.performrandompxlsampleinmasklowpxlcount(inputimage=forestimg, outputimage=forestimgsamp, gdalformat='kea', maskvals=[1], numsamples=nsamples) imageutils.performrandompxlsampleinmasklowpxlcount(inputimage=hardsurfaceimg, outputimage=hardsurfaceimgsamp, gdalformat='kea', maskvals=[2], numsamples=nsamples) imageutils.performrandompxlsampleinmasklowpxlcount(inputimage=vegetationimg, outputimage=vegetationimgsamp, gdalformat='kea', maskvals=[3], numsamples=nsamples) imageutils.performrandompxlsampleinmasklowpxlcount(inputimage=waterimg, outputimage=waterimgsamp,

13 CHAPTER 3. WORKED EXAMPLE: CLASSIFIER COMPARISON 13 gdalformat='kea', maskvals=[4], numsamples=nsamples) # Define the layers to be used for classification and the bands in those layers. imgfileinfo = [imageutils.imagebandinfo(sen2img, 'sen2', [1,2,3,4,5,6,7,8,9,10]), imageutils.imagebandinfo(sen2ndvi, 'ndvi', [1]), imageutils.imagebandinfo(sen2wbi, 'wbi', [1])] # Extract the training data to HDF files. imageutils.extractzoneimagebandvalues2hdf(imgfileinfo, forestimgsamp, forestpxltrainvals, 1.0) imageutils.extractzoneimagebandvalues2hdf(imgfileinfo, hardsurfaceimgsamp, hardsurfacepxltrainvals, 2.0) imageutils.extractzoneimagebandvalues2hdf(imgfileinfo, vegetationimgsamp, vegetationpxltrainvals, 3.0) imageutils.extractzoneimagebandvalues2hdf(imgfileinfo, waterimgsamp, waterpxltrainvals, 4.0) # Define the classes to be classified with the training data (HDF5 file) and colour classtraininfo = dict() classtraininfo['forest'] = classimgutils.classinfoobj(id=1, fileh5=forestpxltrainvals, red=51, green=160, blue=44) classtraininfo['hardsurface'] = classimgutils.classinfoobj(id=2, fileh5=hardsurfacepxltrainvals, red=192, green=192, blue=192) classtraininfo['vegetation'] = classimgutils.classinfoobj(id=3, fileh5=vegetationpxltrainvals, red=31, green=248, blue=34) classtraininfo['water'] = classimgutils.classinfoobj(id=4, fileh5=waterpxltrainvals, red=0, green=0, blue=204) ########## Run Extra Trees Classifier ##################### # Create scikit-learn classifier (can be any in the library) skclassifieret = ExtraTreesClassifier(n_estimators=100) # Train that classifer classimgutils.trainclassifier(classtraininfo, skclassifieret) # Apply the classification. outimgclass = 'MidWales_BasicClass_ExtraTrees.kea' classimgutils.applyclassifer(classtraininfo, skclassifieret, validimgmsk, 1, imgfileinfo, outimgclass, 'KEA') ########################################################## ########## Run Decision Tree Classifier ##################### # Create scikit-learn classifier (can be any in the library) skclassifierdt = DecisionTreeClassifier() # Train that classifer classimgutils.trainclassifier(classtraininfo, skclassifierdt)

14 CHAPTER 3. WORKED EXAMPLE: CLASSIFIER COMPARISON 14 # Apply the classification. outimgclass = 'MidWales_BasicClass_DecisionTree.kea' classimgutils.applyclassifer(classtraininfo, skclassifierdt, validimgmsk, 1, imgfileinfo, outimgclass, 'KEA') ########################################################### At the end of this analysis you will get two classifications (Figure 3.1) (a) Result from the Extra Trees Classifier. (b) Result from the Decision Tree Classifier. Figure 3.1: The classification outputs used in this example of performing an accuracy assessment. 3.2 Generate Accuracy Points To generate accuracy points which we ll use the ClassAccuracy plugin to review we will use the following RSGISLib command to create 100 random points per class. import rsgislib.classification rsgislib.classification.generatestratifiedrandomaccuracypts('./midwales_basicclass_extratrees.kea', 'AccuracyPoints.shp', 'ClassName', 'ETClass', 'RefClass', 100, 1, True, False) rsgislib.classification.popclassinfoaccuracypts('midwales_basicclass_decisiontree.kea', 'AccuracyPoints.shp', 'ClassName', 'DTClass', 'RefClass2')

15 CHAPTER 3. WORKED EXAMPLE: CLASSIFIER COMPARISON Run the Class Accuracy Plugin You can now open QGIS, load the Sentinel-2 image and Google Earth Imagine along with your accuracy points. You will notice that both commands created a reference column. We will ignore one of them and just use the first one we created RefClass. In QGIS add an extra column Processed to record whether a point has been visited or not please ensure that all the rows in this column has a value of 0 before you start the plugin. You can then run the plugin and enter the columns in the drop down boxes (Figure 3.2). Figure 3.2: The Class Accuracy Plugin with the columns selected and ready to run. Now, visit all the points and define a reference class. Please note, to go to the next point you can just press the Enter key and to change the class from what was classified to correct it you can use the number keys in the same order the classes are listed in the drop down box (i.e., for first class in the drop down box you can press 1 ). 3.4 Create Error Matrix So, you have finished going through your points. Simply press Calc Error Matrix and it will ask you where you want to save the CSV file to. However, in this case we have two classifications we want to compare so we will use the rsgiscalcaccmatrix.py tool. First you need to convert your attribute table into a CSV file:

16 CHAPTER 3. WORKED EXAMPLE: CLASSIFIER COMPARISON 16 ogr2ogr -f CSV AccuracyPoints.csv./AccuracyPoints.shp You can then run the command specifying the columns and output file: rsgiscalcaccmatrix.py -c ETErrMatrix.csv -i./accuracypoints.csv -r ETClass -g RefClass rsgiscalcaccmatrix.py -c DTErrMatrix.csv -i./accuracypoints.csv -r DTClass -g RefClass For me, this provided an overall accuracy of 96 % for the extra trees algorithm and 92 % for the decision tree. If you have L A TEX installed you might also want to export a tex file with a formatted version of the error matrix (Tables 3.1, 3.2), use the commands below. rsgiscalcaccmatrix.py -c ETErrMatrix.csv -t ETErrMatrix.tex \ -i./accuracypoints.csv -r ETClass -g RefClass rsgiscalcaccmatrix.py -c DTErrMatrix.csv -t DTErrMatrix.tex \ -i./accuracypoints.csv -r DTClass -g RefClass Table 3.1: Error Matrix for Extra Trees Classifier. Overall Accuracy: 95.5% Kappa: 0.94 Vegetation Water HardSurface Forest User Vegetation % Water % HardSurface % Forest % Prod 95.83% 94.34% 97.89% 94.17% 95.5% Table 3.2: Error Matrix Decision Tree Classifier. Overall Accuracy: 92.0% Kappa: Vegetation HardSurface Water Forest User Vegetation % HardSurface % Water % Forest % Prod 89.58% 93.68% 96.23% 88.35% 92.0%

17 Chapter 4 Conclusions Hopefully this document has provided you the information you require to generate accuracy assessment points using RSGISLib and to use the Class Accuracy QGIS plugin to assess the accuracy of your remotely sensed classifications. If you have any further questions/issues please contact our rsgislib mailing list. 17

18 Bibliography Bunting, P., Clewley, D., Lucas, R. M., Gillingham, S., Jan The Remote Sensing and GIS Software Library (RSGISLib). Computers and Geosciences 62, Congalton, R. G., A review of assessing the accuracy of classifications of remotely sensed data. Remote Sensing Of Environment 37 (1), Foody, G. M., Oct Sample size determination for image classification accuracy assessment and comparison. International Journal Of Remote Sensing, Foody, G. M., Mar Valuing map validation: The need for rigorous land cover map accuracy assessment in economic valuations of ecosystem services. Ecological Economics 111,