BerkeleyImageSeg User s Guide

Transcription

1 BerkeleyImageSeg User s Guide 1. Introduction Welcome to BerkeleyImageSeg! This is designed to be a lightweight image segmentation application, easy to learn and easily automated for repetitive processing of large jobs, testing and embedding in image processing chains. BerkeleyImageSeg performs segmentation, one of the first steps in Object Based Image Analysis. Later steps may include some other analysis of the resulting segments, classification, or generation of additional shape and spectral statistics. BerkeleyImageSeg contains basic statistics generation and classification, but the primary function is generating large numbers of segmentation results, which can be customized based on a set of parameters and switches accessed through either a command line or graphical user interface. This manual will guide you through accessing the features of BerkeleyImageSeg through the command line and the GUI. Lastly, we will provide examples of using BerkeleyImageSeg in a processing chain using Weka an open source data mining software, and from external applications. 2. Algorithm BerkeleyImageSeg uses region merging for the segmentation algorithm (Benz et al. 2004). Initially, every pixel is an object and merging proceeds iteratively. For any object, consider its contiguous neighbors. Let any pair of contiguous objects be described as object a, object b, and their possible union ab as the merged object. Let the difference in spectral heterogeneity h p for the merged object by defined as: hp wi ( nab i ab ( na i, a nb i, I, b 0 w 1, 1 are the weights for i=0,1 I image bands, n denotes the area of an i w i I object in pixels, and σ i is the standard deviation in band i for an object. )) Let the difference of compactness and smoothness, objects be defined as: hc and hs respectively, of the Copyright (c) 2009 Nicholas Clinton Revised June 5,

2 n l n l n l hc n n n ab ab a a b b ab a b n l n l n l hs b b b ab ab a a b b ab a b l is object perimeter length and b is the perimeter of the object s bounding box length. Let the difference in shape heterogeneity be defined as: h t w h c c w h s s 0 w c, ws 1, wc ws 1 and w c is the user selected compactness parameter (w s is the smoothness parameter). Define the scale rate r as: r w h p p w h t t 0 w, w 1, w w 1 and w t is the user selected shape parameter. p t p t The program will iterate through the objects, merging contiguous objects if the estimated scale rate r is below t=0,1,2 T. The user selected scale threshold parameter T determines the number of iterations through the objects or the number of merging cycles. So the higher the scale, the more merging will occur and the larger the objects in the segmentation will be. 3. Performing Segmentation The Wizard Now that we have an idea of what the program is doing, let s see how to run it. BerkeleyImageSeg will read images in a variety of formats (contact support for currently supported formats), perform a segmentation and output the results as images, an ESRI shapefile, and/or comma delimited text files (for tabular output). This is the BerkeleyImageSeg Wizard: Copyright (c) 2009 Nicholas Clinton Revised June 5,

3 First, select an input image to process by pressing the Browse button and navigating to the file. The default is a sample image (of agricultural fields) in the installation/sample/ directory (there is also a sample training shapefile: ag-test_train.shp, in the sample/ directory). Next, choose parameter values for threshold (1,2 Iterations), shape [0,1] and compactness [0,1] and enter them to the wizard using the text boxes. We will use threshold=50, shape=0.5, and compactness=0.5, the default values, in this example. As described in Section 2. Algorithm, this represents an equal weighting of compactness and smoothness, shape and spectral heterogeneity in the segmentation. The threshold specifies that 50 iterations of merging will occur. Click the Advanced and Preview buttons to enable additional features and show a thumbnail preview of the image to be processed. This is the Wizard in Advanced view with the Preview enabled. Copyright (c) 2009 Nicholas Clinton Revised June 5,

4 Note that there are now checkboxes to select some output options. The Segments image is a one band image, in the same format as the input and in the same directory as the input, with segment id (as a 32-bit integer) as the pixel value. This image of segment membership is the default output and will remain checked. The other options are for an ESRI Shapefile, placed in the input directory with filename input_image_name_scaleshape-compactness.shp (ag-test_ shp for example). The other output option is the Statistics table, which is output as a comma delimited text file (.csv) with a row for each segment identified by the id field. There are also fields for spectral statistics in each band (maximum, minimum, mean and standard deviation) and some shape statistics (pixel count, area, bounding box perimeter, compactness, fractal dimension, perimeter:area ratio, perimeter, shape index, and smoothness). This table will also be placed in the input directory with input_image_name_scale-shape-compactness.csv filename. Copyright (c) 2009 Nicholas Clinton Revised June 5,

5 Several classification options (described at are available within BerkeleyImageSeg: the unsupervised k-means clustering algorithm, the supervised k-nearest neighbor (k-nn) where k is a user parameter for number of clusters or neighbors to use, and a fuzzy neural network. It is also possible to enter multiple parameters as comma separated lists and the wizard will automatically process all the combinations. When the Enable Automation box is checked, the values, separated by commas, can be entered into the text boxes for the scale, shape and compactness parameters. If the Train: and Rank segmentations boxes are checked, a shapefile of polygons can be supplied as training data in order to select the best parameter combination. A polygon comparison metric (described in the white paper is used to rank the segmentation results according to which has segments that best match the training shapes. The best matching segments are also used to train the classifier. When the Run button is pressed, the program will process the selected image, placing the output files into the same directory as the input image. The Command Line window will display the progress of the program and any important messages from BerkeleyImageSeg. If multiple parameter combinations are specified, multiple output sets will be generated. The jobs are processed sequentially, within nested scale parameters. Using the above example, the sample/ directory should now look like this: Notice that a shapefile has been generated, as well as some other outputs. Copyright (c) 2009 Nicholas Clinton Revised June 5,

6 The Command Line The Command Line interface to BerkeleyImageSeg exposes some additional features of the program and disables the classification and preview options. This is the window that appears when the Command Line BerkeleyImageSeg is invoked: Each option can be turned on or off with + or, respectively. The options specific to the Command Line are: [g] Grid image of segments for import to GIS/RS. [r] Rank goodness of each segmentation run compared to the training shapes, write to '_rank.csv' file. Requires training to be enabled. [m] Take thresholds each as a Minimum Mapping Unit in georeferenced area. [p] Parallel process jobs using system max [or N] number of processors/cores. The output to the console will be overlapping. [x] Split large image into tiles, start segmentation, then merge and finish. Copyright (c) 2009 Nicholas Clinton Revised June 5,

7 4.1 Sample Process Clustering Process the ag-test.bmp sample image according to the example in the previous sections. The ag-test_ csv file should now be in the /sample directory (generated as an output specified with the Statistics option). This is a comma separated file with the spectral and shape statistics for each segment. Our first goal is to use a powerful data mining software to identify some patterns in the feature space of these spectral and shape variables. The following example uses Weka ( an open source data mining package to do some clustering, or unsupervised classification of these data. The ag-test_ csv file can be opened directly by Weka. This is what ag-test_ csv in the clustering tab of the Weka Explorer: This setup shows a run using the Expectation Maximization algorithm to do the clustering while filtering out the id and filename attributes. The results can be visualized in Weka or saved for output to some other application. To export the attributes is a little convoluted, but will be obvious in a second. First, right click the run identifier in the Result list (bottom left of Weka window). Select Visualize Cluster Assignments. Copyright (c) 2009 Nicholas Clinton Revised June 5,

8 Another window will appear for visualizing the results. Click the save button and save to a.arff file (text, using Weka format) that will include the cluster assignments. We want to convert the.arff to a.csv, so open the.arff in Weka Explorer, then from the Preprocess tab, click save and simply save it to a.csv file. This process is simply for converting the file type (which could also be done manually, using a text editor). You should now have a.csv file that includes the original statistics, an id field, and the cluster assignment. Let s take a look at these outputs in a GIS. Here is the original input image with the shapefile of segments overlaid. In the attributes of the segment shapefile, there is an id field. Open the table of statistics and cluster assignments created in Weka and join it to the attributes of the segments shapefile using the id field. Now, the cluster assignment (based on the spectral and shape attributes of the segments) can be used to render the polygons in the segmentation shapefile. Just to summarize what we have done, in our example, we opened ag-test_ csv in Weka, filtered the first two attributes, performed a clustering (Expectation Maximization with 4 clusters), saved the clusters as ag-test_ _em_clusters.arff, opened this file in Weka, then saved it again as ag-test_ _em_clusters.csv. Next, we opened ag-test_ _em_clusters.cvs in our GIS and joined it to the attributes of the segments using the id field. Finally, we used the Cluster field to assign color to each segment. The segments, rendered according to cluster membership, are displayed in the following image. Copyright (c) 2009 Nicholas Clinton Revised June 5,

9 This example is intended to illustrate how to incorporate BerkeleyImageSeg results into other software for more powerful classification and visualization. While we used clustering for demonstration purposes, polygons could be chosen from the results for training, either through visual interpretation, or automatically using training polygons. Once training segments have been identified in an attribute of the statistics file, any of the supervised algorithms in Weka will also be available. 4.2 Sample Process Parameter Selection Consider some image of interest for which a set of training objects is available. The following image shows our ag-test.bmp image with some hand digitized polygons (agricultural fields) overlaid (provided in ag-sample/test_train.shp). Copyright (c) 2009 Nicholas Clinton Revised June 5,

10 Suppose we wish to use these polygons to select a better parameter combination than the default 50, 0.5, 0.5 we used in the previous example. To do so, we can rank the segmentations using either the Wizard or the Command Line interface. In the wizard, we simply check the Train: and Rank segmentations boxes and select the training shapefile. The setup looks like this: Copyright (c) 2009 Nicholas Clinton Revised June 5,

11 When the Run button is pressed, BerkeleyImageSeg will run all the parameter combinations and rank each one, creating an output table called ag-test_rank.csv which is also placed in the input directory along with all the segmentation results corresponding to the various parameter combinations. Open this file in Excel or some other viewer and observe that the various parameter combinations that were evaluated have been ranked according to how well they match the training polygons. The following image shows how the results are ranked according to the d metric, with lower values of d indicating a better match to the training polygons. The ranking table (which has been sorted according to the dmetric column) shows that the 60, 0.7, 0.3 parameter combination results in the segmentation with the lowest d, meaning that, of the parameter combinations tested, 60, 0.7, 0.3 is optimal with regard to the provided training objects. We can open the corresponding shapefile and compare it to the 50, 0.5, 0.5 result shown above. The following image shows the 60, 0.7, 0.3 segmentation overlaid on the input image. Copyright (c) 2009 Nicholas Clinton Revised June 5,

12 This result is considerably different from the default 50, 0.5, 0.5 result, with approximately half as many segments. We can now evaluate a classifier relative to this improved segmentation, using the Weka based method described above. Observe that some extra fields are appended to the ag-test_ csv statistics file as a result of the training and ranking. The fields identify segments that intersect the training objects and quantify how well they match a particular class. These fields could be utilized for training a supervised classifier, or simply ignored for a clusterer. 4.3 Sample Process Supervised Classification Supervised classification of image objects involves provision of a set of training objects to the classification software. The training data set is then used to build a suitable classifier from which other objects can be classified. In the present context, the objects consist of the segments, their shape and spectral attributes, and a class label. Similar to the clustering example in Section 4.1, we can use Weka to perform a supervised classification. However, we must first specify the class labels of some subset of segments from the segmentation of the image of interest. This can be performed automatically, using a training file, or interactively, by selecting segments that are known to have a particular label. Consider the following true color image of the Sierpe region of Costa Rica. The imagery is bands 5, 3, 1 of the MODIS/ASTER airborne simulator (MASTER, The segmentation result is from a 30, 0.3, 0.5 run of 13 MASTER bands from the visible, NIR and SWIR. Copyright (c) 2009 Nicholas Clinton Revised June 5,

13 Suppose we know that these segments represent water, mangrove forest, agriculture, etc. We will use visual interpretation in a GIS to specify a subset of segments from each class, labeling them in an attribute we add called class. First, add the attribute to the resultant shapefile, select segments to be used as training for each class and enter the label into the class attribute. Save the resultant table. For example, the following image illustrates our selection for water training polygons. Copyright (c) 2009 Nicholas Clinton Revised June 5,

14 Once some subset of polygons has been chosen for training, join the *.csv table output by BerkeleyImageSeg to the attributes of the shapefile, using the id field. Some of the segments in this shapefile will have a labeled class, some will not. Export the attributes, with the new class field, joined to the BIS *.csv table, to a comma delimited text files. This will be the Weka training. Once these tables have been output to comma-delimited text, they can be read by Weka. (WARNING: make sure you look at the text file in a text editor to make sure there is nothing extra in there, like quotation marks.) However, there are some extraneous columns in there that we don t need. All we really want are all the shape and spectral attributes, the class label and the id field. The extraneous fields include id fields, and the segmentation file label. We can handle these in Weka. Open your exported *.csv in Weka. To deal with the extra attributes, from the classification pane, select Meta FiltererClassifier. Select the classifier of choice and parameterize it accordingly. Then, for the filter, go to Unsupervised Attribute Remove and choose the indices to remove (which are conveniently displayed on the Preprocess frame). Make sure the class attribute is set, choose a classifier and click Start. When the program has run, right click the run identifier in the Result list (bottom left of Weka window). Select Visualize Cluster Assignments. Another window will appear for visualizing the results. Click the save button and save to a *.arff file (text, in Weka format) that will include the class assignments. We want to convert the.arff to a.csv, so open the.arff in Weka Explorer, then from the Preprocess tab, click save and simply save it to a.csv file. This process is simply for converting the file type (which could also be done manually, using a text editor). You should now have a.csv file that includes the Copyright (c) 2009 Nicholas Clinton Revised June 5,

15 original statistics, an id field, and the cluster assignment. The results can now be viewed by joining this file back to the attributes and using the predictedclass field. The result for our Sierpe image is shown below. 4. Conclusion BerkeleyImageSeg is a powerful segmentation software that can be learned in a few minutes and be producing powerful information almost immediately. As a result of its ease of use and simplicity, the results are easy to incorporate to other applications for further processing. The features of BerkeleyImageSeg are intended to perform the first step of Object Based Image Analysis: segmentation. Using the automation capability and a training file, meaningful results can be identified automatically from large sets of parameter combinations. Once an optimal result has been found, BerkeleyImageSeg will generate a complete set of spectral and shape statistics that can be used for classification in your software of choice. The outputs are designed with the same ease of use philosophy as the input, and are highly interoperable with other software such as GIS, Copyright (c) 2009 Nicholas Clinton Revised June 5,

16 data mining, or spreadsheets. We hope you fill find the joy of image segmentation using BerkeleyImageSeg! 4. References Benz, Ursula C., Peter Hofmann, Gregor Willhauck, Iris Lingenfelder, Markus Heynen Multi-resolution, object-oriented fuzzy analysis of remote sensing data for GISready information. ISPRS Journal of Photogrammetry & Remote Sensing. 58: Clinton, Nicholas, Ashley Holt, James Scarborough, Li Yan, Peng Gong. forthcoming. Accuracy Assessment Measures for Object-based Image Segmentation Goodness. Photogrammetric Engineering & Remote Sensing Copyright (c) 2009 Nicholas Clinton Revised June 5,