Automatic Road Extraction and Reconstruction using High Resolution Satellite Images

Transcription

1 Automatic Road Extraction and Reconstruction using High Resolution Satellite Images Jiayuan Wang and Guowei Yan December 1, Project Idea Road network detection from high resolution satellite images has been the purpose of many works in the image processing field. One of its application is for automated road map making. Although an expert can label road pixels in a given satellite image, this operation is very time consuming. Due to its importance and complexity, much effort is devoted to finding solutions to this problem. The goal of this project is to develop an automatic method to detect the road network in a given satellite image in a robust manner and also reconstruct the road net work as a graph. Our data is all collected from Goole Earth and Google Map. 2 Method Our processing contains two steps: one is road extraction, and another is road reconstruction. In the road extraction, we denoise the satellite image treating the roads as a signal. The outcome of the road extraction serves as the income of road reconstruction. In the road reconstruction, we build a geometrical graph based on the skeleton of the road. The final output allows us to solve graph-related problems such as shortest distance and max flow. 2.1 Road Extraction Smoothing and Edge Detection One of the physical characteristics of a road is that they are smooth and built using materials such as concrete or asphalt. Because of this, there exists a sharp change from a road to other background objects. Thus an natural thought is to run edge detection algorithms to find such boundaries. We know the primary problem in edge detection is dealing with image noise. So at the first, we tried Gaussian smoothing and use sobel edge detection for testing. 1

2 (a) An airport (b) A residential area Figure 1: Examples of satellite image. Taken from Google Earth. (a) An airport (b) A residential area Figure 2: Output of running edge detection on the satellite images in Fig 1 We observed that, for a simple case such as Fig 1a, the combination of smoothing and edge detection can indeed find the boundaries of the road networks (see the edge detection result in Fig 2a). However, it also outputs other high contrast image regions such as the building edges, which are not desired. Moreover, when the scenario becomes more complicated (see the residential area in Fig 1b), it fails to detect all the edge points and also generated lot of noises (see the edge detection result in Fig 2b). Intensity Thresholding One interesting question is that, how do our vision system work to find out all the road network quickly as well as accurately when we see the satellite images? One possibility is that we differentiate different objects in the images by their colors. We tried to analyze the color feature of road and non-road part from RGB color space and also from images converted to grayscale from RGB space. Our experiment shows that thresholding of grayscale images does do a good job on denoising when roads are regarded as signals. After denoising, we only need to perform dilation and erosion, and keep several largest connected component to get a satisfactory outcome for road reconstruction. Fig 3 shows the whole process of road extraction using residential area satellite image. 2

3 (a) (b) (c) (d) Figure 3: The whole process of road extraction. (a) is The original satellite image. (b) is threshold with around 90% highest pixel values after converting (a) to a greyscale image. (c) is the dilations and erosions on (b). (d) is keeping several largest connected components in (c). Moreover, We want to know why simply thresholding on grayscale images really works. We take the satellite image of the residential area (Fig 1b) to compare the color distributions of the pixels on roads with those not on roads to see which color feature makes them so different. We use YIQ which decouples intensity and color of the image since the RGB color space does not tell us intensity informations. In the YIQ space, The Y component provides all monochrome/intensity information,i and Q components carry the color information. We fit the three component of the road pixels into normal distribution respectively and then compare them with the three component of the non-road pixels. (a) I component distribution (b) Q component distribution (c) Y component distribution Figure 4: Fitted normal distributions of I/Q/Y components of the roads and non-roads. The blue curves are roads, the red curves are non-roads. Fig 4 shows the fitted normal distributions of I/Q/Y components of the roads and nonroads. The color components(i and Q) distributions are similiar for road pixels and non-road pixels, which overlaps a lot. This indicates that the intensity component is the main difference between the two regions, thus can be possibly used to for our purpose. 3

4 2.2 Road Reconstruction To reconstruct the road, we only want to keep the skeleton of the outcome from the last section. We take out the skeleton by removing pixels so that an object without holes shrinks to a minimally connected stroke, and an object with holes shrinks to a connected ring halfway between each hole and the outer boundary. The skeleton extracted from the previous section is only a set of 2D points. In this section, we want to reconstruct the graph information from the skeleton, which allows us to solve graph related problems (for example, shortest path problem, maximum flow problem) thus take good use of the satelitte image. We come up with a reconstruction algorithm by ourselves. It is based the intuition to build a graph from the skeleton. Our graph reconstruction algorithm has two steps: first we labeling all 2D points as vertex points or edge points. Then we only keep the vertex point and connect them using connection information in the original skeleton. Reconstruction algorithm Step1 In step1 we labeling all 2D points as vertex points or edge points. We compute the degree of the point as the number of 1s in the 8-connected neighborhood of the point. We call a point is a vertex point if its degree is not 2 or when its degree is 2, the angle between two edges that incident to the point is less than 90. For all points that are not vertex points, we call them edge points. Fig 5 shows examples of vertex points. Figure 5: Examples of vertex point with degree 1, 3, 4, 2, 2 from left to right. The red pixel is the the point we take into consideration, the black pixels are its 8-connected neighthood. Figure 6: Examples of edge point. Step2 In step2, we reconstruct a graph G = (V, E) where V is all the vertex points from step1 and E is obtained from the connection of the original skeleton. For every vertex point v, we pick one of its adjacent neighbor v, check if v is a vertex point or not. If v is a vertex point, then add edge (v, v ) to E, otherwise go to the adjacent neighbor of v (since a edge point must have degree, we only have one choice to go), repeat until we reach a vertex point v, then add edge (v, v ) to E. Then we repeat for all v s adjacent points. Fig 7 gives an example of how the algorithm works. 4

5 Figure 7: Illustration of the algorithm. The red pixel is the vertex point we take into consideration, the black pixels are the edge points, the blue pixels are the vertex points we find from the red pixel. Note that in step 1, we can define the vertex point flexibly that we can also label a point with degree 2 and its two adjacent edges have the angle less than 135 instead of 90. This results in larger size of the vertex set of the reconstruction graph G, thus the output graph approximates the skeleton better. Table 1 shows the distribution of angles of adjacent edges for points with degree 2 of the image for the residential area. If we change our standard of choosing a vertex point from 90 to 135, we will add 5757 point as vertex and even more for edges, which will certainly become a problem when the original image is large. Degree # Table 1: The distribution of angles of adjacent edges for points with degree 2 of the skeleton for the residential area. Fig 8 and 9 shows the final results of road reconstruction. The second column and the third column have different choice of vertex points and edge points. The second column fails to capture some turns in the skeleton. The third column almost approximate the skeleton perfectly but it also has a much larger data size. There is a trade-off between the accuracy and the efficiency. Fig 10 is the images of the maps, which could be used to evaluate our outputs. 5

6 (a) (b) (c) Figure 8: Road reconstruction of the airport satellite image. (a) is the original image. (b) is the reconstruction when choosing a point with degree two as a vertex when the adjacent angle is less than 90. (c) is when the angle is less than 135. (a) (b) (c) Figure 9: Road reconstruction of the residential area satellite image. (a) is the original image. (b) is the reconstruction when choosing a point with degree two as a vertex when the adjacent angle is less than 90. (c) is when the angle is less than 135. Figure 10: Maps of the airport and the residential area. 3 Problems encountered At the beginning we plan to use edge detection techniques to extract the road information. We encounter two problems: first, for one road, the edge detection returns two parallel sides of that road, which is not the input we desire for the road reconstruction. We ve tried dilations and erosions to close the space between the two borders of one road but didn t get satisfactory result. The second problem with edge detection is that the output is polluted by the noise from buildings outside the roads. So we then turn to the intensity thresholding, which works pretty well. 6

7 4 Lesson Learned Although at first look it seems that the road network in the satellite image differs from the background in color information, in our experiments, the intensity information and the connectively information turn out to be the main factor. Only by comparison we know which one would be better to use. Another thing is that graph reconstruction is actually a quite difficult problem. But we have a very concrete settings, compared to the general graph reconstruction problems, which enables us to solve it using simple algorithms. We come up with the simple and intuitive algorithm to reconstruct the graph from the skeleton after learning how people would usually do in this area. We are enlightened by the papers in the reference. 5 Allocation of work Jiayuan provides the ideas of how to complete the projects and is responsible for collecting data, the road extraction by edge detection and the road reconstruction part. The report is integrated and arranged by Jiayuan. Guowei is responsible for the road extraction part by intensity thresholding. 6 Furture work Blocked regions The images we use for our algorithm requires that road regions should be clear and visible. However, when they are partially blocked, like the case shown in Fig 11, our algorithm fails to rebuild all the road network. In the future, we would like to solve this problem and make our algorithm more robust. Figure 11: An example of blocked regions. The satellite image is the OSU oval. Better road reconstruction algorithm Currently the standard of taking a point as vertex point is quite simple and only capture the information in the 8-connected neighborhood of one pixel. One obvious problem is that we can t capture curves in the images. Also, the vertex number is very sensitive with regard to how we choose the vertex points (see Table 1). To fix it, we may try take either a pixel s larger neighborhood such as 5 5 one or consider multiple pixels at the same time. References [1] Aanjaneya, M., Chazal, F., Chen, D., Glisse, M., Guibas, L., & Morozov, D. (2012). Metric graph reconstruction from noisy data. International Journal of Computational Geometry & Applications, 22(04), [2] Ge, X., Safa, I. I., Belkin, M., & Wang, Y. (2011). Data skeletonization via Reeb graphs. In Advances in Neural Information Processing Systems (pp ). 7

8 Matlab Code Part 1 contains 3 m files: extract1.m, postprocess.m, graphimformation.m. extract1.m does the road extraction, postprocess.m process the image into the skeleton after denoising, graphimformation.m reconstructs the road. Part 2 contains the functions used in the m files. We omit some minor functions here. Part 1 extract1.m 1 clc ; 2 clear all ; 3 4 % %%%%%%%%%%%%%% preprocess %%%%%%%%%%%%%%% 5 filename = map. png ; 6 rgbim = im2double ( imread ( filename )); 7 grayim = double ( rgb2gray ( rgbim )); 8 [rows, cols ] = size ( grayim ); 9 10 figure ; 11 subplot (3,3,1); 12 imshow ( rgbim ); 13 subplot (3,3,2); 14 imshow ( grayim ); % first erode the input image 17 se = strel ( disk,3); 18 grayim = imerode ( grayim,se); % observation : roads are connected together, thus it is likely that they 21 % woulbe be the first few largest connected componets 22 thresholded_grayim = grayim > threshold_percent ( grayim,0.9) ; 23 dia late d_thr esho lded_gray Im = dilationforn ( thresholded_grayim,2) ; 24 largestcc_mask = createmask ( dialated_thresholded_grayim, 0.03) ; masked_grayim = grayim.* largestcc_mask ; 27 masked_rgbim (:,:,1) = rgbim (:,:,1).* largestcc_mask ; 28 masked_rgbim (:,:,2) = rgbim (:,:,2).* largestcc_mask ; 29 masked_rgbim (:,:,3) = rgbim (:,:,3).* largestcc_mask ; subplot (3,3,3); 32 imshow ( d ialat ed_t hres holde d_gr ayim ); 33 subplot (3,3,4); 34 imshow ( masked_rgbim ); 35 subplot (3,3,5); 36 imshow ( masked_grayim ); 37 subplot (3,3,6); 38 imshow ( largestcc_mask ); 8

9 39 40 % deal with possible road discontinuity and remove small components 41 se = strel ( disk,3); 42 largestcc_mask = imdilate ( largestcc_mask, se); largestcc_mask = createmask ( largestcc_mask, 0.46) ; masked_grayim = grayim.* largestcc_mask ; 47 masked_rgbim (:,:,1) = rgbim (:,:,1).* largestcc_mask ; 48 masked_rgbim (:,:,2) = rgbim (:,:,2).* largestcc_mask ; 49 masked_rgbim (:,:,3) = rgbim (:,:,3).* largestcc_mask ; subplot (3,3,7); 52 imshow ( masked_rgbim ); 53 subplot (3,3,8); 54 imshow ( masked_grayim ); 55 subplot (3,3,9); 56 imshow ( largestcc_mask ); figure ; 59 BW1 = edge ( largestcc_mask, Sobel ); 60 imshow ( BW1 ); figure ; 66 % skeim = bwmorph ( largestcc_mask, skel,30000) ; 67 skelimg = bwmorph ( largestcc_mask, thin, inf ); 68 imshow ( skelimg ); % this part tried to use color information for separation, but did not seem to work. 71 % this can be indicated by the histogram information % gray_average = sum ( sum ( masked_grayim ))/ largestcc_num ; r_average = 74 % sum ( sum ( masked_rgbim (:,:,1)))/ largestcc_num ; g_average = 75 % sum ( sum ( masked_rgbim (:,:,2)))/ largestcc_num ; b_average = 76 % sum ( sum ( masked_rgbim (:,:,3)))/ largestcc_num ; 77 % 78 % new_rgbim = rgbim ; new_grayim = grayim ; dist_t = 0.06; for r =1:1: rows 79 % for c =1:1: cols 80 % dist_r = abs ( rgbim (r,c,1) - r_average ); dist_g = 81 % abs ( rgbim (r,c,2) - g_average ); dist_b = 82 % abs ( rgbim (r,c,3) - b_average ); dist_gray = 83 % abs ( grayim (r,c)-gray_average ); scale = 0; if( dist_r > dist_t 9

10 84 % dist_g > dist_t dist_b > dist_t ); 85 % new_rgbim (r,c,1) = new_rgbim (r,c,1) * scale ; new_rgbim (r,c,2) 86 % = new_rgbim (r,c,2) * scale ; new_rgbim (r,c,3) = 87 % new_rgbim (r,c,3) * scale ; 88 % end if( dist_gray > dist_t ) 89 % new_grayim (r,c) = new_grayim (r,c) * scale ; 90 % end 91 % end 92 % end 93 % 94 % subplot (3,3,7); imagesc ( new_rgbim ); subplot (3,3,8); imagesc ( new_grayim ); 95 % colormap ( gray ); figure ; histogram ( masked_rgbim ); 96 % histogram ( masked_grayim ); return ; % %%%%%%%%%%%%%% edge detection %%%%%%%%%%%%%%% filename = map. png ; 104 rgbim = im2double ( imread ( filename )); 105 grayim = double ( rgb2gray ( rgbim )); % setting parameters 108 sigma = 5; 109 percentage = 0.9; 110 removecc_per = 0.9; % Extract edge 113 [ Gx, Gy ] = gaussderiv2d ( sigma ); 114 Im = double ( rgbim (:,:,3)); 115 %Im= double ( rgb2gray ( rgbim )); 116 gxim = imfilter ( Im, Gx, replicate ); 117 gyim = imfilter ( Im, Gy, replicate ); 118 magim = sqrt ( gxim.^2 + gyim.^2) ; 119 imagesc ( magim ); 120 axis ( image ); 121 colormap ( gray ); 122 captionstr =[ sigma = num2str ( sigma )]; 123 title ( captionstr, FontSize, 15) ; 124 pause ; 125 % Denoise1 126 tim = magim > threshold_percent ( magim, percentage ); 127 tmagim = magim.* tim ; 128 imagesc ( tim ); 129 captionstr =[ threshold = num2str ( percentage )]; 130 title ( captionstr, FontSize, 15) ; 131 axis ( image ); 10

11 132 colormap ( gray ); 133 pause ; 134 % Denoise2 Dilation 135 d_bsim = dilationforn ( tim,3) ; subplot (1,3,1); 138 imshow ( d_bsim ); 139 title ( Dilation ); 140 outim = removesmallcc ( d_bsim, removecc_per ); subplot (1,3,2); 143 imshow ( outim ); 144 title ( Remove small connected component after dilation ); 145 diaim = dilationforn ( outim,7) ; subplot (1,3,3); 148 imshow ( diaim ); 149 title ( Dilation again ); postprocess.m 1 2 subplot (1,3,1); 3 imshow ( largestcc_mask ); 4 % dialation 5 diaim = dilationforn ( largestcc_mask,5) ; 6 subplot (1,3,2); 7 imshow ( diaim ); 8 % blur 9 sigma = 8; 10 G = fspecial ( gaussian, 2* ceil (3* sigma )+1, sigma ); 11 gim = imfilter ( largestcc_mask, G, replicate ); 12 % double images need range of subplot (1,3,3); 14 imshow ( gim ); figure ; 17 skelimg = bwmorph (diaim, thin, inf ); 18 imshow ( skelimg ); graphimformation.m 1 % parameter 2 % sample_p = 0.7; 3 range = sqrt (2) ; 4 max_degree = 10; 5 edge_angle = 180; 6 7 % convert to xy index 8 ind = find ( skelimg ); 9 [row, col ] = ind2sub ( size ( skelimg ),ind ); 11

12 10 n = size (row,1) ; 11 coor = zeros (n,2) ; 12 coor (:,1) = col ; 13 coor (:,2) = row ; % build connectivity 16 connext_idx = rangesearch ( coor, coor, range ); 17 deg = zeros (n,1) ; 18 adj = zeros (n, max_degree ); 19 for i = 1: n 20 adj_array = cell2mat ( connext_idx ( i)); 21 degree = numel ( adj_array ) -1; 22 deg (i) = degree ; 23 adj (i,1: degree ) = adj_array (2: degree +1) ; 24 end % label vertex point ( index should be enough ) 27 degree_not2 = find ( deg ~=2) ; 28 degree_2 = find ( deg ==2) ; 29 degree_2_vertex = []; 30 costheta = zeros ( numel ( degree_2 ),1); 31 for i = 1: numel ( degree_2 ) 32 ind = degree_2 (i); 33 ind_incident1 = adj ( ind,1) ; 34 ind_incident2 = adj ( ind,2) ; 35 x = coor (ind,1) ; 36 y = coor (ind,2) ; 37 x1 = coor ( ind_incident1,1) ; 38 y1 = coor ( ind_incident1,2) ; 39 x2 = coor ( ind_incident2,1) ; 40 y2 = coor ( ind_incident2,2) ; vec1 = [x1 -x,y1 -y]; 43 vec2 = [x2 -x,y2 -y]; 44 costheta (i) = dot (vec1, vec2 )/( norm ( vec1 )* norm ( vec2 )); 45 if costheta ( i) == 0 46 degree_2_vertex = [ degree_2_vertex ; ind ]; 47 % elseif abs ( costheta ( i) ) < % degree_2_vertex = [ degree_2_vertex ; ind ]; 49 end 50 end 51 vertex_pts_ind = [ degree_not2 ; degree_2_vertex ]; % reconstruct the edges 54 vertex_binary = zeros (n,1) ; 55 vertex_binary ( vertex_pts_ind ) = 1; 56 n_vertex = numel ( vertex_pts_ind ); 57 edge = zeros ( n_vertex, max_degree ); 58 for i = 1: n_vertex 59 ind = vertex_pts_ind ( i); 12

13 60 for j = 1: deg ( ind ) 61 start_pt = ind ; 62 next_pt = adj (ind,j); 63 while ( vertex_binary ( next_pt ) ==0) 64 % loop till find a vertex point 65 pre_pt = start_pt ; 66 start_pt = next_pt ; 67 next_adj1 = adj ( next_pt,1) ; 68 next_adj2 = adj ( next_pt,2) ; 69 if next_adj1 == pre_pt 70 next_pt = next_adj2 ; 71 else 72 next_pt = next_adj1 ; 73 end end 76 % next_pt == vertex 77 edge (i,j) = next_pt ; 78 end 79 end % plot the graph for i = 1: n_vertex 84 ind = vertex_pts_ind ( i); 85 for j = 1: deg ( ind ) 86 ind_edge = edge (i,j); 87 pt1 = coor (ind,:) ; 88 pt2 = coor ( ind_edge,:) ; 89 plot ([ pt1 (1),pt2 (1) ],[ pt1 (2),pt2 (2) ], r ); 90 hold on; 91 end 92 end 93 hold on; 94 scatter ( coor (:,1),coor (:,2),5, filled, MarkerFaceColor, b, MarkerFaceAlpha,0.1) ; 95 axis ij; % plot the skeleton % % subsample 102 % nsample = floor ( sample_p *n); 103 % sample_index = randi ([1 n],1, nsample ); 104 % 105 % coor = zeros ( nsample,2) ; 106 % coor (:,1) = col ( sample_index ); 107 % coor (:,2) = row ( sample_index ); 108 % % plot ( coor (:,1),coor (:,2),. ); 13

14 109 % 110 % % connectivity 111 % edge_idx = rangesearch ( coor, coor, range ); Part 2 1 function [ largestcc_mask ] = createmask ( Imin, percentage ) 2 % mask is the first largest n componets 3 4 [rows, cols ] = size ( Imin ); 5 [L, num ] = bwlabel (Imin, 8); 6 alllabel = unique (L); 7 labelcount = zeros ( num,1) ; 8 for i = 1: numel ( alllabel ) 9 labelcount (i)= sum ( sum (L== alllabel (i))); 10 end 11 [ afsort, sortin ]= sort ( labelcount ); 12 n = int32 ( percentage * numel ( alllabel )); 13 largestcc_rc = zeros (0,2) ; 14 for i = 1:1: n 15 [r_i, c_i ] = find (L == alllabel ( sortin ( numel ( sortin ) - i))); 16 largestcc_rc_i = [ r_i, c_i ]; 17 largestcc_rc = vertcat ( largestcc_rc, largestcc_rc_i ); 18 end 19 largestcc_num = numel ( largestcc_rc (:,1)); 20 largestcc_mask = zeros ( rows, cols ); 21 for i = 1:1: largestcc_num 22 largestcc_mask ( largestcc_rc (i,1), largestcc_rc (i,2) ) = 1.0; 23 end end 1 function [ outim ] = dilationforn ( Im, n ) 2 % dilate the image for n times 3 outim = Im; 4 for i =1: n 5 outim = bwmorph ( outim, dilate ); 6 end 7 8 end 1 function [ outim ] = erodeforn ( Im, se, n ) 2 % erode the image for n times 3 outim = Im; 4 for i =1: n 5 outim = imerode (outim,se); 6 end 7 8 end 14

15 1 function [ Gx, Gy ] = gaussderiv2d ( sigma ) 2 % compute and display the 2 D Gaussian derivative masks Gx and Gy 3 % for a given sigma 4 5 h= ceil (3* sigma ); 6 Gx = -1* ones (2* h +1) ; 7 Gy = -1* ones (2* h +1) ; 8 ii =1; 9 jj =1; 10 % row is for x axis, column is for y axis 11 for i=h: -1: -h 12 jj =1; 13 for j=-h:h 14 Gx(ii,jj)=- gaussianderiv2dx (j,i, sigma ); 15 Gy(ii,jj)= gaussianderiv2dy (j,i, sigma ); 16 jj=jj +1; 17 end 18 ii=ii +1; 19 end end function [ value ] = gaussianderiv2dx (x,y, sigma ) 24 coe =-x /(2* pi* sigma ^4) ; 25 value = coe * exp (-(x ^2+ y ^2) /(2* sigma ^2) ); 26 end function [ value ] = gaussianderiv2dy (x,y, sigma ) 29 coe =-y /(2* pi* sigma ^4) ; 30 value = coe * exp (-(x ^2+ y ^2) /(2* sigma ^2) ); 31 end 1 function [ outim ] = removesmallcc ( tim, threshold ) 2 % Compute the 8- connected cc, then remove threshold % of them 3 [L, num ] = bwlabel (tim, 8); 4 labelcount = zeros ( num,1) ; 5 alllabel = unique (L); 6 7 for i = 1: numel ( alllabel ) 8 labelcount (i)= sum ( sum (L== alllabel (i))); 9 end 10 [ afsort, sortin ]= sort ( labelcount ); 11 startin = ceil ( threshold * numel ( alllabel )); outim = zeros ( size ( tim )); for i = startin : numel ( alllabel ) 16 outim = outim + (L== sortin (i)); 17 end end 15

16 1 function [ threshold_value ] = threshold_percent ( array, percentage ) 2 % Return the threshold value such that certain percantage values of the original 3 % array are larger than the threshold. 4 onedarray = array (:) ; 5 onedarray = sort ( onedarray ); 6 n= numel ( onedarray ); 7 index = ceil (n* percentage ); 8 threshold_value = onedarray ( index ); 9 end 16