Feature enhancing aerial LiDAR point cloud refinement

Transcription

1 Feature enhancing aerial LiDAR point cloud refinement Zhenzhen Gao a and Ulrich Neumann b Dept. of Computer Science, Univ. of S. Calif./941 W. 37th Pl., LA, USA ABSTRACT Raw aerial LiDAR point clouds often suffer from noise and under-sampling, which can be alleviated by feature preserving refinement. However, existing approaches are limited to only preserving normal discontinuous features (ridges, ravines and crest lines) while position discontinuous features (boundaries) are also universal in urban scenes. We present a new refinement approach to accommodate unique properties of aerial LiDAR building points. By extending recent developments in geometry refinement to explicitly regularize boundary points, both normal and position discontinuous features are preserved and enhanced. The refinement includes two steps: i) the smoothing step applies a two-stage feature preserving bilateral filtering, which first filters normals and then updates positions under the guidance of the filtered normals. In a separate similar process, boundary points are smoothed directed by tangent directions of underlying lines, and ii) the up-sampling step interpolates new points to fill gaps/holes for both interior surfaces and boundary lines, through a local gap detector and a feature-aware bilateral projector. Features can be further enhanced by limiting the up-sampling near discontinuities. The refinement operates directly on points with diverse density, shape and complexity. It is memory-efficient, easy to implement, and easily extensible. Keywords: aerial LiDAR point cloud, refinement, bilateral filter, smoothing, up-sampling, feature-preserving, feature-enhancing, edge-aware 1. INTRODUCTION Aerial LiDAR point clouds captured using commercial laser range scanners are invariably noisy, mostly caused by scanner artifacts and alignment errors between scans. 1 In addition, occluded or sharp regions (such as boundaries, ridges, ravines, crest lines, etc.) are often under-sampled. Due to noise and under-sampling, a direct rendering or surface reconstruction of raw points produces grainy surfaces, gaps and holes, and irregular boundaries. These artifacts are especially visually-disturbing for buildings where planar surfaces and straight lines are universal, as shown in Figure 1(a) a rendering example of a raw building roof. As buildings are the most important objects in urban scenes, the focus of this paper is to alleviate visual symptoms of aerial LiDAR building points. (a) Figure 1. Rendering of a building roof. (a) Raw points. (b) Refined points. Refining raw points in a pre-process is one effective way to address the above problem. In contrast to point data gathered by other scanners, 2, 3 aerial LiDAR building points bear two unique properties that make the refinement more challenging. Firstly, as captured by laser scanners equipped on a low-flying aircraft, the sampling resolution of aerial LiDAR points is relatively low. Compared to other point data that have hundreds or Further author information: (Send correspondence to Zhenzhen Gao) Zhenzhen Gao: zhenzheg@graphics.usc.edu Ulrich Neumann: uneumann@graphics.usc.edu (b)

2 even thousands of points/m 2, a typical aerial LiDAR data only has points/m 2. Most refinement approaches rely on dense data to infer and reconstruct the latent object. The low sampling rate makes those approaches inapplicable. Due to lack of information, the goal of accurately recovering the underlying buildings for aerial LiDAR points is impractical. Secondly, aerial LiDAR data can be characterized as being 2.5D 4 in nature. For a building, the sensor is only able to capture details of roofs but few points on vertical walls, leaving roof boundary points discontinuous in the positions. As far as we know, existing point-based refinement approaches cannot regularize such boundary points. To make the problem tractable, we take as input a raw oriented aerial LiDAR point cloud of a single building, which has all points lie on the roof but no points on other objects like trees, walls or ground. The building consists of either a single roof block or several roof blocks of different heights (Figure 11). The only assumption we make is piece-wise smoothness of roofs regardless of shape and complexity. A smooth surface can be described as one having smoothly varying normals 5 whereas features appear as discontinuities either in the normals or in the positions. Ridges, ravines and crest lines are formed by discontinuities in the normals where the underlying surface is still continuous, they are called normal discontinuous features. Formed by discontinuities in the positions, boundaries are called position discontinuous features. We aim at a refinement approach that is fast, robust, memory-efficient, easy to implement, and easily extensible to other objects and large data sets. The output is a new set of points with smoothed noise, filled gaps and holes, and enhanced features both of normal and position discontinuities. The presented refinement approach extends recent developments in geometry smoothing and up-sampling to leverage unique properties of aerial LiDAR building points. Importantly, position discontinuous features, i.e., boundary points, are explicitly regularized. The feature-aware two-step approach is guided by normal and boundary direction (tangent direction of the underlying line of a boundary point). The first smoothing step applies a two-stage robust bilateral filtering, which first filters normals and then updates positions under the guidance of the filtered normals. A separate similar pass explicitly filters boundary directions and then updates boundary positions to match the new directions. This step effectively removes noise as well as preserves features. The second up-sampling step uses a local detector to locate gaps of the latent surface, and then fills with interpolated new points via a feature preserving bilateral projection operator. Gaps on boundaries are detected and filled explicitly under a similar process. A global uniform point density can be achieved, which is adjustable by a single parameter of a global neighborhood radius. Finally, the up-sampling step is extended to enhance features by inserting new points only in the vicinity of features. Figure 1(b) shows that our approach is effective to remedy perceptual artifacts conveyed in Figure 1(a). Contributions: To the best of our knowledge, we are the first point-based refinement approach allowing preserving and enhancing position discontinuous features. We extend recent developments in geometry refinement to leverage unique properties of aerial LiDAR building points: sparse sampling density and the 2.5D characteristic RELATED WORK This section first reviews work closely related to our smoothing and up-sampling step, then introduces existing approaches that can be adapted to regularize position discontinuous features. 2.1 Smoothing The literature is abundant in feature preserving surface smoothing, 6 11 mostly inspired by image processing work on scale-space and anisotropic diffusion. 12 All diffusion-based feature preserving techniques are, in essence, all local and iterative, therefore they require significant computation time. 13 A popular alternative to anisotropic diffusion is the bilateral filter, which was also originally proposed in 14, 15 image processing. It is a non-linear filter where the output is a weighted average of the input. Durand et al. 16 found that bilateral filtering is essentially similar to anisotropic diffusion, but a more stable non-iterative robust estimator. Methods of 13, 17 extend the bilateral filter to feature preserving mesh smoothing based on

3 robust statistics 18, 19 and local first-order predictors of the surface. The weight depends not only on the spatial distance, but also on a similarity term that penalizes values across features. The bilateral filtering is also extended to the application of feature preserving point cloud smoothing. The work of 20 combines the bilateral filtering idea with moving least squares (MLS) methods. 21 The work of 22 uses and filters surface curvature of an iteratively computed higher order surface approximation. Several two-stage feature preserving surface smoothing methods, 23, 24 which first filter normals and then update positions under the guidance of the filtered normals, have shown improved results than a single pass filtering of only normals or positions. The presented smoothing step applies a similar two-stage bilateral filtering. 2.2 Up-sampling Lipman et al. 25 introduce the parameterization-free locally optimal projector (LOP) for surface approximation from point-set data. Up-sampling can be achieved by a few iterations of LOP. A weighted version of LOP 26 can deal with non-homogeneous point density. The point cloud consolidation scheme of 27 extends LOP to allow evenly distributed up-sampling. However, due to the iterative nature, LOP-based methods become inefficient when applying to large data sets. Several up-sampling methods are based on a smooth point-based representation called MLS surface. First introduced in, 21 the MLS surface is defined implicitly by a local projection operator. Algorithms of 28, 29 upsample a point set through Voronoi point insertion in local tangent spaces followed by MLS projection. The work of 30 achieves uniform up-sampling based on a particle simulation and MLS projections. However, these approaches are computationally expensive, because the generation of either Voronoi diagram or particle system is time-consuming, and the computation of the local projection operator is a non-linear optimization problem. Targeted at real-time point cloud refinement, the up-sampling algorithm of 31 iteratively and uniformly inserts new points over a local neighborhood. Our up-sampling step applies a simplified neighborhood selection of 31 but only inserts new points near gaps. By coupling with the fast bilateral projection operator, 27 through one single pass, our approach is able to deliver feature preserving and global uniform point sampling for input points with non-homogeneous density. 2.3 Position discontinuous feature regularization Existing point cloud refinement methods only target at preserving normal discontinuous features on continuous surfaces. Several feature smoothing approaches identify features in a point cloud, connect the feature points using a minimum spanning tree or simple snapping, and return a set of complete and smooth lines or curves via fitting. Recent primitive-based geometry modeling 35 fits local planes to points and defines features as intersections of planes. Through orientation and placement alignment of planes to global regularities, they can create building models of high accuracy. These approaches can be adapted to regularize position discontinuous features like boundary points, however, they are not suitable for point-based applications since feature points are converted to lines or curves. A recent surface reconstruction approach 36 also fits planes to points and locates creases/corners as plane intersections. By re-sampling the planes back to points, the point representation can be preserved. However, geometry fitting is known to be time-consuming and sensitive to noise. The presented new boundary regularization approach does not require fitting while preserving the point representation throughout the process. 3. PREPARATION The aerial LiDAR point cloud taken as input is a set of 3D points P = {p 1, p 2,..., p N }, p i R 3, where p i is a position and the corresponding normal is denoted as n pi. The average point spacing is s. Due to the 2.5D characteristic of the data, 4 positions of P cover a much larger span along x and y axis than that along z axis. We use a k-d tree 37 partitioned on the x-y plane to store P for its simplicity in implementation and good performance in neighbor searching. Two local neighborhood definitions are supported.

4 k-nearest neighbors N k (p), which are k neighbors nearest to a point p P, regardless of the distance. r-radius neighbors N r (p), which include all neighbors in the sphere of center p and radius r. The input point set P represents roof blocks of a single building, so roof boundary points are position discontinuous features. These features can be differentiated from interior points based on local environment analysis, as neighbors of a boundary point cover only part of the environment but neighbors of an interior point are distributed all around. For a point p P, its r-radius neighborhood N r (p) is analyzed similar to 38 to decide whether it is a boundary point or not. To find a pertinent one-ring neighborhood, we set r = βs, β [1, 2] as in. 31 Such a neighborhood selection may include points from nearby roof blocks different from that of p. By removing neighbors from N r (p) whose z-coordinate difference with p is larger than h, neighbors are limited to those on the same roof block. Neighbor points p i are then projected onto the tangent plane of p and sorted such that their projections q i form increasing counterclockwise angles with the positive x-axis. Finally, we check the angle difference between any π two immediate ordered neighbors to see if it exceeds a limit θ c, e.g. 2. Once this criterion is fulfilled, p is classified as a boundary point. Figure 2 shows that boundary points of each roof block are correctly labeled, with parameters β = 2, θ c = π 2 and h = 2s. Figure 2. Boundary points (red) detection. We use B to denote the set of all boundary points of the input data and I for all interior points, where I = P B. Three additional local neighborhoods for boundary points are defined as below. k-nearest same-roof boundary neighbors N Bs k (p), which are k nearest boundary points that share the same roof block with p. r-radius boundary neighbors N B r (p) collect boundary points whose projection on x-y plane are within the r-radius circle at p s projection. Using 2D distance is to cover boundary points on different roof blocks but are relevant to the regularization of p. r-radius same-roof boundary neighbors Nr Bs (p) collect r-radius boundary neighbors that share the same roof block with p. We have Nr Bs (p) Nr B (p). 4. SMOOTHING The smoothing step is a two-stage bilateral filtering performed on all points (Section 4.1) and then on boundary points only (Section 4.2). The first stage filters normals or boundary directions. The second stage displaces point positions under the guidance of smoothed normals or boundary directions. Bilateral filtering is selected because it is a simple, efficient, non-iterative, and feature preserving robust estimator. The estimate for a point is the weighted average of predictions from neighboring points. The weight of a neighbor depends not only on the spatial distance (spatial weight f), but also on the influence difference (influence weight g) that penalizes values across features. For a point p and one of its neighbors p i, the spatial weight f always takes the Euclidean distance p p i. The influence weight g has more options. Jones et al. 13 uses the distance between the prediction and the original position: Π pi (p) p (1)

5 (a) Input points. (b) Normal estimation. (c) Normal smoothing. (d) Position smoothing. (e) Boundary direction estimation. (f) Boundary direction smoothing. (g) Boundary smoothing. position (h) Output points. Figure 3. The smoothing process on a synthetic building with two roof blocks. One edge of the wedge sits right above one edge of the rectangular roof. The data is corrupted with Gaussian noise of zero-mean and standard deviation of 1% of the diagonal of the bounding box. (b)-(d): cross-sections and close-up views of the squared area in (a). (e)-(f): close-up views of boundary points of the rectangular area in (a). where the prediction is simply the projection of p onto the plane through p i with normal n pi : Π pi (p) = p + (p i p) n pi n pi. Fleishman et al. 17 instead uses: < n p, p i p > (2) to give less weights to neighbors that do not lie on the plane through p with normal n p. Duguet et al. 22 uses normal difference: 1 < n p, n pi > (3) Given a point p on edge e 1 and some of its neighbors on another edge e 2, Figure 4(a) shows the case when Equation (1) gives all neighbors on e 2 a large g, and Figure 4(b) shows the case when Equation (2) gives neighbors on e 2 that are close to p a large g. But Equation (3) can handle both cases by assigning a much smaller g to all points on e 2, as normal difference is the best measurement to distinguish points on different surfaces of a feature. Experiments also show that Equation (3) works best for both normal and position filtering, so it is taken for the influence weight g. (a) Figure 4. Special cases for different influence weight g. Arrows for normals; dotted lines for vectors connecting two points; blue point is p. Red points will get large g if evaluated with (a) Equation (1). (b) Equation (2). The bilateral filter of 22 applies an additional quality coefficient to weigh the sum of least square errors of a fitted second-order jet surface. Experiments with a similar quality coefficient show negligible improvement for our data, so this coefficient is not included. The same filtering equation is used for normal, position, boundary direction as well as boundary position, (b)

6 the quantity to be smoothed is denoted as X: X (p) = 1 k(p) p i N r(p) where k is the sum of weights as the normalizing factor k(p) = f( p i p )g(1 < n p, n pi >) p i N r(p) The spatial and influence weight functions are Gaussians: f( p i p )g(1 < n p, n pi >)X pi (p) (4) ( ( ) f( p i p ) = e p 2 i p σ f, g(1 < n p, n pi >) = e 1 n T p np i 1 cos(σg ) The spatial weight f is to remove noise by including a large number of neighbors in the estimate, we use a constant σ f = r 2 as.13 The influence weight g is to handle outliers by decreasing the weight of neighbors with large normal differences, we use a constant angle parameter σ g = 15 as Smoothing for all points Normal Smoothing For each point p P, we run the bilateral filter over its neighborhood N r (p), where the normal prediction from p i N r (p) is the normal of p i, i.e., n pi. Figure 3(c) shows the smoothed normals. ) 2 Position Smoothing Now, we run the bilateral filter to smooth positions of each point p P, where the prediction from a neighbor point p i N r (p) is the projection of p onto the plane through p i with normal n pi. Instead of displacing p to the new position p, we constrain the point to move along its normal n p, i.e., move to the projection of p on the line through p with direction n p. This preserves the original sampling density. Figure 3(d) shows the smoothed positions. 4.2 Smoothing for boundary points Boundary Direction Estimation To estimate boundary direction of a point b B, similar to, 32 we take the direction of the underlying line of b, which is the eigenvector corresponding to the largest eigenvalue of the covariance matrix formed by b and its neighbors. The boundary direction of b is denoted as d b. Boundary Direction Smoothing Equation (4) should only consider boundary points since interior points do not affect boundary alignment. The 2.5D characteristic of building structures 4 infers that neighboring roof blocks are connected by the same vertical walls. Therefore, boundary points of different roof blocks should also influence each other when projected on the x-y plane. We break the smoothing of boundary direction into two parts: on the x-y plane and along z-axis. For the smoothing on x-y plane, all boundary neighbors in Nr B (b) are evaluated. In Equation (4), the prediction from a neighbor point b i is d bi projected on the x-y plane; f weighs the Euclidean distance between b and b i projected on the x-y plane; and the normal difference in g is adapted to the difference of boundary directions projected on the x-y plane. The smoothing along z-axis only considers boundary neighbors from the same roof block (Nr Bs (b)). Similarly, Equation (4) is updated with projections on z-axis for the prediction and f. But g uses the difference of 3D boundary directions, as the projected difference on z-axis is often too small. Figure 3(f) shows the smoothed boundary directions.

7 Boundary Position Smoothing Similar to the smoothing of boundary direction, boundary position is also smoothed in two parts. The complete prediction from a neighbor point b i is the projection of b onto the line through b i with direction d bi. It is split into two projections on the x-y plane and along z-axis for Equation (4). The displacement of b is restricted to move on the tangent plane defined by b and n b, and along the direction perpendicular to d b. The constraint preserves results of previous smoothing passes and the original sampling density along boundary direction. The underlying continuous surface is formed by both boundary and interior points, it will be problematic if the displacement of boundary points is independent of that of interior points. Given a boundary point b to be moved to a new position b, the displacement vector is v = b b. Interior points in the neighborhood N v (b) (dotted circle in Figure 5 (a)) may be left outside the boundary after re-positioning b, e.g., the blue point in Figure 5 (b). To avoid the problem, in addition to displacing b to b, we also displace every interior neighbor that is in the way of the displacement of b. For any interior neighbor b i N v (b) b i I, if it is in the way determined by v T (b i b) > 0, it should be displaced to a new position b i = b i + v, as seen in Figure 5 (c). Figure 3(g) shows the smoothed boundary positions. (a) (b) (c) Figure 5. The problem caused by displacing boundary points. (a) Boundary points (red) will be moved to the dotted line. The dotted circle is centered at the red point with radius v. (b) After displacing boundary points only. The interior point (blue) is left outside the boundary. (c) After displacing boundary and affected interior points. The problem of (b) is avoided. 5. UP-SAMPLING The up-sampling step treats interior and boundary points separately. The insertion of new interior points aims to fill gaps on the latent surface, while the insertion of new boundary points is to fill gaps along the underlying boundary lines. For both cases, we first use a local detector inspired by boundary detection in Section 3 to locate gaps, then apply an edge-aware bilateral projector 27 to insert new points. Finally, the up-sampling step is extended to enhance features. 5.1 Interior point up-sampling The boundary detection in Section 3 is based on the neighborhood environment analysis of point p. The angle between two immediate ordered neighbors define the spatial clearance within the r-radius sphere centered at p. Under-sampling is most likely to happen in the area formed by two neighbors that have the maximum angle difference, as depicted in Figure 7(a). This idea is used to design a simple and effective local gap detector. For each interior point p I, given a global radius γ, we project all neighbors p i N γ (p) onto its tangent plane centered at p. Note that p i can be either interior or boundary. All neighbors are ordered by their counterclockwise angle formed with the positive x-axis. A gap is found if the maximum angle difference of any two immediate neighbors exceed a threshold θ c. A new point is inserted to fill the gap as detailed below. This process continuous till the maximum angle difference of p is smaller than θ c. Given an under-sampled region formed by p and two neighbors p i and p j, we now find another point q to interpolate the new sample. For each neighbor point p k lies in between the spheres with radius γ and γ max, i.e., p k N γmax (p) p k N γ (p), ordered by increasing Euclidean distance to p, we project p k onto the tangent plane through p with normal n p, and label it as q if it fulfills the following conditions: p k is ordered in between of p i and p j by the angle criterion. p k is not too close to p i and p j, i.e., the angle formed by p i pp k and p k pp j are both larger than θ min.

8 (a) Smoothed points. (b) Interior upsampling. (c) Boundary upsampling. (d) Interior enhancing. (e) Boundary enhancing. (f) Output points. Figure 6. The up-sampling process on the data of Figure 3(h). (a)-(e) show close-up views of the square in Figure 3(a). To find neighbors covering the entire environment of an interior point, we use θ c = π 2 and γ max = βs as in Section 3, and θ min = π 8 as in.31 Figure 7(b) shows the end point q found by the above criteria. Now, the task is to insert a new point p new with normal n new based on end points p and q. The edge-aware bilateral projector 27 is applied to constrain the projection direction to be along n new, i.e.,, where b new is the base point and d new the distance along n new. p new = b new + d new n new (5) (a) (b) (c) Figure 7. 2D view of the up-sampling process. (a) Point p (blue) and its γ-radius neighbors (black). p i and p j form the under-sampled area. (b) Interpolation end point q (red) and the base point b new (green). Inner circle has γ-radius and outer circle has γ max-radius. (c) Two candidate positions (crosses) to insert a new point based on b new. For regular sampling in the local region defined by the sphere at p with radius γ, the base location needs to be within that sphere, as p q b new = p + min{, γ} (q p) 2 Figure 7(b) shows the selected base point b new. Similar to, 27 the projection distance under a given normal n is obtained by (n T (b new p i ))f( b new p i )g(1 < n, n pi >) d new = p i N γmax (b new) p i N γmax (b new) f( b new p i )g(1 < n, n pi >)

9 Constrained to the set of n p, n q, the normal n new is selected as the one with the minimum d new evaluated above. Now, a new point can be inserted with Equation (5). As illustrated in Figure 7(c), b new is inside the γ-sphere at p, if n new = n p, p new will be within the sphere. Thus, the gap between p i and p j is filled, and the gap detector can proceed without stalling. However, if n new = n q, p new will be pushed outside the sphere, leaving the gap detector in an infinite loop. As a remedy, n new is constrained to be n p. In case n q wins the d new evaluation above, we skip this end point q and continue the search in neighbors. It won t be problematic if no new points are inserted for the gap formed by p, p i and p j, as this gap will be detected and filled when processing point q. We simply skip this gap for now and continue to the second largest gap till all gaps of p are processed. (a) Original (b) γ = 2.0 (c) γ = 1.3 (d) γ = 1.0 Figure 8. Interior point up-sampling for a 2D surface with both inner and outer boundaries. The original sampling is non-uniform. Inserted points are in red. The new point p new is immediately added to the storage k-d tree as an interior point. Therefore, a single pass of all interior points can fill all gaps in a neighborhood of γ radius. Compared to the up-sampling of 27 which relies on a global priority queue to manage the insertion order, our method is more memory efficient by locating under-sampled areas via a simpler local detector. As seen in Figure 8, our approach is effective to deliver global uniform sampling, which is easily adjustable by the radius γ. It also shows that the approach works for input data with non-uniform sampling while preserving both inner and outer boundaries. Figure 6(b) shows the result of interior up-sampling. 5.2 Boundary point up-sampling The up-sampling of boundary points is similar to that of interior points. One difference is to only consider boundary points in a neighborhood. The other difference is to use boundary directions instead of normals when evaluating d new and n new. Figure 6(c) shows the result of boundary up-sampling. 5.3 Feature enhancing extension By inserting more points near a feature, the feature is enhanced. Normal discontinuous features are interior points and position discontinuous features are boundary points. We enhance two types of features by separately up-sampling interior and boundary points near discontinuities. Given an interior point p and its N γ (p), the neighbors share similar normal with p form a new neighborhood Nγ sn (p), where for any p i Nγ sn (p), we have 1 n T p n pi < 1 cos(σ g ). The points in Nγ sn (p) are all from the same smooth surface as p. With this neighborhood, an under-sampled point found by the local gap detector is a point closest to a feature in a local region. γ is set as the final radius of the regular up-sampling step, in order to skip interior points that are far away from features. The set of near feature points is denoted as F. For each point f F, we insert new points to fill gaps with a user defined radius γ e (γ e < γ) as detailed in Section 5.1. Newly inserted points are also added to F. Note that the neighborhood of f is still formed by points in P as before. The resulting point cloud has uniform sampling of radius γ in flat area and uniform sampling of radius γ e near features. As γ e < γ, we see more densely sampled features, i.e., features are enhanced. Figure 6(d) shows the result with interior features enhanced. The feature enhancing up-sampling for boundary points is easily achieved by replacing normals with boundary directions. Figure 6(e) shows the result with boundary features enhanced. Finally, the rendering of feature enhanced points is shown in Figure 6(f), which has greatly improved visual appearance than the rendering of raw points in Figure 3(a).

10 6. EXPERIMENTS Robustness. Results of the smoothing step are used to show the robustness to handle noise. The kernel of the smoothing step is the well-studied bilateral filter, one is referred to previous studies 13, 17, 22 for its robustness on smoothing interior surfaces. For the smoothing of noisy boundaries, we use a synthetic square with a circular hole where both straight and curved boundaries are present. Gaussian noises are added by perturbing each point a random vector with size 50% and 100% of the point spacing. Figure 9 visually shows that the smoothing quality does not degrade much as the noise increases. Table 1 measures average distance and standard deviation from smoothed interior and boundary points to the ground truth, to quantitatively verify the robustness. Figure 8 shows that the up-sampling step is also robust to handle non-homogeneous point density. (a) 50% noise, original (b) 50% noise, smoothed (c) 100% noise, original (d) 100% noise, smoothed Figure 9. Smoothing results over different noise levels. The distance between an actual point and its ground truth is color coded as the rightmost bar. Degradation of smoothing quality of both interior surface and boundaries is negligible. Table 1. Errors for Figure 9. MD: mean distance; SD: standard deviation. Data set Interior-MD Interior-SD Boundary-MD Boundary-SD (a) (b) (d) (e) Stability. The test data sets are sampled from a synthetic wedge with vastly different densities, and each is corrupted with 1% Gaussian noise as that of Figure 3(a). Figure 10 compares the rendering results before and after the refinement. From left to right, wedges have decreasing number of samples: 100%, 20% and 5% proportion to the samples of the leftmost wedge (15, 625 samples). Same parameters are applied, so the resulting refined wedges have approximately the same sampling density. Our approach is stable as most normal and position discontinuous features are sharply recovered for all data sets. Table 2 quantitatively demonstrates the stability by measuring distances from points to the ground truth surface. (a) 100%, original (b) 100%, refined (c) 20%, original (d) 20%, refined (e) 5%, original (f) 5%, refined Figure 10. Stable refinement over various input sampling density. Point clouds of real buildings with various density are also tested. Buildings of Figure 1, 2, 11(a)-(c) and 15 are extracted from the aerial LiDAR point cloud of Atalanta with 25 points/m 2 density, whereas buildings of Figure 11(d), 13 and 14 are from USC with 5.8 points/m 2 density.

11 Table 2. Errors for Figure 10. MD: mean distance; SD: standard deviation. Data set 100% 20% 5% Original-MD Refined-MD Original-SD Refined-SD Versatility. Rendering results of Figure 1 and 11 show that our refinement is effective to buildings with various roof shapes and complexity by providing smooth planar areas, filled gaps, and sharpened both normal and position discontinuous features. (a) Original (b) Original (c) Original (d) Original (e) Refined (f) Refined (g) Refined (h) Refined Figure 11. Versatility on real data with different shapes and complexity. Colored using a normal map. Extensibility. The presented approach is designed to refine roof points of a single building, however, it is easily extensible to other objects with similar surface properties, e.g. the ground, as seen in Figure 14. Because of the local property of the approach, it can be extended to large-scale data that needs streaming. Figure 14 shows part of USC to demonstrate that several buildings as well as the ground can be refined in a single pass with a single set of parameters. (a) 100% noise, original (b) After RIMLS smoothing (c) After our smoothing Figure 12. Smoothing results for the data set in Figure 9(c). Our approach has less errors on boundaries. Comparison with previous work. The presented approach is superior to previous point-based refinements in the sense that position discontinuous features are explicitly regularized. Figure 12 visually compares smoothed

12 results of RIMLS39 and the presented smoothing step, our approach has obviously less errors on boundaries. A recent surface reconstruction36 and geometry modeling34 can regularize position discontinuous features as intersections of fitted planes. They have shown to be effective for straight boundaries. But because of the planar assumption on intersecting surfaces, curved boundaries are not well handled, as seen in Figure 13(c). Without any geometric assumption, our approach provides smoother boundary as seen in Figure 13(d) which is also more accurate as compared to the ground truth image in Figure 13(a). (a) Ground truth image (b) Original points (c) Geometry models (d) Refined points Figure 13. Comparison of geometry modeling and visually-complete rendering of points refined by our approach. Our approach provides better looking and more accurate curved boundaries. Performance. The testing computer is equipped with dual-core Intel c Core(TM) i3 3.20GHz CPU, 6GB RAM, Seagate c 7200RPM hard disk and Windows 7 c. The program uses a single core without any optimization. Table 3 lists the processing time for each step. The refinement is fast in the sense that every step takes only a few seconds for all test data. Table 3. Processing time (in seconds) of each refinement step. IP-N: number of input points; SM-T: time for smoothing; US-T: time for up-sampling; US-N: number of points after up-sampling; EU-T: time for feature enhancement up-sampling; OP-N: number of output points. Data set Figure 1 (b) Figure 6 (f) Figure 11 (e) Figure 11 (f) Figure 11 (g) Figure 11 (h) Figure 15 (b) IP-N 5,594 23,500 9,079 7,546 4, ,310 4,498 SM-T US-T US-N 11,597 52,303 15,603 12,146 9, ,164 6,403 EU-T OP-N 16,590 57,461 25,720 21,409 16, ,279 7,913 Applications. The refinement is a useful pre-process to deliver visual improvement for direct rendering of points as exemplified in Figure 11. Visually-complete aerial LiDAR point cloud visualization40 is a direct beneficiary of the refinement. As seen in Figure 14, walls look more visually appealing as the boundary roof points are effectively regularized. To take advantages of the refinement, real-time visualization applications can pre-process the smoothing step and the regular up-sampling step with γ = s to handle noise and under-sampling of the entire data; and then dynamically perform the feature enhancing up-sampling step in a view dependent manner to add just enough details for distant objects. The refinement also improves visual quality for surface reconstruction of points, exemplified by APSS41 and RIMLS39 from MeshLab.42 We use default parameter setting in the experiment; one can achieve better results

13 (a) Original (b) Refined Figure 14. Refinement for a larger data set with several buildings. by fine tuning of parameters. Figure 15 visually compares the results of these two surface reconstruction methods performed on a point cloud before and after the refinement. It shows that surface reconstruction quality is greatly improved by refinement because of smoother planes and sharper features. (a) Before: points (b) After: points (c) Before: APSS (d) After: APSS (e) Before: RIMLS (f) After: RIMLS Figure 15. Quality of surface reconstruction gets improved after the refinement on raw points. 7. CONCLUSION By extending recent developments in geometry refinement to leverage unique properties of aerial LiDAR building points, we present a specialized two-step refinement approach where position discontinuous features (boundary points) are explicitly regularized. The smoothing step applies a two-stage feature preserving bilateral filtering to smooth out noise in normals, boundary directions and positions. The up-sampling step employs a local gap detector and a feature preserving bilateral projector to deliver uniform up-sampling. A simple extension allows feature enhancing by increasing samples in the vicinity of features. The presented approach is fast, robust, simple to implement, and easily extensible. Experiments on both synthetic and real data show that the refinement effectively removes noise, fills gaps, and regularizes both normal and position discontinuous features. It is a beneficial pre-process for other applications such as direct rendering and surface reconstruction. Due to the local and non-iterative nature, it is straightforward to parallelize each refinement operation. This is one future work for better performance. The refinement quality is limited by a point cloud s sampling density. To achieve better refined aerial LiDAR points, one direction is to resort to other information such as aerial images; another direction is to add more restricted geometric assumptions as in urban modeling. Incorporating refinement into real-time city-scale point cloud visualization is another challenging future work. ACKNOWLEDGMENTS We thank the reviewers for their useful comments. We thank Airborne 1 Corp. for providing real LiDAR data. We appreciate the programming support from Miao Wang, useful comments, reviews and support from Dr. Luciano Nocera and Dr. Saty Raghavachary. REFERENCES [1] Gross, M. and Pfister, H., eds., [Point-based graphics ], The Morgan Kaufmann Series in Computer Graphics (2007). [2] AIM@SHAPE Shape Repository. (1999). [3] The Stanford 3D Scanning Repository.

14 [4] Zhou, Q.-Y. and Neumann, U., 2.5d dual contouring: A robust approach to creating building models from aerial lidar point clouds, in [Computer Vision - ECCV 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part III], Daniilidis, K., Maragos, P., and Paragios, N., eds., Lecture Notes in Computer Science 6313, , Springer (2010). [5] Lee, K.-W. and Wang, W.-P., Feature-preserving mesh denoising via bilateral normal filtering, in [Computer Aided Design and Computer Graphics, Ninth International Conference on], 6 (2005). [6] Bajaj, C. L. and Xu, G., Anisotropic diffusion of subdivision surfaces and functions on surfaces, ACM TRANSACTIONS ON GRAPHICS 22 (2002). [7] Clarenz, U., Diewald, U., and Rumpf, M., Anisotropic geometric diffusion in surface processing, (2000). [8] Desbrun, M., Meyer, M., Schrder, P., and Barr, A. H., Anisotropic feature-preserving denoising of height fields and bivariate data, in [In Graphics Interface], (2000). [9] Desbrun, M., Meyer, M., Schröder, P., and Barr, A. H., Implicit fairing of irregular meshes using diffusion and curvature flow, in [Proceedings of the 26th annual conference on Computer graphics and interactive techniques], SIGGRAPH 99, , ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1999). [10] Meyer, M., Desbrun, M., Schrder, P., and Barr, A. H., Discrete differential-geometry operators for triangulated 2-manifolds, 35 57, Springer-Verlag (2002). [11] Zhang, H. and Fiume, E., Mesh smoothing with shape or feature preservation, Advances in Modeling, Animation, and Rendering, (2002). [12] Perona, P. and Malik, J., Scale-space and edge detection using anisotropic diffusion, IEEE Trans. Pattern Anal. Mach. Intell. 12, (July 1990). [13] Jones, T. R., Durand, F., and Desbrun, M., Non-iterative, feature-preserving mesh smoothing, in [ACM SIGGRAPH 2003 Papers], SIGGRAPH 03, , ACM, New York, NY, USA (2003). [14] Smith, S. M. and Brady, J. M., Susan - a new approach to low level image processing, International Journal of Computer Vision 23, (1995). [15] Tomasi, C., Bilateral filtering for gray and color images, (1998). [16] Durand, F. and Dorsey, J., Fast bilateral filtering for the display of high-dynamic-range images, in [Proceedings of the 29th annual conference on Computer graphics and interactive techniques], SIGGRAPH 02, , ACM, New York, NY, USA (2002). [17] Fleishman, S., Drori, I., and Cohen-Or, D., Bilateral mesh denoising, in [ACM SIGGRAPH 2003 Papers], SIGGRAPH 03, , ACM, New York, NY, USA (2003). [18] Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., and Stahel, W. A., [Robust Statistics: The Approach Based on Influence Functions (Wiley Series in Probability and Statistics)], Wiley-Interscience, New York (2005). [19] Huber, P. J., ed., [Robust statistics], John Wiley and Sons (1981). [20] Mederos, B., Velho, L., and de Figueiredo, L. H., Robust smoothing of noisy point clouds, in [PROC. SIAM CONFERENCE ON GEOMETRIC DESIGN AND COMPUTING], (2003). [21] Levin, D., Mesh-independent surface interpolation, in [Geometric Modeling for Scientific Visualization], (2003). [22] Duguet, F., Durand, F., and Drettakis, G., Robust Higher-Order Filtering of Points, Tech. Rep. RR-5165, INRIA (Apr. 2004). [23] Nociar, M. and Ferko, A., Feature-preserving mesh denoising via attenuated bilateral normal filtering and quadrics, in [Proceedings of the 26th Spring Conference on Computer Graphics], SCCG 10, , ACM, New York, NY, USA (2010). [24] Sun, X., Rosin, P. L., Martin, R. R., and Langbein, F. C., Fast and effective feature-preserving mesh denoising, IEEE Transactions on Visualization and Computer Graphics 13, (Sept. 2007). [25] Lipman, Y., Cohen-Or, D., Levin, D., and Tal-Ezer, H., Parameterization-free projection for geometry reconstruction, in [ACM SIGGRAPH 2007 papers], SIGGRAPH 07, ACM, New York, NY, USA (2007). [26] Huang, H., Li, D., Zhang, H., Ascher, U., and Cohen-Or, D., Consolidation of unorganized point clouds for surface reconstruction, in [ACM SIGGRAPH Asia 2009 papers], SIGGRAPH Asia 09, 176:1 176:7, ACM, New York, NY, USA (2009).

15 [27] Huang, H., Wu, S., Gong, M., Cohen-Or, D., Ascher, U., and Zhang, H., Edge-aware point set resampling, ACM Transactions on Graphics, 1 12 (2012). [28] Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., and Silva, C. T., Point set surfaces, in [Proceedings of the conference on Visualization 01], VIS 01, 21 28, IEEE Computer Society, Washington, DC, USA (2001). [29] Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., and T. Silva, C., Computing and rendering point set surfaces, IEEE Transactions on Visualization and Computer Graphics 9, 3 15 (Jan. 2003). [30] Pauly, M., Keiser, R., Kobbelt, L. P., and Gross, M., Shape modeling with point-sampled geometry, in [ACM SIGGRAPH 2003 Papers], SIGGRAPH 03, , ACM, New York, NY, USA (2003). [31] Guennebaud, G., Barthe, L., and Paulin, M., Real-Time Point Cloud Refinement, in [Symposium on Point-Based Graphics, Zurich, 02/06/ /06/2004], Alexa, M., Gross, M., Pfister, H., and Rusinkiewicz, S., eds., 41 49, Eurographics/IEEE Computer Society TCVG (juin 2004). [32] Daniels, J. I., Ha, L. K., Ochotta, T., and Silva, C. T., Robust smooth feature extraction from point clouds, in [Proceedings of the IEEE International Conference on Shape Modeling and Applications 2007], SMI 07, , IEEE Computer Society, Washington, DC, USA (2007). [33] Pauly, M., Keiser, R., Gross, M., and Zrich, E., Multi-scale feature extraction on point-sampled surfaces, (2003). [34] Zhou, Q.-Y. and Neumann, U., Fast and extensible building modeling from airborne lidar data, in [Proceedings of the 16th ACM SIGSPATIAL international conference on Advances in geographic information systems], GIS 08, 7:1 7:8, ACM, New York, NY, USA (2008). [35] Zhou, Q.-Y. and Neumann, U., 2.5d building modeling by discovering global regularities, Computer Vision and Pattern Recognition, IEEE Computer Society Conference (2012). [36] Lafarge, F. and Alliez, P., Surface reconstruction through point set structuring, in [Proc. of Eurographics], (2013). [37] Cormen, T. H., Stein, C., Rivest, R. L., and Leiserson, C. E., [Introduction to Algorithms], McGraw-Hill Higher Education, 2nd ed. (2001). [38] Linsen, L., Point cloud representation, tech. rep. (2001). [39] Oztireli, C., Guennebaud, G., and Gross, M., Feature Preserving Point Set Surfaces based on Non-Linear Kernel Regression, Computer Graphics Forum 28(2), (2009). [40] Gao, Z., Nocera, L., and Neumann, U., Visually-complete aerial lidar point cloud rendering, in [Proceedings of the 20th International Conference on Advances in Geographic Information Systems], SIGSPATIAL 12, , ACM, New York, NY, USA (2012). [41] Guennebaud, G. and Gross, M., Algebraic point set surfaces, in [ACM SIGGRAPH 2007 papers], SIG- GRAPH 07, ACM, New York, NY, USA (2007). [42] Cignoni, P., Corsini, M., and Ranzuglia, G., Meshlab: an open-source 3d mesh processing system, ERCIM News, (April 2008).