Edge-Aware Point Set Resampling

Transcription

1 Edge-Aware Point Set Resampling HUI HUANG Shenzhen Key Lab of Visual Computing and Visual Analytics / SIAT SHIHAO WU South China University of Technology MINGLUN GONG Memorial University of Newfoundland DANIEL COHEN-OR Tel-Aviv University URI ASCHER University of British Columbia and HAO (RICHARD) ZHANG Simon Fraser University Points acquired by laser scanners are not intrinsically equipped with normals, which are essential to surface reconstruction and point set rendering using surfels. Normal estimation is notoriously sensitive to noise. Near sharp features, the computation of noise-free normals becomes even more challenging due to the inherent undersampling problem at edge singularities. As a result, common edge-aware consolidation techniques such as bilateral smoothing may still produce erroneous normals near the edges. We propose a resampling approach to process a noisy and possibly outlier-ridden point set in an edge-aware manner. Our key idea is to first resample away from the edges so that reliable normals can be computed at the samples, and then based on reliable data, we progressively resample the point set while approaching the edge singularities. We demonstrate that our Edge-Aware Resampling (EAR) algorithm is capable of producing consolidated point This work is supported in part by grants from NSFC ( ), Guangdong Science and Technology Program (2011B ), National 863 Program (2011AA010503), Shenzhen Science and Innovation Program (CXB A), NSERC (293127, and ) and the Israel Science Foundation. Authors addresses: H. Huang (corresponding author), Shenzhen Key Lab of Visual Computing and Visual Analytics/SIAT; hhzhiyan@gmail.com; S. Wu, South China University of Technology, China; M. Gong, Memorial University of Newfoundland; D. Cohen-Or, Tel-Aviv University, Israel; U. Ascher, University of British Columbia; H. Zhang, Simon Fraser University. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY USA, fax +1 (212) , or permissions@acm.org. c 2013 ACM /2013/01-ART9 $15.00 DOI / sets with noise-free normals and clean preservation of sharp features. We also show that EAR leads to improved performance of edge-aware reconstruction methods and point set rendering techniques. Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling Curve, surface, solid, and object representations General Terms: Algorithms Additional Key Words and Phrases: Point set resampling, normal estimation, upsampling, surface reconstruction, surfel point set rendering, point set, sampling, edge aware ACM Reference Format: Huang, H., Wu, S., Gong, M., Cohen-Or, D., Ascher, U., and Zhang, H Edge-Aware point set resampling. ACM Trans. Graph. 32, 1, Article 9 (January 2013), 12 pages. DOI = / INTRODUCTION The last two decades have seen a considerable amount of work on surface reconstruction from scanned point clouds. The use of points as a modeling and rendering primitive has also been studied extensively [Gross and Pfister 2007]. Both tasks heavily rely on having a quality normal associated with each point sample. In particular, popular surface reconstruction techniques such as Poisson [Kazhdan et al. 2006] and RBF [Carr et al. 2001] are guided by normal information and the well-known surfel-based point set rendering [Pfister et al. 2000] operates on oriented samples. However, points acquired by laser scanners are not intrinsically equipped with normals; they must be estimated from acquired image or geometry data. Acquired data are often tempered with noise and even outliers which hinder the computation of normals. We stress that normal estimation is sensitive to noise since normals are measured as first-order derivatives and numerical differentiation amplifies noise. Computing noise-free normals is much more challenging in the presence of sharp features, for example, see Figures 1 and 2. The desire to preserve the sharp features disallows the use 9

2 9:2 H. Huang et al. Fig. 1. Points (222K) acquired by a laser scan (b) are corrupted with noise and not intrinsically equipped with normals. Resampling the data without accounting for surface singularities may smear the sharp features after surface reconstruction (c). Our Edge-Aware Resampling (EAR) leads to a piecewise smooth reconstruction (d) while preserving the sharp edges. Point colors are the result of normal maps and the original object is shown in (a). of a naïve point smoothing operator prior to normal estimation, since such smoothing blurs the edges. Alternatively, one can compute normals over the noisy point set, typically via Principal Component Analysis (PCA), and then apply an edge-aware robust smoothing operator, such as bilateral filtering [Jones et al. 2004] or l 1 -minimization [Avron et al. 2010], to the normals. While such solutions can generally separate the two sides of an edge, in the vicinity of the edge some erroneous normals may still persist since the accuracy of these methods is limited by the noise level and sampling rate. In practice, data near an edge are often unavoidably undersampled and contain more noise than smooth regions, making it difficult to recover sharpness directly at the edges. Given a noisy point cloud with (possibly noisy) normals computed by PCA, as shown in Figure 2(a) and 2(b), respectively, Figure 2(c) and 2(d) show results of applying bilateral filtering over the points and PCA normals. As can be seen, near the edge some points have incorrectly assigned normal directions to agree with point normals at the wrong side of the edge. These seemingly small errors may be amplified on the reconstructed surface, leading to visible artifacts, as demonstrated in Figure 2(e) by an edge-aware surface reconstruction scheme using Robust Implicit Moving Least Squares (RIMLS) [Öztireli et al. 2009]. To circumvent this inherent problem, our strategy is to resample the noisy point set judiciously and in an edge-aware manner. In particular, since normal estimation close to edges is not reliable, our key idea is to first resample away from the edges. The result is a set of oriented points away from the edges that are endowed with reliable normals. Then based on these oriented points, we progressively resample the point set while approaching the edges. Thus, normal estimation proceeds from more reliable regions to less reliable regions (close to edges). Using the resampling strategy, we avoid having to compute noise-sensitive derivative measures in difficult regions. Resampling and normal estimation near edges are guided by reliable data, in particular, reliable normals. Our resampling algorithm is built on a robust edge-aware projection operator which produces samples away from edges and a novel bilateral projection operator which upsamples progressively so as to fill the edge regions. Resampling away from edges is enabled by incorporating normal information into the projection, allowing the projector to be edge aware. Robustness to noise and outliers is enabled by the use of the l 1 -median for data fitting. As we approach the edges, the bilateral projector considers both positional and current normal information when computing the base, direction, and distance of the projection. While the two latter attributes are the result of optimizing a bilateral objective function, the base location is chosen to achieve even point distribution and fast convergence. The result of our resampling algorithm is a consolidated point set with noise-free normals and uniform point distribution throughout and clean preservation of sharp features; see Figure 2(g). Previous works which also resort to resampling for point cloud consolidation assume that the underlying surface is smooth [Alexa et al. 2003; Lipman et al. 2007b; Huang et al. 2009; Miao et al. 2009; Öztireli et al. 2010]. Our Edge-Aware Resampling (EAR) scheme respects singularities, allowing effective handling of piecewise smooth surfaces. We show that EAR and its associated normal estimation facilitate the reconstruction of such surfaces by edgeoblivious methods such as Poisson [Kazhdan et al. 2006], Algebraic Point Set Surfaces (APSS) [Guennebaud and Gross 2007], and Delaunay-based Cocone [Dey and Giesen 2001]. At the same time, it enhances the performance of edge-aware reconstruction methods such as RIMLS; see Figure 2(h). We also show that EAR can be applied to upsample a point set, leading to superior rendering results [Pfister et al. 2000; Vergne et al. 2010] near sharp features. 2. RELATED WORK Most point set resampling schemes aimed at consolidating a raw point scan assume that the underlying surface is smooth. Early work by Alexa et al. [2001] upsamples a point set through Voronoi point insertion in local tangent spaces followed by Moving Least Squares (MLS) projection. Lipman et al. [2007b] introduce the parameterization-free, Locally Optimal Projector (LOP), which is driven by the use of the l 1 -median. LOP is shown effective in enhancing point sets while being robust to outliers and noise. Huang et al. [2009] propose a weighted version of LOP which better deals with nonhomogeneous point density in the input. Miao et al. [2009] develop a simplification scheme for nonuniformly distributed point samples, which adaptively reflects the intrinsic geometric features of the underlying shape. Recent work by Öztireli et al. [2010] presents a high-quality isotropic point sampling technique for a given surface representation based on spectral analysis, kernel methods, and matrix perturbation theory. None of these schemes was designed to handle sharp features, and some required reliable normals as part of the input. Our resampling approach also relies on l 1 -median, but unlike LOP, it estimates and utilizes normals, doing so in an edge-aware manner. Existing edge-aware point set resampling schemes either assume that the sharp features are given or resort to explicit construction of surface patches. For an accurate display of intersection curves between point set surface sheets, the works in Pauly et al. [2003], Guennebaud et al. [2004, 2008] present accurate rendering of sharp creases and corners. These methods assume that the intersection curves that define the sharp features are given. Fleishman et al. [2005] develop a piecewise surface-refitting algorithm which

3 Edge-Aware Point Set Resampling 9:3 (a) noisy input with an edge (b) PCA normals (c) bilateral point smoothing (d) bilateral normal smoothing (e) RIMLS + marching cubes on (d) (f) resampling(b) away from edge (g) resampling(f) close to edge (h) RIMLS + marching cubes on (g) Fig. 2. Comparison between bilateral smoothing and the proposed EAR approach. Each point is colored according to its normal direction and such normal maps are used throughout the article for rendering the results. The input (a) represents a fin shape containing a sharp edge; the point cloud (1K points) is corrupted with noise. In each subsequent result (b h), we show a 2D cross-section view taken at the blue curve indicated in (a), as well as a zoomed-in 3D view into the red window shown in (a). The results demonstrate that our resampling approach can effectively remove noise, provide reliable normals, preserve sharp features, and facilitate edge-aware reconstruction methods such as RIMLS [Öztireli et al. 2009]. allows projecting points and sampling the intersection of surface patches from two sides of a sharp feature. Their method requires a dense point set to start with while in practice, regions near sharp features are often undersampled [Salman et al. 2010]. Generally speaking, reliable fitting of surfaces is computationally expensive and sensitive to input data noise and outliers. Our method does not rely on explicit feature identification or patch fitting; it deals with heavy noise and undersampling near edges by first resampling away from edges and then propagating reliable information obtained there to the edges. Sharp features are dealt with extensively in the mesh processing literature. A typical task is to smooth a mesh surface while preserving sharp features, for example, Hildebrandt and Polthier [2004], Sun et al. [2007], and Huang and Ascher [2008]. Bilateral filtering [Fleishman et al. 2003; Jones et al. 2003] for mesh smoothing is related to our bilateral projector since both perform optimization under a bilateral objective function. However, mesh smoothing assumes an explicit connectivity among the points. More relevant are meshless methods which smooth point set surfaces that contain sharp features. The methods of Adamson and Alexa [2006] and Guennebaud and Gross [2007] assume explicit representation of sharp features. Salman et al. [2010] first extract sharp features from the point set and then directly generate piecewise smooth surface triangle meshes. Fleishman et al. [2005] and Lipman et al. [2007a] analyze the point set around sharp features and fit local surface patches over which points are projected. Algorithms proposed in Merigot et al. [2009] and Weber et al. [2010] mainly focus on detecting sharp features of a piecewise smooth surface from its point cloud sampling. Other recent robust methods [Öztireli et al. 2009; Avron et al. 2010] reconstruct surfaces from point clouds while respecting sharp features. All of these methods can produce less than satisfactory results, for example, erroneous normal estimates, near sharp features as a result of severe noise and undersampling. Resampling the point set prior to feature extraction, patch fitting, or surface reconstruction enhances the performance of these methods on a raw point scan. Given the importance of normals in surface reconstruction and point rendering, it is not surprising that there has been a tremendous amount of work on normal estimation from raw point data. Most methods resort to PCA or its variants [Hoppe et al. 1992; Alexa et al. 2001; Pauly et al. 2002; Mitra et al. 2004; Lange and Polthier 2005; Huang et al. 2009]. Near or on sharp features, PCA normals tend to smear information across discontinuities. Bilateral smoothing of PCA normals [Öztireli et al. 2009] provides some remedy but can still produce erroneous results (see Figure 2(d) and 2(e)). Moreover, sharp features are often undersampled in point scans which further hinders the performance of PCA. Another set of techniques use Voronoi poles [Dey and Sun 2006] or Voronoi-PCA [Alliez et al. 2007] to estimate normals. With undersampling, such interpolation-based methods cannot infer accurate normals near sharp features. In our work, we also rely on bilateral processing of PCA normals, but only to roughly detect edge locations so as to enable a resampling away from edges. Normals at points close to edges are then derived from reliable normals estimated away from edges instead of using PCA. 3. OVERVIEW Our EAR algorithm takes as input an unorganized and unoriented 3D point scan corrupted with noise, outliers, and undersampling. It produces a clean, uniform, and feature-preserving set of oriented points that well approximates the underlying surface. An advantage of the resampling approach is that the density of the point set can be adjusted; for example, for point set rendering, a dense point set is obtained. To obtain noise-free normals especially near the possibly undersampled edge singularities, our resampling algorithm is separated into two phases, as shown in Figure 3. The first phase resamples a set of points away from edges, for which reliable normals can be computed. In the second phase, we upsample to increase point density, progressively filling regions near the edges. Point insertion approaching the edges relies on the reliable normals

4 9:4 H. Huang et al. (a) noisy input (b) resampling away from edges (c) edge-aware upsampling (d) upsampling for rendering Fig. 3. Overview of EAR scheme. Given a noisy point scan (a) with 163K points, we first resample away from edges, leaving gaps near sharp features (b). Based on reliable normals associated with the point set thus obtained, we upsample while approaching the edges and filling the gaps (c). Point density can be further increased through upsampling to obtain a quality point set rendering (d). (Models Courtesy of AIM@SHAPE Shape Repository.) (a) (b) (c) (d) 1 Fig. 4. Upsampling via bilateral projection near a sharp feature: (a) using both positions and normals yields a better point insertion location (solid green dot) than using positions alone (dashed green dot); (b) both projection directions shown fit nearby normals, but the one leading to solid red dot better preserves point uniformity; (c) using bilateral weights properly projects the point onto the latent surface (solid red dot), whereas using unilateral weight pulls the point away (dashed red dot); (d) a series of progressive projection operations upsample the sharp feature. generated in the first phase, leading to clean reconstruction of the sharp features. Resampling away from edges. Starting with a noisy scan, such as the one shown in Figure 3(a), we first perform PCA to estimate normal directions and their consistent orientations as in Huang et al. [2009]. Next we apply bilateral smoothing of these normals where the weighting scheme accounts for both positional and normal information. Iterative bilateral smoothing alone can generally distinguish normals near edges and roughly reveal edge locations. However, the normals may still smear across edges as shown in Figure 2(d). Our strategy is then to not compute normals too close to edges but first resample away, where normals can be estimated reliably. The resampling is accomplished by a locally anisotropic projection which accounts for the current normals. The result of this phase is a set of oriented points with reliable normals, but leaving gaps close to sharp features, as shown in Figure 3(b). Edge-preserving upsampling. Based on the reliable normals, oriented points are inserted and projected onto the latent surface, which is the unknown underlying surface defined by the input point set. For each inserted point, we first select its base location at a midpoint between two existing points; see Figure 4(a). Then, the critical question is along which direction to project the point, the green dot in Figure 4(b), to better preserve the sharp feature. To simplify the problem, we constrain the normal of the inserted point to be the 3 2 same as the projection direction. Hence, a careful determination of the direction serves two important objectives: (i) it fits the normal distribution of nearby points, and (ii) it helps maintain local point uniformity. Once the projection direction is determined, we compute an optimal projection distance so that the inserted point can be moved onto the latent surface; see Figure 4(c). The steps for determining the base, direction, and distance of the projection offer an integrated solution for inserting an oriented point at a sparse spot anywhere on the surface. To properly handle sharp features, both positional and normal information are accounted for in all steps, making the projection operator bilateral and edge aware. As shown in Figures 4(d) and 3(c), repeating the preceding upsampling process incrementally fills the gaps along edge singularities and reconstructs the sharp features cleanly. The upsampling process can continue to increase point densities to facilitate rendering of the point set surface, as shown in Figure 3(d). 4. RESAMPLING AWAY FROM EDGES The input to our algorithm is an unorganized set of points Q = {q j } j J R 3, typically unevenly distributed and containing noise and outliers. The output of the resampling step described in this section is an oriented point set S ={s i } i I ={(p i, n i )} i I R 6, consisting of cleaned point locations p i that better represent the underlying smooth surface away from edges, as well as their associated reliable and edge-aware normals n i. The process of estimating normals near sharp features is particularly delicate and challenging. Traditional normal estimation methods, such as Hoppe et al. [1992], usually work accurately when the underlying surface is smooth, but tend to smear information across singularities such as corners or intersecting planes. Taking it as an initial input, our approach amounts to an iteration between: (i) separating and smoothing normals over the obtained point set, and (ii) resampling the points away from edges while holding the normals fixed. Step (i) reveals the location of edges via normal separation around edge regions, which enables an effective anisotropic projection operator in step (ii). The latter resampling away from edges, in turn, can emphasize the edge locations (by sparsity) so that the process in step (i) in the next iteration operates more accurately. Separating and smoothing normals. The popular PCA-based method for computing surface normal approximations from point cloud data [Hoppe et al. 1992] uses local point neighborhoods. Certain errors may thus be retained when the input is highly noisy, as is typically the case near sharp features; see Figure 2(b). In order to preserve sharp features, we therefore estimate the normal based on an anisotropic neighborhood, as in the framework of bilateral

5 Edge-Aware Point Set Resampling 9:5 normal smoothing [Jones et al. 2004; Öztireli et al. 2009]. In particular, for a given point s i = (p i, n i ), we measure the difference between its assigned normal n i and other normals in its neighborhood as f (p i, n i ) = n i n i 2 θ( p i p i )ψ(n i, n i ), (1) s iw N si where is the l 2 -norm, N si ={s i s i S p i p i <σ p } under a given neighborhood size σ p. The spatial and normal weight functions are defined by θ(r) = e r2 /σ 2 p, ψ(n i, n i ) = e ( 1 n ) 2 i n i 1 cos(σn) where the angle parameter σ n scales the similarity of neighboring normals and we set σ n = 15 by default. Our goal is to minimize i I f (p i, n i ), that is, the normal differences between all points on the surface and their neighbors. This can be achieved through iteratively updating n i for all i with s n i i N si θ( p i p i )ψ(n i, n i )n i s i N si θ( p i p i )ψ(n i, n i ). (2) The preceding formula works well to distinguish normals across discontinuities, classifying their directions into two disjoint clusters near each sharp edge due to the high variance of PCA normals in the vicinity; see Figure 2(c). However, we can also see that direction clustering does not really work for the point locations themselves, and several points are incorrectly assigned a wrong normal direction. Noise and outliers in the input make it impossible to directly obtain a clear classification with respect to an edge in both spatial and directional senses. Indeed, given such an oriented point set, RIMLS may fail to reconstruct a surface that preserves sharp features; see Figure 2(d) and 2(e). Thus, with the normals {n i } i I obtained by (2), we now turn to adjusting the locations {p i } i I to complete a split-step iteration for the set S. Resampling. Several efficient resampling operators have been designed to consolidate a noisy point set [Alexa et al. 2003; Lipman et al. 2007b] while being oblivious to normals. These work well when the underlying surface is smooth. For a resampling operator to be edge aware, however, it must account for normals, even if they may not be entirely accurate. We seek such an operator that would be easily implemented, robust to heavy noise, and utilize the estimated normals around edges. To this end, we alter the LOP operator [Lipman et al. 2007b; Huang et al. 2009] and make it normal or edge aware, allowing for resampling away from edges. LOP takes as input a noisy point cloud, possibly with outliers, and outputs a new point set which more faithfully adheres to the underlying shape. The strength of LOP is that it operates well on raw data, without relying on a local parameterization of the points or on their local orientation. In intuitive terms LOP distributes a set of points, within an optimization framework, to approximate their l 1 -median so as to achieve robustness to outliers and noise. At the same time, a repulsion force is integrated into the optimization formulation to obtain an even point distribution. Although LOP works robustly for geometry reconstruction from raw data, it is still an isotropic operator since the spatial weight function θ(r) used there does not consider sharp geometric features; see Figure 5(b). However, in our context, following the bilateral smoothing step (2), the processed normal directions indicate where the edges approximately are. Thus we define φ(n i,p i q j ) = e (n i (p i q j )) 2 /σ 2 p,, (a) 1% noise (b) LOP (c) σ p = h (d) upsampling (e) 2% noise (f) σ p = h (g) σ p =3h (h) upsampling Fig. 5. EAR under different noise levels and the effect of neighborhood size σ p. Given a noisy input (a) with 18K points, LOP effectively removes noise (b) but the points are smeared across the edges. In contrast, our resampling (c) yields reliable data including both point locations and normals away from edges, which facilitates the subsequent upsampling (d) towards the edges. When a higher level of noise is present in the input (e), the resampling should employ a larger local neighborhood size σ p to ensure a clearer patch separation about the edges and better smoothness away from edges, for example, contrasting (g) with (f), leading to higher-quality upsampling (h). and adjust the locations of the point p i by replacing θ( p i q j ) with φ(n i,p i q j ) in the first term of the LOP expressions of Huang et al. [2009]. Specifically, given a set of points Q as in the beginning of this section, our anisotropic LOP algorithm defines a set of projected points P = {p i } i I R 3 by a fixed point iteration P k+1 = G(P k ),k= 0, 1,...,where G(P k ) = argmin P ={p i } + i I p i q j φ(n i,p k i q j ) (3) i I j J η( p i p k i )θ( pk i p k i ). λ i i I\{i} The repulsion function is η(r) = r. We choose the initial set P 0 as a random subset of Q and its initial normal estimates are computed using PCA and smoothed using (2). The balancing terms {λ i } i I vary at each point but depend on just one parameter, 0 μ<.5, controlling the repulsion force, as explained in Lipman et al. [2007b] and [Huang et al. 2009]. In general, larger values of μ impose higher penalties on points that get close to other points. We set μ = 0.45 by default. Besides the repulsion parameter μ, another important parameter σ p used in both spatial weighting functions φ and θ is tunable based on a rough density measure h = d bb / m, where d bb is the diagonal length of the bounding box and m the size of the input set. The default setting is σ p = h. The resulting operator is therefore edge aware and has the visible effect of gently pushing points away from edges; see Figure 5(c) for example. This is because the normal-dependent weight function φ down-weights large variation in geometric similarity, defined as the height difference of point q j over the tangent plane of the point p i. If the noise level is high, we should increase the supporting neighborhood size σ p to ensure the necessary pushing-away strength for a clearer point patch separation about the edges and piecewise smoothness away from the edges, for example, compare

6 9:6 H. Huang et al. Surface s k s j s k b k v s i v s i (a) (b) (a) input (750 points) (b) LOP (375 points) (c) LOP (1,500 points) Fig. 6. Base location selection: (a) Inserting a point s k at the center of the V-cell v then projecting it onto the surface may push it out of the V-cell; (b) selecting midpoint under orthogonal distance picks b k instead of v. After inserting s k and projecting it along its normal, the clearance at b k becomes zero, preventing other points being inserted at b k. the resampling results in Figure 5(f) and 5(g). The latter leads to higher-quality upsampling and hence surface reconstruction results. The bilateral normal adjusting formula (2) and the anisotropic LOP (3) can be applied alternately several times (our default value is 3), benefiting each other. Referring to the examples in Figure 5, starting from our resampling with reliable normals, we are able to progressively upsample the point set towards the edges; see Figure 5(d) and 5(h). Figure 2 further shows that such resampling enables the RIMLS procedure to retain sharp features; see Figure 2(h), in marked improvement over Figure 2(e). 5. EDGE-PRESERVING UPSAMPLING Given an oriented point set S with visually bald patches along edges as, for example, in Figure 5(c), we now describe how to carefully resample points, approaching edges with reliable data. This is done through a sequence of insertion operations. For each insertion we add a new oriented point (p k, n k ) that fulfills three objectives: (i) p k lies on the underlying surface; (ii) n k is perpendicular to the underlying surface at p k ; and (iii) points are evenly distributed in the local neighborhood after insertion. Finding p k and n k directly under the aforesaid objectives requires searching within the 5D solution space, which can be difficult, especially in the vicinity of sharp features. To make the problem tractable, we design a novel projector, where the projection direction is constrained to be along the normal of the inserted point, that is, p k = b k +d k n k. The computation of the oriented point (p k, n k )is then realized in three steps: finding a near-sparsest insertion base b k (objective (iii)), optimizing the projection distance d k to move the point onto the latent surface (objective (i)), and computing the normal direction n k so that it fits the neighborhood normal distribution and best preserves sharp features (objective (ii)). Base selection. The goal of choosing a good base b k is to ensure fast convergence to an even point distribution within the local neighborhood, where the ensuing projection must be taken into account. Here we first discuss how to choose a base location in the neighborhood of an existing point s i. Discussion on how to pick s i is deferred to the end of this subsection. When finding the base location for point insertion the classical approach, for example, Alexa et al. [2003], uses a local Voronoi construction and picks the center of the largest empty Euclidean ball. However, the Euclidean distance does not take into account the neighboring normals which dictate the ensuing projection. As shown in Figure 6(a), in the vicinity of sharp features the projection may push the inserted point outside its clearing space, keeping the ball empty. This would attract subsequent points being inserted (d) small σ p (1,500 points) (e) large σ p (1500 points) Fig. 7. Resampling on a planar point set, which extends the example in Figure 6 of Lipman et al. [2007b]. While LOP excels at downsampling (b), using it for upsampling (c) leads to uneven point distributions. Our approach inserts at midpoints of (b) with the largest clearances, yielding an evenly distributed point set (d). The inner boundary of (d) is well preserved under a small neighborhood size. Using a large neighborhood respects the outer boundary only and fills the interior with evenly distributed points (e). into the same base, namely the center of the ball, thus stalling the upsampling. Therefore, we incorporate projection direction information into the distance consideration when searching for the largest clearance ball. Specifically, considering an existing point s i and its neighbors in the set N si, we wish to insert at a location b in s i s neighborhood that maximizes C(b) = min si N si D(b, s i ). Rather than using the Euclidean distance for the distance function D(b, s i ), we define it as the orthogonal distance between b and the normal extension at s i, D(b, s i ) = b p i n i (b p i )n i. Since the distance calculation considers the normal directions of inserted points, which are also their projection directions, the stalling problem is solved. Once a new oriented point s k is inserted at b,wehavec(b) = D(b, s k ) = 0. The chance of further insertion near b is significantly reduced since even after projection, the normal extension at point s k passes through b; see Figure 6(b). Computing the optimal position b in the neighborhood of s i based on D(, ) requires solving a constrained quadratic equation. To quickly find an approximate solution, here we limit our search to the midpoints between s i and its neighbors in N si and select the midpoint with the largest clearance as the base b k. Figure 7 shows that while selecting among midpoints is only an approximation, it is able to resample points with even distribution. During the base selection we need to decide globally into which neighborhood the next point should be inserted. Depending on the application we may either want to insert the point where the density is lowest (for uniform resampling) or place it along sharp boundaries (for sharp feature enhancement). To achieve both objectives in a unified approach, we define the priority for the neighborhood of a given point s i as P (s i ) = max s i N si (2 n i n i ) ρ C ( pi + p i 2 ),

7 Edge-Aware Point Set Resampling 9:7 Surface s 6 s 5 s 4 n k p k b k b p (b) i d k s 1 s 2 s s 3 1 s 4 s 5 s 6 s 2 (a) input (850 points) (b) ρ =1 (c) ρ =2 (a) s 3 (c) s 4 s 5 s 6 s 1,s 2,s 3 1 T i (d) ρ =3 (e) ρ =4 (f) ρ= 5 Fig. 10. Projection distance calculation. For each point s i in (a), the Euclidean distance weight and range space weight are obtained using the corresponding weight functions shown in (b) and (c), respectively. The final projected location p k is a weighted average of the projections of these points on the straight line defined by b k and n k. Fig. 8. Upsampling results by EAR for different edge-sensitivity parameter values ρ. The input (a) contains 850 points and was upsampled to 1,500 points in all the cases shown in (b f). Larger values of ρ give higher priority to inserting points along the sharp edge. si nk nk sj sj si bk si bk bk nk sj f k f k f k 0 dk n 0 n 0 dk dk n n k nk nk 0 n 0 n 0 n (a) (b) (c) (a) input (b) RIMLS (c) EAR (d) RIMLS Fig. 9. EAR facilitates surface reconstruction. Given a sparse set (a) with only 2,500 points, RIMLS reconstruction (b) does not produce a quality piecewise smooth surface. With upsampling by EAR to 7,500 points (c), the same RIMLS scheme results in a reconstruction of the Fandisk with better quality and sharp feature preservation. where ρ is an edge-sensitivity parameter. When ρ = 0, the neighborhood priority for s i is determined solely by the largest midpoint clearance. Hence, inserting a new point in the neighborhood with the highest priority actually places its base at the midpoint that has the largest clearance over the whole surface. When ρ>0, higher priority is given to insert points along sharp edges where the normals vary; see Figure 8. In our context, as we first resample away from potential edges to obtain reliable normals, at this resampling step for edge regeneration, we use the default value ρ = 5togive higher priority near edges, achieving the effect of sharp feature enhancement, as shown, for example, in Figures 3(c) and 9(c). It is worth noting that the ability to adaptively sample the latent surface is an important feature of our method. Careful inspection of Figure 8 reveals that, with a high ρ value, our approach places all newly inserted points along sharp edges and corners. Hence, if the goal is to facilitate sharp feature reconstruction rather than direct point set rendering as in Figure 3(d), only a small number of point samples need to be inserted; see Figures 9 and 13 for examples. This allows our approach to handle very large models. Projection distance under a given normal. With the base location b k selected, we now discuss how to project it onto the latent surface along a given direction n, through determining the step size d k. This is achieved by minimizing a weighted total projection distance between p = b k + d k n and existing points in the neighborhood. To handle points near sharp features, the weights of neighboring points are determined based on distances in both Euclidean and directional Fig. 11. A 2D example for normal determination. The first row shows locally optimal normal directions (red arrow) under different situations. The next two rows plot the directional difference and the projection distance, respectively. Note that (b) and (c), while appearing similar, have different desired normal directions for preserving even distribution after projection. Table I. Timing for EAR on Four Raw Scans IP-N FP-N RA-T OP-N RC-T Figure 1 222,543 22K 49 sec 800K 42 sec Figure ,994 16K 31 sec 700K 36 sec Figure 16 99,416 10K 17 sec 600K 29 sec Figure ,365 30K 58 sec 1.8M 73 sec IP-N: number of input points; FP-N: number of noise-free points; RA-T: time for resampling away from edges; OP-N: number of output points; RC-T: time for upsampling close to edges. All examples were run on an Intel(R) Core(TM) i7 CPU 860@2.80GHz with 2GB RAM. spaces, giving the objective function (n (p p i )) 2 θ( p p i )ψ(n, n i ). s i N bk By fixing the spatial weight θ at b k, we thus obtain the step size s i N bk (n (b k p i ))θ( b k p i )ψ(n, n i ) d k (b k, n) =. (4) s i N bk θ( b k p i )ψ(n, n i ) As shown in Figure 10, the previous calculation is equivalent to projecting all existing points onto the straight line defined by b k and n = n k and then computing the weighted average location based on weights determined using both Euclidean and directional terms. In particular, the directional term ensures that points with normals more similar to the input n have higher weights. When the input normal n is accurate, this property ensures that the inserted point

8 9:8 H. Huang et al. Input 1: 120 angle at edge; 1.0% noise. Input 2: 150 angle at edge; 1.0% noise. Input 3: 150 angle at edge; 1.5% noise. (a) a profile showing normals computed by classical PCA are still noisy (red boxes show zoom-ins near the edge) (b) PCA normals after bilateral normal smoothing do not reveal the edges (c) normals after applying RIMLS to (b) showing quality degradation as noise and edge angle increase (d) normals after applying l 1 -minimization to (a) showing quality degradation as noise and edge angle increase (e) normals after applying EAR to (a) showing better handling of noise and revelation of soft edges Fig. 12. A comparison between EAR (e) and several well-known normal estimation and processing schemes including PCA (a), bilateral normal smoothing (b), RIMLS (c), and l 1 -minimization (d) [Avron et al. 2010] (results Courtesy of Haim Avron). Three synthetic input point sets (shown in the first row) were tested at two noise levels. For each input, the underlying shape is characterized by a soft edge (120 or 150 angle) with 1K points. The results shown via 2D cross-sections demonstrate that our resampling approach is not only robust to noise but also capable of handling soft edges. can be projected onto the latent surface even in the vicinity of sharp features. Finding a good normal direction is therefore critical. Normal determination. To finally determine p k = b k + d k n k,we now compute the normal vector n k. As shown in Figure 11, here we have two selection criteria: (i) n k fits the normal distribution in the local neighborhood of b k,thatis,f(b k, n k ) of (1) is small; and (ii) the moving distance d k (b k, n k ) defined by (4) is also kept small so that the even distribution we had sought can be better preserved. To compute n k efficiently, we limit our search to the two neighborhoods surrounding n i and n j, respectively, where n i and n j are the normals of the two endpoints used for generating b k ; see Figure 11. We first decide which neighborhood to use based on moving distance d k,thatis, l = argmin d k (b k, n l ), l {i,j} and then compute the normal n k by minimizing f (b k, n) while holding the directional weight ψ(n, ) at the fixed normal direction n = n l. This allows us to compute n k using (2) with a single iteration. Note that when both endpoints s i and s j are on the same smooth surface, for example in Figure 11(a), the two neighborhoods surrounding n i and n j overlap. Therefore, the final solution n k

9 Edge-Aware Point Set Resampling 9:9 (a) input (515 points) (b) APSS (c) RIMLS (d) MLS upsampling (4,000 points) (e) APSS over MLS (f) RIMLS over MLS (g) EAR upsampling (4,000 points) (h) APSS over EAR (i) RIMLS over EAR Fig. 13. Power of edge-preserving upsampling in our EAR scheme. For a clean and oriented point set with a rather low density (a), APSS (b) and RIMLS (c) cannot provide a good surface definition without upsampling. MLS upsampling (d) improves the performances of APSS (e) and RIMLS (f), but smears sharp features. With EAR (g), APSS (h) and RIMLS (i) successfully preserve the sharp features. (Models Courtesy of AIM@SHAPE Shape Repository.) (a) raw scan (c) Cocone over MLS (b) direct Cocone (d) Cocone over EAR Fig. 14. Result comparison using the edge-oblivious Cocone surface reconstruction [Dey and Giesen 2001]. (Models Courtesy of Andrei Sharf.) may be close to the average of n i and n j, providing a smooth interpolation of existing normals. 6. RESULTS AND DISCUSSION The presented EAR algorithm was tested on a variety of raw and synthetic point scans. Processing times on raw scans are provided in Table I. Unless specifically indicated in the captions, the default parameter values as given in Sections 4 and 5 were applied for obtaining the presented results. Next we first elaborate on some typical obtained results and then discuss limitations. Results. We first show EAR at work on a few synthetic models containing both sharp and soft features separated by smooth surface patches. In Figure 3, we use the Fandisk model to demonstrate (a) input (b) σ n =60 (c) σ n =30 (d) σ n =15 Fig. 15. Results of applying EAR with different parameters before Poisson surface reconstruction on a dense noisy Ramesses model (800K points). As expected, small values of σ n tend to better enhance sharp features, but may over-sharpen highly curved areas. (Models Courtesy of AIM@SHAPE Shape Repository.) the capability of EAR to handle noisy input data; the input point cloud was corrupted with both noise (2% of the bounding box) and outliers (10% of the bounding box). The results show that EAR not only smooths out noise in point positions, but also effectively handles smooth variations and sharp changes in normals. Figure 12 compares our resampling scheme with several well-known normal estimation and smoothing schemes including PCA, bilateral normal smoothing, RIMLS, and l 1 -minimization filtering on a set of shallow fin-shapes with different edge angles and noise levels. These results demonstrate superior performance of our method in terms of robustness to noise and effective recovery of soft edge features. Figure 13 demonstrates the power of EAR, compared to MLS, in resampling and more specifically, edge-preserving upsampling, a highly sparse point set surface containing sharp features. With a small amount of new insertions, the performance of reconstruction methods such as APSS and RIMLS can be noticeably improved. Figures 1 and all demonstrate how our EAR scheme can enhance the performance of existing surface reconstruction methods. Four representative techniques are chosen: RIMLS, APSS, Poisson, and Cocone. The implementations of the first three are from MeshLab and the last was provided by its authors. We also implemented the well-known MLS projector which uses Voronoibased upsampling [Alexa et al. 2003], as the classical resampling operator to compare our EAR scheme with. For all the existing algorithms, we show the best results we were able to generate, following the guidelines provided by the distributed codes and published parameters. The inputs in Figures 1, 14 and 16 are raw scans of real objects. The typical imperfections associated with digital scans, such as noise, outliers, nonuniform point distribution, and missing data, are ubiquitous in these datasets. EAR removes noise and outliers, fills in the missing parts, preserves sharp features and, at the same time, facilitates various surface reconstruction methods. In particular, we can observe the benefits of applying resampling prior to surface reconstruction and EAR is evidently outperforming MLS with respect to preservation of sharp features. The proposed two steps, resampling away from edges and edge-preserving upsampling, are both indispensable for a satisfactory reconstruction, as shown in Figure 16 for a wide variety of parameter values. Figure 15 presents results of applying EAR with different σ n before Poisson surface reconstruction on a large noisy dataset, which is

10 9:10 H. Huang et al. (a) raw scan (b) poisson over PCA normals (c) poisson over MLS (d) poisson over EAR (e) RIMLS over PCA normals (f) RIMLS over S with σ p = h (g) RIMLS over S with σ p =10h (h) RIMLS over EAR with σ p = h Fig. 16. Result comparison on a raw scan (a) using the edge-oblivious Poisson [Kazhdan et al. 2006] and edge-aware RIMLS surface reconstructions. (b) Poisson over oriented PCA normals. (c) Poisson over a filtered and upsampled point set using MLS. (d) Poisson over output of EAR. (e) RIMLS over the same oriented PCA normals used in (b). (f) RIMLS over an oriented point set S after resampling away from edges with σ p = h. (g) RIMLS over the same point set S with a much larger σ p. (h) RIMLS over the output of the complete EAR with the same σ p as in (f). The ability of EAR to lead to piecewise smooth and feature preserving reconstructions in both scenarios is evident. (Models Courtesy of Andrei Sharf.) (a) raw scan (b) MLS (c) radiance scaling over (b) (d) EAR (e) radiance scaling over (d) Fig. 17. EAR for surfel point set rendering, where each output surfel is displayed using a single pixel and colored by its normal direction. The input scan (a) of a shutter blind is noisy and unevenly distributed. MLS resampling (b) smears the edges whereas EAR (d) preserves them well. Comparing (c) and (e), dominating edges are enhanced on EAR point set surface using radiance scaling [Vergne et al. 2010]. more challenging and contains geometric details of different sizes. By decreasing the parameter value σ n, we are capable of better enhancing sharp features. However, if σ n is set too small, some highly curved areas may be over-sharpened. We also show the results of converting noisy and unoriented point sets into clean point set surfaces with reliable, edge-aware normals for direct surfel-based point set rendering. Both synthetic (Figure 3) and raw (Figure 17) point scans were employed, demonstrating the

11 Edge-Aware Point Set Resampling 9:11 (a) Raw scan (82K points). (b) EAR results. Fig. 18. With accurate normals estimated near regions of close-by surface sheets (a), our resampling algorithm performs well; see the middle zoomin in (b). However, the upsampling step is limited by the choice of the σ p parameter and may not fill sufficiently large holes present in the point sampling; see zoom-in s with black borders in (b). from surface singularities is followed by a novel bilateral projector which progressively upsamples toward these singularities. Both phases are edge aware and hence the resampling results preserve sharp features. Specifically, the presented resampling algorithm enjoys the following properties: (i) sharp edges are preserved and smooth surfaces are maintained under a unified approach; (ii) the resampling can reach any density requirement specified by users; (iii) the reliable normals generated greatly facilitate existing surface reconstruction methods. In the future, besides research on open boundary handling, we aim to develop a point set surface modeling system, where users can perform local editing such as smoothing, edge enhancing, and hole-filling directly using resampling tools presented in this article. We would also like to design a real-time GPU-based resampling algorithm, providing users with unlimited zoom capabilities when viewing point set surfaces, without the need for mesh generation. Finally, it is desirable to include an adaptive parameter to locally control the sharpness of our resampling results, in particular when sharp regions are severely corrupted and undersampled. ACKNOWLEDGMENTS The authors would like to thank all the reviewers for their valuable comments. Fig. 19. One limitation of our resampling scheme is that it is not designed to handle open point sets; it may produce rough boundaries as a result. capability of EAR for handling smooth surfaces as well as sharp features under a unified framework. Limitations. When the noise level in the input point cloud is high or the captured object contains close-by surface sheets, the initial normal orientation using oriented PCA may be erroneous, subsequently causing errors in the resampling. This problem can be alleviated by a more capable normal estimation technique, for example, Huang et al. [2009], which can handle the close-by surface sheet problem; this is illustrated in Figure 18. In the same figure, we also see that relatively large holes in the input point sampling are not filled by our upsampling step in EAR since this step is limited by the choice of σ p, which measures the size of the neighborhood. A specifically chosen large σ p wouldleadtobettergapfilling. Also, we currently use a fixed neighborhood radius σ p and angle parameter σ n. Although the default parameter setting has worked well for most of our test cases, certain special situations may require a careful tuning of these two parameters to deal with elusive sharp features. Another limitation of our EAR scheme is that it is not designed to handle open boundaries and may produce less than satisfactory results in that situation, as shown in Figure 19. Finally, under severe noise or undersampling, our scheme may oversmooth geometric details and sharp features or oversharpen an edge depending on the edge sharpness, as can be observed in Figure 16 by comparing (e) with (h) and in Figure 14(d), for example, see edges in the black boxes. Currently, our scheme does not have an adaptive parameter to locally control the sharpness of the resampling results. 7. CONCLUSIONS We have described a novel resampling algorithm for converting noisy scan data into a clean point set surface endowed with reliable normals. The core of our algorithm consists of two phases: a robust edge-aware method which computes reliable normals away REFERENCES ADAMSON, A. AND ALEXA, M Point-sampled cell complexes. ACM Trans. on Graph. 25, 3, ALEXA, M.,BEHR, J.,COHEN-OR, D., FLEISHMAN, S., LEVIN, D., AND SILVA, C. T Point set surfaces. IEEE Trans. Vis. Comp. Graph ALEXA, M.,BEHR, J.,COHEN-OR, D., FLEISHMAN, S., LEVIN, D., AND SILVA, C. T Computing and rendering point set surfaces. IEEE Trans. Vis. Comp. Graph. 9, 1, ALLIEZ,P.,COHEN-STEINER, D., TONG,Y.,AND DESBRUN, M Voronoibased variational reconstruction of unoriented point sets. In Proceedings of the Eurographics Symposium on Geometry Processing AVRON, H., SHARF, A., GREIF, C., AND COHEN-OR, D l 1 -sparse reconstruction of sharp point set surfaces. ACM Trans. Graph. 20, 5,135. CARR,J.C.,BEATSON, R. K., CHERRIE,J.B.,MITCHELL,T.J.,FRIGHT,W.R., MCCALLUM,B.C., AND EVANS, T. R Reconstruction and representation of 3D objects with radial basis functions. ACM Trans. Graph DEY, T. AND SUN, J Normal and feature approximations from noisy point clouds. In Proceedings of the Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS). Lecture Notes in Computer Science, vol. 4337, DEY, T. K. AND GIESEN, J Detecting undersampling in surface reconstruction. In Proceedings of the Symposium on Computational Geometry FLEISHMAN, S., COHEN-OR, D., AND SILVA, C. T Robust moving leastsquares fitting with sharp features. ACM Trans. Graph. 24, 3, FLEISHMAN, S., DRORI, I., AND COHEN-OR, D Bilateral mesh denoising. ACM Trans. Graph. 22, 3, GROSS, M. AND PFISTER,H.2007.Point-Based Graphics. Morgan Kaufman. GUENNEBAUD, G., BARTHE, L., AND PAULIN, M Real-Time point cloud refinement. In Proceedings of Symposium on Point-Based Graphics GUENNEBAUD, G., GERMANN, M., AND GROSS, M Dynamic sampling and rendering of algebraic point set surfaces. Comput. Graph. Forum (Special Issue of Eurographics). 27, 3,