Sketch-Based Aesthetic Product Form Exploration from Existing Images Using Piecewise Clothoid Curves

Transcription

1 Sketch-Based Aesthetic Prodct Form Exploration from Existing Images Using Piecewise Clothoid Crves Günay Orbay, Mehmet Ersin Yümer, Levent Brak Kara* Mechanical Engineering Department Carnegie Mellon University Pittsbrgh, PA Abstract We present a new sketch-based prodct form exploration techniqe that works from images and sketches of existing prodcts. At the heart of or approach, is a mlti-stroke crve beatification method and a crve-based image deformation algorithm. The proposed approach converts grops of strokes into piecewise clothoid crves in order to prodce visally pleasing shapes. The deformation diffsion algorithm then spatially distribtes the ser specified deformations throgh ot the image to prodce smooth transformations from the original image to the reslting image. We demonstrate the techniqe on a variety of images inclding photo-realistic images, real prodct images, and sketches. Keywords: sketching; prodct design; aesthetics; crve design; image editing 1. Introdction Image editing tools are becoming increasingly more biqitos with the wide availability of a variety of photo editing software [1]. However, sch tools remain difficlt to master de to the complex natre of their ser interaction methods and the parameters that need to be controlled to achieve the desired effects. While many projects no dobt reqire advanced skills, experience, and appropriate task planning, even relatively simple modifications become difficlt to implement with sch tools. As a reslt, the maniplation and exploration of shape ideas sing existing visal references remain a significant challenge in prodct design, forcing designers to generate a new sketch or a visal for each of their new ideas. In this paper, we propose a new sketch-based method for the rapid exploration of new prodct forms from existing images and sketches. The proposed method enables the sketch inpt to control image deformations in which the visal attribtes contained in the original image or sketch are preserved. By allowing image modifications to be made throgh simple sketched strokes, it redces the effort reqired in maniplating high fidelity images and sketches. Moreover, it advances existing image editing methods which se fixed deformation masks, pick/drag type crve-based deformations, or area-based deformation lattices. At the heart of or approach is an aesthetic crve constrction method that works from mltiple strokes, followed by a deformation field calclated by a diffsion algorithm. To achieve the desired visal properties, we se the previosly identified forms of aesthetic crves, specifically piecewise clothoid crves. Figre 1 shows the sage of or system. Shape variations can be intitively and flidly explored on different kinds of images inclding photo-realistic images, real prodct images, and sketches. The ser starts from an image or a sketch of an existing prodct in order to explore new designs. The ser draws a set of strokes via a tablet interface to define a nmber of anchor crves, and sorce and target crve pairs which act as deformation handles for the editing. Each grop of strokes that define an individal crve, is converted into a single crve with the help of a point ordering algorithm. The sorce and anchor crves are aligned with the nderlying featres sing active contors, while the target crves are converted into piecewise clothoid crves. The inpt image is then deformed sing a smooth deformation field that is calclated from the sorce and target crve pairs. As the target crves are designed to be visally pleasing, the deformed featres in the final image also carry similar properties. 2. Contribtions The proposed work makes specific contribtions in tilizing sketch-based interaction in rapid prodct form exploration, and in technical methods that enable this exploration. The first contribtion of the proposed method is that it provides a means to tilize intitive sketching inpt in exploring visally pleasing design forms from images and sketches of existing prodcts. This, in retrn, allows the ser to still benefit from the intitive and natral way of externalizing shape ideas sing the sketch inpt. Moreover, the ser focses on the reslting prodct form directly, instead of obtrsively paying attention to the visal qality and accracy of each drawn stroke. The second contribtion is that the proposed method enables the conversion of overlapping strokes to be seamlessly converted into piecewise clothoid crves. This allows the ser Preprint sbmitted to Jornal of Visal Langages and Compting May 2, 2012

2 R User Inpt Initial Design Sorce and Target Strokes Final Design Deformation Gradient fx Raw Inpt Strokes Raw Inpt Strokes Ordered Points Crvatre and Chord Estimation Crvatre Segmentation fy Preprocessed Crves Sorce Crve Alignment Target Crve Piecewise Clothoid Fitting Figre 1: The overview of the proposed method. The ser inpts an image of an existing prodct in order to explore new designs. The ser then inpts a set of strokes via a tablet interface to define a nmber of sorce crves and corresponding target crves which act as image controllers. These inpt strokes are preprocessed into desirable comptational forms throgh a series of operations. The inpt image is then deformed into the final design sing a smooth deformation field that is calclated form the sorce and target crve pairs. to work on the design sing the natral sketch inpt withot ptting emphasis on the accracy of the strokes. The last contribtion is that the proposed method prodces images that contain visally pleasing featres and shapes as it grabs the initial featres in the image and deforms them into piecewise clothoid crves with a smooth deformation field. specify, deform and morph featres. They similarly tilize active contors to accrately grab featres and a mltilevel freeform deformation algorithm to deform those featres. Likewise, Fang et al. [10] developed a crve-based image deformation tool mainly focsed on textre regeneration on distorted regions. Shngang et al. [11] sed cbic splines and Moving Least Sqares for image deformation sing crves as handles. In this work, however, we attempt to extend crve-based image deformation methods by combining with sketch-based crve design in order to tilize in rapid prodct form design. 3. Related Work 3.1. Image Editing The main focs in image editing have been in ct-and-paste type operations and image enhancing. Elder et al. [2] presented a contor based image editing techniqe where modifications are applied directly in the image contor domain. These modifications are then transferred to the final image sing edge based image reconstrction [3]. This approach enables robst ctand-paste type operations. Sand et al. [4] have introdced a perceptally-spported editing tool where they combine a variety of previosly stdied image editing methods [5, 6] deployed in an interactive environment. In an earlier work [6], the athors stdied decomposing simple images into geometric primitive based shapes for enabling simple shape based editing. In 2002, Seitz et al. [7] introdced a set of image editing techniqes which enable a seqence of images to be modified throgh operations applied to one of the images in the seqence. Their method makes se of the plenoptic fnction throgh which the modifications are propagated in a physicallyconsistent way. Perez et al. [8] presented a set of image editing operations based on second order discrete Poisson eqations in image domain. Their approach enables seamless region mapping between images and modification of original illmination, color and textre in arbitrary regions of a given image, or method focses on nonlinear warping of a single image throgh ser defined strokes Piecewise Clothoid Fitting Different approaches have been proposed for fitting clothoid crves into a set of organized and norganized points [12, 13, 14, 15, 16]. Pal and Ntborne [12], Schechter [13] a year later, proposed methods that fit piecewise clothoids given two end points with specified positions, tangents and crvatres. Stoer [14] developed a nmerical algorithm to fit single and piecewise clothoids to points. Meek and Walton [15] described the algorithms that match crvatres and tangents of mltiple clothoids at their attached ends. Davis [16] described a rapidly convergent total least-sqares algorithm. For a broader collection of clothoid fitting methods [17] can be referred. Related to or sketch-based piecewise clothoid fitting approach is the work by Harada and Yoshimoto [18]. They applied b-spline fitting methods on point sets and then converted them into piecewise monotonic crvatre crves. McCrae and Singh [19], developed a sketching interface that converts individal strokes into piecewise clothoids for road design. They based their approach on a crvatre analysis followed by a segmentation algorithm. Similarly, Baran et al. [20] introdced a graph based method that identifies an optimm path of local fits that reslts in a piecewise crve composed of lines, arcs and clothoids. Althogh these approaches are sccessfl in processing individal strokes, they do not readily spport crve fits from a grop of overlapping strokes Crve-Based Image Deformation Closely related to or work, Lee et al. [9] proposed an image morphing method that provides an easy-to-se means to 2

3 3.4. Differences from Existing Work Or approach differs from the above works in a nmber of ways. Or main goal is to enable rapid sketch-based exploration of new prodct forms from existing prodct images and sketches. Specifically, we attempt to tilize conventional sketching in design of fair crves, and se these crves to control image deformations that reslt in visally pleasing shapes. As a reslt, or approach attempts to enable rapid shape exploration sing the convenient sketch inpt from largely available images and sketches of existing prodcts. 4. Overview and User Interaction Figre 1 illstrates the details of or system. The inpt to or system is an image and a set of ser-drawn strokes which define deformation handle crves. The sers may draw strokes to define crves in three categories: Anchor Crves: (Orange color) These crves indicate regions of the image that the yser wants to preserve nchanged. Any deformation ceases at these crves. Sorce Crves: (Ble color) These crves demarcate featres in the image that will ndergo ser-specified deformations. Inpt sorce crves are first modified nder local image gradients that precisely align them with the salient image featres. Target Crves: (Green color) For each crve stroke, a target crve indicates the new desired shape. Each target crves is atomatically beatified into a piecewise clothoid crve sing crvatre estimation, segmentation, and crve fitting to the original strokes. Once processed, stroke grops in each category are internally represented as crves of the same category. Given the anchor, sorce, and target crves, a smooth vector deformation field is calclated on the image. The compted deformation field is then applied to the image, reslting in the modified image. The key property of the deformation is that the image attains the precise shape dictated by the target crves, while the remainder of the image ndergoes a locally smooth deformation. The sers repeat this process as many times as necessary, which reslts in the final form of the design. calclated target crves mst be beatified into visally pleasing crves so as to eliminate the artifacts that may arise from a literal interpretation of the raw strokes Mltiple Stroke Analysis In the first step of preprocessing, or approach first analyzes each stroke grop and converts it into a single crve. To be able to tilize existing crve fitting methods, it is necessary to determine an estimated direction and length of the crve. Note that incoming strokes can be drawn in arbitrary directions and order. Hence, a global analysis of the sampled points that comprise the constitent strokes is necessary to identify a natral spatial order for the sampled points, and hence the strokes. Both cases, however, reqire that the sampled points comprising the inpt strokes are pre-processed to identify a natral spatial ordering and a chord length parameterization of the points. As demonstrated later, this ordering facilitates an energy-based featre edge identification as well as crve synthesis from a crvatre profile. Or approach ses a spectral point ordering method and a moving circle analysis to spport crve fitting calclations on both the sorce and the target crves. This analysis prodces an accrate estimate of the chord length parametrization (ths an approximate crve length) as well as a noise free estimate of the crvatre along the crve. a) b) c) d) 5. Preprocessing the Inpt The inpt to or system is a set of ser-drawn strokes that define crves in three categories: (1) sorce crves, (2) target crves, (3) anchor crves. Dring sketching, the sers typically draw grops of overlapping strokes each of which is intended to represent a single crve (Fig. 2.a). From this inpt, or approach seeks to prodce crves that satisfy the following three reqirements. First, each stroke grop mst be converted into an individal crve that closely approximates the strokes in the grop, regardless of its category. Second, the calclated sorce crves mst be accrately aligned with the salient featre edges in the image. This garantees that the featres in the image will properly deform into the shapes of the target crves. Third, the 3 Figre 2: A crve consisting of mltiple strokes are analyzed to identify a natral ordering of the points that make p the original strokes. The identified ordering facilitates fitting of a piecewise clothoid crve that closely approximates the original strokes. The crvatre profile of the final crve is locally linear except at the transitions between the neighboring clothoid segments. We start by collecting the points sampled along each stroke into a point set. We tilize the spectral point ordering method proposed by Orbay and Kara [21] to calclate an initial ordering of the points. Once this ordering is calclated we next estimate the chord length parametrization and the crvatre profile sggested by the ordered points. To do so, we se a moving circle approach which simltaneosly calclates local estimates of the chord length parameter and the crvatre. As shown in Fig. 3, a least-sqares circle is fit to the points located inside a sliding

4 window. Using the vectors formed by the circle center and each of the sampled points, the points are ordered within the window and assigned a chord-length parameter. For each sliding window, we se the weights of a Hamming window as the confidence vales of the chord length, and the crvatre (i.e. 1/R). The weights of the Hamming window is calclated as follows: ( w(s) = cos 2π sn + 1 ) (1) N 1 where s = [0, 1] is the normalized chord length within the window, and N is the nmber of points in the window. The closer the points are to the middle of the sliding window the higher confidence vale they attain. The final estimates are calclated over the aggregate local estimates and their associated aggregate confidence vales. An example chord length and crvatre estimate for the stroke grop shown in Fig. 2.a is shown in Fig. 4. R w(s) s Figre 3: The crvatre and chord length parameters are estimated simltaneosly sing a sliding window over the initially ordered points sing a spectral analysis. A circle is fit within each window and the radis of the circle and the chord length on the circle circmference is compted. The Hamming window fnction sed in the calclation of the point weights Sorce Crves: Alignment on to Featre Edges In this step, we extract and align the sorce crves on to the nderlying featres of the image. Before we begin this process, we first define a polyline sing the chord length estimation that we calclated in the previos step. Before we align the polyline on to the featre, we smooth the crve to eliminate the jaggedness of the initial configration (Fig. 2.b). We se active contors [22] to gradally pll the sorce crves on to the nderlying featres of the image. For this, an edge map is extracted from the image sing Canny Edge Detection [23], and the points comprising the polyline are plled toward the identified edges. The process is initialized by anchoring the end points of the sorce crve to the nearest stable points on the edge map, and sing the reglar active contors formlation to achieve a smooth crve that is aligned with the nderlying featre. approach, we se one of many possible sch crves, namely the piecewise clothoid crves with piecewise linear crvatre profiles. In this step, we beatify the target strokes into piecewise clothoid crves throgh a series of operations. We first begin by identifying a segmentation of the crve aided by the previosly calclated crvatre and chord length estimation. We then se a nmerical fitting algorithm that minimizes the spatial error between the original points and the piecewise clothoid crve by controlling the crvatre vales at the segmentation points Crvatre Segmentation Clothoid crves, as their crvatre profiles are restricted to be linearly changing, can represent only a limited range of shapes. They have at most one inflection point and a constant rate of change of the crvatre with respect to the chord length. As a conseqence, for a proper representation of a crve exhibiting rather complicated crvatre profiles with clothoids, it is of tmost importance to decide where to break the crve and what nmber of segments to se. This decision directly dictates the trade off between the overall smoothness of the crve and the approximation accracy of the points. On one extreme, sing only one clothoid crve is globally smooth yet not able to represent complicated crves. On the other extreme, having a large nmber of clothoid segments reslts in rapidly changing trend in the crvatre profiles ths leading to visally less pleasing reslts. To be able to control this trade off with a ser defined parameter, we adopted the crvatre segmentation method proposed by McCrae and Singh [19]. A matrix of costs of the optimal segmentations between any given two points is calclated in a bottom-p fashion sing the following cost fnction as defined in the original work [19]: M(a, b) = min a<k<b {M(a, k) + M(k, b), E f it(a, b) + E cost } (2) where M(a, b) represents the minimal fit error of the associated configration of line segments between points a and b. The fitting error is simply as the sm of absolte errors as E f it (a, b) = b f it ( i ) i (3) i=a where i, i and f it ( i ) are the estimated chord length, estimated crvatre, and the fit crvatre at point i, respectively. E cost is the cost per line segment in the crvatre space and indirectly controls the total nmber of segments in the minimal fit case. This parameter is of paramont importance since the nmber of segments dictate the qality of the approximation to the inpt strokes. We will have detailed discssions on how this parameter controls the compromise between the fidelity of the approximation and the reslting level of global smoothness Target Crves: Piecewise Clothoid Fitting To achieve the desired visal properties on the deformed images, we make se of previosly identified aesthetic crve forms to beatify the target crves. Recent stdies [24, 25, 26] have demonstrated that crves with piecewise linear logarithmic crvatre histograms (LCH) exhibit visal appeal. In or Piecewise Clothoid Fitting The reslt of the previos crvatres segmentation step is a list of crvatres ( i ) and associated chord length vales ( i ) which represent the breaking points of a linearly piecewise crvatre profile. In this section we describe and derive a nmerical fitting method that works specifically from a previosly cal-

5 (signed crvatre) 0 This transformation incldes not only the necessary amont of rotation for alignment, bt also the scaling and shearing effects. For the alignment, however, we only se the rotation part sing polar decomposition [27] as: [T] = [R][S ] (10) (chord length) Figre 4: The final segmentation in the crvatre space of strokes given in Fig. 2.a. The reslting nmber of clothoid segments is controlled by a single parameter (E cost in Eqn. 2). We then transform the original points sing the decomposed rotation part of the initially calclated transformation: Here, we define the error as: [ x p ȳ p ] = [T][x p y p ] (11) clated segmentation in the crvatre space 1. Mathematically, the crvatre profile can be defined sing piecewise linear interpolation fnctions as: where Φ n ( i ) = N ( i ) = n Φ n ( i ) (4) n=1 i 1 i i 1 if [ i 1, i ] i 1 i+1 i if [ i, i+1 ] 0 elsewhere Using this crvatre fnction, we first write the angle in terms of the chord length as a smmation: θ( i ) = (5) i ( m ) m (6) m=1 where θ(0) = 0, m = m m 1 and 1 = 0. Similarly, we write the spatial coordinates of the associated points on the crve as a smmation: x( i ) = y( i ) = i cos(θ( m )) m (7) m=1 i sin(θ( m )) m (8) m=1 where x( i ) = y( i ) = 0. Given a set of points (x p, y p ) and a piecewise clothoid crve, we define a spatial error fnction to be minimized. For a reliable error definition, we first align the original points and the piecewise clothoid crve. To do so, we translate both the crve and the points sch that their centroids are located at the origin. We then calclate a transformation that aligns the original points on to the associated points on the crve as dictated by the chord length parameter. [x c y c ] = [T][x p y p ] (9) 1 Please see the Appendix for more details on the clothoids. 5 E = N ( x i p xc) i 2 + (ȳ i p y i c) 2 (12) i=1 Minimizing E can be achieved by minimizing a residal vector defined as: { } xp x {R} = c (13) ȳ p y c Since the spatial coordinates of the clothoid crve is a nonlinear fnction of the crvatre, minimizing R reqires an iterative soltion. For this, we se the following Newton-Raphson iteration: [ ] 1 {R} { i+1 } = { i } {R i } (14) {} The Jacobian [ {R} {} where i ] can be derived sing Eqations 7 and 8 as: J x i, j = x i j = J y i, j = y i j = and from Eqation 4 and 6 [ ] {R} J {} = x J y θ( m ) j = i m=1 i m=1 (15) sin (θ( m )) θ( m) j (16) cos (θ( m )) θ( m) j (17) m φ j ( k ) k (18) k=1 We se the crvatre vales calclated at the crvatre segmentation step as the initial vales for the minimization. At each iteration of Eqn. 14, we calclate a different transformation as defined in Eqn. 11. Since the coefficients of the crvatre vales are fnctions of the estimated chord length parameter, the Jacobians can be calclated with matrix operations. The described minimization yields the piecewise clothoid crve that has the same segmentation as calclated in the previos step and it minimizes the spatial error between the original points and the crve. However, as discssed earlier, the amont of the minimized error is directly dictated by the crvatre segmentation. Figre 5 demonstrates the trade off between accracy (i.e. the spatial error) and overall smoothness of the

6 crve. For the set of strokes shown (Fig. 5.a), sing a high nmber of segments can case ndesirable flctations in the crvatre profile (Fig. 5.b), whereas sing very few segments can severely affect the approximation accracy (Fig. 5.f,g). We leave the control over this trade off to the ser and let them manally control the cost parameter that dictates the reslting crvatre segmentation. (e) This step calclates a smooth spatial deformation that transforms each sorce crve into the corresponding target crve, while preserving the anchor crves. For this prpose, we se an isotropic vector diffsion method to spatially diffse the deformation vectors between the sorce and target crve pairs. Althogh there are arbitrarily many vector fields that can prodce the desired deformation, in this work we se a formlation similar to the gradient vector flow field (GVF) method [28]. This approach sitably diffses the initial deformation vectors compted between each sorce-target crve pair, ths prodcing deformations that vary smoothly throgh the image space. The GVF is defined as the continos 2D vector field V(x, y) = [(x, y), v(x, y)] that minimizes the following energy fnctional [28]: ɛ = (µ( 2 x + 2 y + v 2 x + v 2 y) } {{ } Smoothness (19) + f 2 V f 2 )dxdy } {{ } Pointing objects where f (x, y) is a 2D scalar field (e.g. black and white scanned medical images in the original work), is the gradient operator in Cartesian coordinates, (x, y) and v(x, y) are the x and y components of the vector field at point (x, y) while the sbscripts denote partial derivatives. µ is the coefficient which controls the amont of diffsion. In this energy fnctional, the first term forces the local smoothness of the calclated vector field while the second term forces V(x, y) to converge to the original deformation vector in regions of high deformation. The vector field V(x, y) minimizing this energy fnctional can be calclated sing calcls of variation by solving the following two decopled Eler eqations: (f) (g) Figre 5: The crvatre space (i.e. the estimated chord length vs. crvatre) of the initial point set is sed to compare the effect of the nmber of piecewise clothoid segments. This nmber is directly controlled by the cost parameter, E c ost. As the cost parameter increases from to (g), the nmber of segments ths the approximation accracy increases. However, the visal appeal of the fits with higher nmber of segments becomes debatable as the fit approaches to the strokes which are originally overtraced and noisy. (g) On the other hand, the low nmber of segments reslt in weak approximations to the strokes. The compromise between the approximation accracy and the visal appeal is to be decided by the ser. 6. Crve-Based Image Deformation The inpt to the image deformation step is a set of preprocessed sorce and target crves, and the set of anchor crves. 6 µ 2 ( f x )( f 2 x + f 2 y ) = 0 µ 2 v (v f y )( f 2 x + f 2 y ) = 0 where the Laplacian operator is defined as: (20) 2 (x, y) = 2 x y 2 (21) The two discretized eqations can be solved independently to prodce the vector field components and v separately. In this application, the discretization of the domain natrally comes from the pixelated image. In or implementation, we se the deformation vectors drawn between sorce and target crves to instantiate the f x and f y vales in Eqn. 20. An iterative mltigrid solver that enables an efficient comptation of the soltion to the differential eqations yields the reslting vector field. We note, however, that any conventional differential eqation solver can be similarly adopted. The bondary conditions to the above set of eqations arise from the anchor crves (zero deformation), and the deformation vectors from each sorce crve to the corresponding target crves. These vectors are preserved throghot the comptation. The remainder of the vector field is compted sing the above two eqations.

7 80 Prior to initializing the bondary conditions, we resample the target and sorce crves to have the same nmber of points along their paths. The sampling density is chosen sch that the distance between adjacent sample points is always less than the pixel length. This ensres all pixels along the crves to attain a niqe deformation vector. Moreover, a simple direction check is performed to map each end of the sorce crve to the intended end of the target crve, thereby eliminating sensitivity to drawing directions. A problem arises if the compted deformation field is applied directly to the original image, as the deformation vectors do not garantee a complete mapping from the original image to the yet-to-be compted deformed image (Fig. 6.c). To overcome this difficlty, we tilize deformation vectors that originate from the target crves and point toward the sorce crves. The sbseqent diffsion is then applied in this new configration. This allows, for every pixel in the deformed image to point to a realvaled location in the original image (throgh the deformation vector) and attain a vale as an interpolation from the original image. To avoid aliasing effects, we se a bicbic spline interpolation. Figre 6 shows intermediate steps of deformation for two examples. Another important property of the reslting deformation field is that, its gradient is discontinos arond the target crves. Figre 8 shows a 3D plot of the deformation field calclated for the example in Fig. 6 to illstrate this property. This rapidly changing gradient necessitates that the sorce crves to be accrately aligned with the featres on the image to avoid ndesirable smdging arond the featres Figre 8: The deformation field is sharp arond the crves with large gradients on the displacement vales. In order to preserve the featre edges in the inpt image, it is crcial to have the sorce crves to be closely aligned with the nderlying edges. Figre 6: Calclation of a spatial transformation field from the sorce and target crves. The displacement vectors from the sorce crves (ble) to the target crves (red) are diffsed to achieve a smooth deformation field. The calclated spatial deformation field of displacements in the x direction. The final image deformed by the calclated deformation field. Figres 6 and 10 assess the natre of the deformations over circlar and rectanglar patterns. As shown, when circlar patterns are forced into rectanglar shapes, the internal and external regions are smoothly deformed yet they cannot preserve their perfect circlar strctres. Similarly, when rectanglar patterns are forced into circlar shapes, althogh the deformed featres shape into the target crves, the neighboring rectangles neither form circles nor stay as rectangles. To selectively preserve sch patterns, the sers can se anchor crves to limit the effective area of the deformation as shown in Fig Figre 9: The anchor crves serve as limiters on propagation of the deformation field. The ser controls the region that the deformation can affect by trapping it region with anchor crves. Figre 10 frther illstrates the effects of the deformation on the final crvatre profiles. The checkers contained in a sqare are deformed sch that the top edge is converted into a single clothoid. As shown, the horizontal lines which were previosly

8 (e) Figre 7: Checker board patters are deformed from rectilinear to circlar and from circlar to rectilinear patterns. The original patterns ndergo (b-d) mltiple steps of deformations to yield the (e) final patterns. Note that the internal and external patterns show smooth shapes as the deformation field itself is smooth. straight, attain similar crvatre profiles as the final shape of the top edge. However, the crvatres deviate form a clothoid profile in the vicinity of the vertical edges at the left and right. Figre 10: The effect of the deformation on the crvatre profiles. The originally flat lines are deformed into still smooth bt not clothoid crves. 7. Implementation and Examples The proposed method is implemented in MATLAB. The ser sketches the inpt crves on the original image. We are sing a mltigrid solver for the deformation field calclation. For a typical image of size , the comptation of the deformation field and its sbseqent application to prodce the deformed image takes on the order of 10 seconds on a dal core 2.5 GHz laptop with 3 GB of memory. However, especially when deformations are large, it might be desirable to apply the deformation in sccessive smaller steps, which will proportionally increase the processing time. In all of or examples we applied the deformations in 8 steps. 8 In Figre 11, intended modifications are sketched in the form of anchor, sorce and target crves. The reslting deformation is shown in Figre 11. Note that the deformations cease at the regions marked by the anchor crves. However, there are discrepancies between the target strokes and their beatified versions. These large discrepancies are the reslt of fitting single clothoids on the target crves, i.e. each target crve is a single clothoid. Figres 12 and Fig. 14 present similar modifications for a mg and a sofa, respectively. As described, all target crves are beatified into piecewise clothoid crves. We adjsted the nmber of segments manally to have a desired level of crve complexity that leads to visally pleasing forms. For instance, in Figre 14, we prposeflly fit single clothoids to the target strokes to get the least amont of crvatre flctations (Figre 14). Or method is also applicable to hand-drawn sketches. Figre 13 shows edits on a manga character. Figre A.15 illstrates a series of progressive edits on a car concept sketch. The sketch is first modified throgh a set of ser drawn crves that define the modifications and constraints on the image (Figre A.15). The reslt of these modifications (Figre A.15) have gone throgh a second set of modifications (Figre A.15), reslting in the final image (Figre A.15(e)). Althogh or system handles a wide variety of images and sketches, exceptions exist. The approach is not designed to properly handle images with apparent textre elements, ths may case textre distortions at ndesirable magnitdes. A remedy wold be combining it with textre analysis and synthesis methods as it has been proposed in [10]. However, for shape exploration in prodct design this shortcoming has limited effect. Likewise, for local modifications the ser is reqired to limit the deformation sing anchor crves. This process can be fa-

9 Figre 11: A car design. The sorce, target and anchor strokes drawn by the ser are given by ble, red, and orange colors, respectively. The sorce crves are aligned with featres, and their corresponding target crves are beatified into clothoid crves. These crves define the intended modifications reslting in the final design. Figre 12: A typical mg design. The sorce, target and anchor strokes drawn by the ser are given by ble, red, and orange colors, respectively. The sorce crves are aligned with featres, and their corresponding target crves are beatified into clothoid crves. These crves define the intended modifications reslting in the final design. of separating the actal object from the backgrond for a more effective handling of the objects shape. 8. Conclsion We present a sketch-based image deformation techniqe for exploration of visally pleasing prodct forms from existing images and sketches. Or method achieves the aesthetic qalities throgh piecewise clothoid fitting from mltiple strokes, while deforming the image accordingly by a smooth deformation field. Or crrent stdies indicate that the approach works effectively for a variety of different images and sketches. Acknowledgements All design images and sketches sed in the paper are cortesy of their respective owners. Figre 13: The sketch of a manga character. The set of crves that define the intended modifications. The reslting modification. The close p view of the sorce and target crve pairs. The colors ble, and red denote, the sorce crves, and the beatified target crves, respectively. Appendix A. Clothoids and the Eler Spiral An Eler spiral is defined as a crve whose signed magnitde of crvatre changes linearly with its chord length. The typical Eler spider passes throgh the origin with zero crvatre and approaches to two niqe locations on the 2D plane. The crve can be defined as a differential eqation sch as: cilitated by a prior segmentation of the image into patches (e.g. the mg and its handle in Fig. 12). The patch nder deformation wold then be limited within itself not affecting the neighboring patches. Another extension wold be an atomated process dθ s ds 9 (A.1)

10 achieved. On the other hand, parametric crves typically reqire high nmber of control points to accrately prodce linearly changing crvatre profiles. Figre 14: A sofa design. The sorce, target and anchor strokes drawn by the ser are given by ble, red, and orange colors, respectively. The sorce crves are aligned with featres, and their corresponding target crves are beatified into clothoid crves. These crves define the intended modifications reslting in the final design. where θ is the angle of the tangent of the crve, and s is the chord length. With different variables, the same eqation can be written as: d 2 C(s) ds 2 = d C(s) ds d C(s) ds 1 s (A.2) where C(s) is the position vector of the point on crve at chord length s. Each and every segment of the Eler spiral is defined to be a clothoid crve, sharing exactly the same properties. Throgh integration, it can be shown that the position of any point on the crve does not have a closed soltion. The coordinates can be represented as Fresnel integrals, or as a nmerical approximation, it can be represented with power series expansions or nmerical integration methods. In or formlations, we se a nmerical integration scheme which calclates tangents and positions sbseqently from finite differences in the chord length. Compared to other commonly sed parametric crve representations, the clothoid crves exhibit varios differences. First of all, nlike polynomial fnction crves sch as splines, clothoids cannot be controlled locally throgh a finite set of control points. Local modifications on the clothoids reqires recalclating the governing eqations as to prodce local changes. This reqirement alongside with the linearly changing crvatre reslts in a constrained set of shapes that can be 10 References [1] Adobe Systems Incorporated, Photoshop, [2] J. Elder, R. Goldberg, Image editing in the contor domain, IEEE Transactions on Pattern Analysis and Machine Intelligence 23 (2001) [3] J. Elder, S. Zcker, Space scale localization, blr, and contor-based image coding, in: IEEE CVPR, pp [4] E. Sand, D. Fleet, D. Larner, J. Mahoney, Perceptally-spported image editing of text and graphics, ACM Transactions on Graphics 23 (2004) 728. [5] S. Bagley, G. Kopec, Editing Images of text, Commnications of ACM 37 (2001) [6] E. Sand, T. P. Moran, A perceptally-spported sketch editor, Proceedings of the 7th annal ACM symposim on User interface software and technology - UIST 94 (1994) [7] S. Seitz, K. Ktlakos, Plenoptic image editing, in: Sixth International Conference on Compter Vision IEEE, Narosa Pblishing Hose, 2002, pp [8] P. Pérez, M. Gangnet, A. Blake, Poisson image editing, ACM SIG- GRAPH 2003 Papers on - SIGGRAPH 03 (2003) 313. [9] S. Lee, K. Chwa, S. Shin, Image metamorphosis sing snakes and freeform deformations, in: Proceedings of the 22nd annal conference on Compter graphics and interactive techniqes, ACM, pp [10] H. Fang, J. Hart, Detail preserving shape deformation in image editing, ACM Transactions on Graphics (TOG) 26 (2007) 12 es. [11] H. Shngang, L. Shaoshai, L. Xiaoxiao, Image deformation based on cbic splines and moving least sqares, in: Comptational Intelligence and Software Engineering, CiSE International Conference on, IEEE, pp [12] T. Pal, A. Ntborne, Two-dimensional crve synthesis sing linear crvatre elements, Compter-Aided Design 9 (1977) [13] A. Schechter, Synthesis of 2d crves by blending piecewise linear crvatre profiles, Compter-Aided Design 10 (1978) [14] J. Stoer, Crve fitting with clothoidal splines, Jornal of Research of the Notional Brea of Standards 87 (1982) 317. [15] D. Meek, D. Walton, Clothoid spline transition spirals, mathematics of comptation 59 (1992) [16] T. Davis, Total least-sqares spiral crve fitting, Jornal of srveying engineering 125 (1999) [17] H. Späth, One dimensional spline interpolation algorithms, AK Peters, Ltd., [18] T. Harada, F. Yoshimoto, Atomatic crve fairing system sing visal langages, in: Information Visalisation, Proceedings. Fifth International Conference on, IEEE, pp [19] J. McCrae, K. Singh, Sketching piecewise clothoid crves, Compters & Graphics 33 (2009) [20] I. Baran, J. Lehtinen, J. Popović, Sketching clothoid splines sing shortest paths, in: Compter Graphics Form, volme 29, Wiley Online Library, pp [21] G. Orbay, L. B. Kara, Beatification of design sketches sing trainable stroke clstering and crve fitting, IEEE Transactions on Visalization and Compter Graphics 99 (2010). [22] M. Kass, A. Witkin, D. Terzopolos, Snakes: Active contor models, International Jornal of Compter Vision 1 (1988) [23] P. Bao, L. Zhang, X. W, Canny edge detection enhancement by scale mltiplication, IEEE transactions on pattern analysis and machine intelligence 27 (2005) [24] T. Harada, F. Yoshimoto, M. Moriyama, An aesthetic crve in the field of indstrial design, in: Visal Langages, Proceedings IEEE Symposim on, IEEE, pp [25] K. Mira, J. Sone, A. Yamashita, T. Kaneko, Derivation of a general formla of aesthetic crves, in: In: Proceedings of the Eighth International Conference on Hmans and Compters, pp [26] K. Mira, A general eqation of aesthetic crves and its self-affinity, Compter-Aided Design and Applications 3 (2006)

11 [27] K. Shoemake, T. Dff, Matrix animation and polar decomposition, in: Proceedings of the conference on Graphics interface, volme 92, Citeseer, pp [28] C. X, J. Prince, Snakes, shapes, and gradient vector flow, Image Processing, IEEE Transactions on 7 (1998) (e) Figre A.15: The original car sketch The set of sorce and target crves that define the first set of intended modification The intermediate reslt The set of sorce and target crves that define the second set of intended modifications (e) The final reslt. The colors ble, green, and orange denote, the sorce, the beatified target, and the anchor crves, respectively. 11