Sketching in the Air: A Vision-Based System for 3D Object Design

Transcription

1 Sketching in the Air: A Vision-Based System or 3D Obect Design Yu Chen 1, Jianzhuang Liu 1 1 Department o Inormation Engineering The Chinese University o Hong Kong {ychen6,zliu}@ie.cuhk.edu.hk Xiaoou Tang 1,2 2 Microsot Research Asia Beiing, China xtang@microsot.com Abstract mirror 3D obect design has many applications including lexible 3D sketch input in CAD, computer game, webpage content design, image based obect modeling, and 3D obect retrieval. Most current 3D obect design tools work on a 2D drawing plane such as computer screen or tablet, which is oten inlexible with one dimension lost. On the other hand, virtual reality based methods have the drawbacks that there are awkward devices worn by the user and the virtual environment systems are expensive. In this paper, we propose a novel vision-based approach to 3D obect design. Our system consists o a PC, a camera, and a mirror. We use the camera and mirror to track a wand so that the user can design 3D obects by sketching in 3D ree space directly without having to wear any cumbersome devices. A number o new techniques are developed or working in this system, including input o obect wirerames, gestures or editing and drawing obects, and optimization-based planar and curved surace generation. Our system provides designers a new user interace or designing 3D obects conveniently. 1. Introduction Despite great progress o 3D modeling in current computer-aided design (CAD) tools, creating 3D obects using these tools is still a tedious ob since they require users to work on a 2D drawing plane. Design in virtual 3D environments enables users to draw obects in 3D space, but this method has the drawbacks that there are awkward devices worn by the user and the virtual environments are expensive. In this paper, we propose a novel vision-based approach to 3D obect design. Dierent rom the current techniques, it works in 3D space without any devices connected to the user. Our target is to develop an inexpensive system that allows the user to design 3D obects conveniently. Our system consists o a PC, a camera, and a mirror, as shown in Fig. 1. In the system, the user designs 3D obects by sketching in the air the wirerames o the obects with an easily-tracked wand. A number o sketching and editing operations are Fig. 1. The sketching system. developed to acilitate obect design. In real time, the 3D positions o the strokes o the wand are captured, and the wirerames and suraces being developed are displayed on the PC screen to guide the user to draw more and more complex obects. The system provides a whole new way o 3D obect design. It requires no special equipments and is easy to set up and use. Its applications include lexible 3D sketch input in CAD, game, education, and webpage content design, generation o 3D obects rom 2D images, and a user-riendly query interace or 3D obect retrieval. 2. Related Work Great eort has been made to develop CAD systems or 3D model design in the past three decades. Current techniques can be classiied into the ollowing our categories: 1) Traditional CAD tools such as AutoCAD [1] and SolidWorks [2]. These tools are sophisticated systems suitable or engineers to input precise geometry o models but are not suitable or designers to rapidly express their ideas at the initial stage o model development. 2) Automatic 3D obect reconstruction rom 2D line drawings. This is one o the main research topics in computer vision and graphics. The methods are mainly based on line labeling, algebra, image regularities, and optimization [18], [11], [17], [12], [5]. The critical problem in these methods is that they can handle only relatively simple pla /08/$ IEEE

2 nar obects at the current stage. 3) Sketch-based modeling user interaces. Most traditional designers still preer pencil and paper to mouse and keyboard in current CAD systems to sketch their ideas o shapes. To bridge the gap between the lexible 2D sketches and the rigid CAD systems, researchers have developed tools that try to convert 2D sketches into 3D models [17], [19], [8], [3], [9]. However, one physical limitation that cannot be overcome by these tools is that the sketching and editing operations are perormed on a 2D plane (tablet or screen). With one dimension missing, the 3D positions o the strokes, suraces, and obects drawn on a 2D plane are oten ambiguous. 4) Design in virtual 3D environments. Virtual reality (VR) has been thought to be the perect CAD system because the designer could work naturally and intuitively in a real 3D environment. However, such systems ace problems in the cost o the equipments, the inlexibility to use, and the slow rame update rates. Researchers are trying to develop better techniques or 3D design in VR [16], [10], [4], [7] but they need special and awkward devices to operate by, or connect to, the user, making the design an unnatural process. From the discussion above, we can see that the current methods or 3D model design are not good enough. Researchers still need to develop more riendly and inexpensive interaces with better design methodology. 3. The Sketching System As shown in Fig. 1, our system consists o a video camera, a mirror, and a PC only. The user draws an obect with a wand in the 3D ree space. The tip o the wand is colored so that it is easy to track. The basic idea o 3D design in this system is that a 3D wirerame o an obect is obtained by tracking the movement o the wand in 3D space, and then an automatic illing-in process generates a surace rom the wirerame. We propose this system based on the observation that a designer thinks not in terms o suraces, but rather in terms o the eature curves o an obect which construct the wirerame. The whole low chart o our system is shown in Fig D Geometry o the System Being able to ind the 3D position o the tip o the wand is the irst step. In the system, the world rame (X, Y, Z) is deined as in Fig. 3, where the XY plane is parallel to the image plane xy o the camera and the distance between the origin O and the image plane is equal to the ocal length. With a simple calibration, the YZ plane can be set orthogonal to the mirror. The angle θ between the Z axis and the mirror is less than 90 degrees so that the tip o the wand P 1 = (X 1,Y 1,Z 1 ) T and its image P 2 =(X 2,Y 2,Z 2 ) T in the mirror always proect to two wirerame in 3D space drawing & editing o the wirerame in 3D space camera the user wirerame on the PC drawing & editing o the wirerame on the screen with the keyboard ace identiication proection on the screen surace generation proection on the screen Fig. 2. Flow chart o 3D obect design in the sketching system. O Y image plane y X p 2 p o 1 P1 B Z 0 θ A mirror Fig. 3. Geometry o the system. dierent points p 1 =(x 1,y 1,) T and p 2 =(x 2,y 2,) T on the image plane. To indthe3dcoordinateop 1,we need to know θ and Z 0,whereZ 0 is the distance rom O to A and A is the intersection o the Z axis and the mirror. The two parameters θ and Z 0 canbeobtainedbythe calibration scheme discussed in Section 4.2. The 3D position P 1 =(X 1,Y 1,Z 1 ) T can be determined rom the known θ, Z 0, p 1, p 2, and the geometrical relationship shown in Fig. 3. The ormula or calculating P 1 is derived in Section 4.1. Our system is able to determine the 3D positions o the wand with suicient accuracy. We have also tested the traditionalstereomethodusingtwocamerastoind the depth o a spatial point. From our experiments, we have ound that the new method has the ollowing advantages: (a) easier to calibrate, (b) less time to track the wand due to only one video sequence to handle, and (c) larger 3D working space or obect drawing i the volume to set up the two systems are the same. The last advantage comes rom the act that i the traditional stereo method is used, the tip o the wand must appear in both image sequences o the cameras, which limits the 3D drawing space Locating the Wand Locating the 3D position P 1 o the wand is the irst step or the system to work. We can represent P 1 = (X 1,Y 1,Z 1 ) T in terms o the points p 1 and p 2 on the image plane, and the parameters, θ, andz 0. First, rom the geometrical relationship in Fig. 3, it is straightorward to relate the spatial positions with the positions in the image plane by X i = x i Z i /, Y i = y i Z i /, i =1, 2. (1) x P 2 Z

3 Y Mirror Y y image plane mirror O Z 1 Z 0 Z 2 Y 1 Q P B H θ C A Fig. 4. Geometry o the system on the YZplane. R W H A' A (a) W S (b) Q' Y 2 r a' s w 1 w 2 h a Fig. 5. (a) A rectangle or calibration. (b) The rectangle in the image. We now consider the geometry o the system on the YZ plane as shown in Fig. 4, where P and P 0 are the proections o P 1 and P 2 onto the YZplane, respectively. Let B be the midpoint o PP 0, and the three lines PQ, P 0 Q 0,andBC be perpendicular to the axis OZ. Then PQ + P 0 Q 0 = 2 BC and OQ + OQ 0 =2 OC. Hence, we have On the other hand, Y 2 = P 0 Q 0 =2 BC PQ = 2H sin θ y 1 Z 1 /, (2) Z 2 = OQ 0 =2 OC OQ = 2(Z 0 H cos θ) Z 1. (3) H = AB = PQ sin θ + AQ cos θ = y 1 Z 1 sin θ/ (Z 1 Z 0 )cosθ. (4) From (1), (2), (3), and (4), we have, y 2 = Y 2 Z 2 = 2H sin θ y 1Z 1 / 2(Z 0 H cos θ) Z 1 = 2y 1Z 1 sin 2 θ (Z 1 Z 0 ) sin 2θ y 1 Z 1 2Z 0 sin 2 θ y 1 Z 1 sin 2θ + Z 1 cos 2θ.(5) Finally, Z 1 is obtained rom (5) as 2Z 0 sin θ(y 2 sin θ cos θ) Z 1 = (y 1 y 2 2 )sin2θ (y 1 + y 2 ) cos 2θ. (6) Given (1), we immediately have the other two dimensions o P 1 : X 1 = x 1 Z 1 / and Y 1 = y 1 Z 1 /. The position o the wand P 1 in the 3D space is hence determined i, θ, and Z 0 are known Calibration The calibration process is to ind the parameters θ and Z 0. It is reasonable to assume that the Z axis in Fig. 3 is P' Z O h o Z 0 A' H θ A Fig. 6. Geometry o the YZplane in calibration. orthogonal to the image plane and passes through the center o o the image displayed on the screen. The ocal length can be known rom the camera and is ixedinthesystem.to calibrate the system, we print out a rectangle (Fig. 5(a)) on a white page and place this page on the central part o the mirror with the side RS approximately parallel to the ground. This rectangle is captured by the camera and displayed on the screen as shown in Fig. 5(b). Then we adust the position o the camera so that the point a is coincident with the image center o, the side rs is horizontal, and w 1 = w 2 in the image. This simple adustment makes the YZplane orthogonal to the mirror. With the known lengths o the sides o the rectangle in Fig. 5(a) and h and w 1 in the image, we can ind θ and Z 0 rom w 1 W = h H sin θ = Z 0 H cos θ, (7) whichisderivedromthegeometryshowninfig Wirerame Input and Obect Editing One diiculty to generate a wirerame o an obect lies in the act that the strokes drawn are invisible to the user. However, they are visible to the camera, and the trace o the moving wand and what has been drawn can be displayed on the screen as the eedback to the user, which can be used to guide the sketching process. First, we propose a solution to locating the position on an uninished wirerame in order to continue to draw. While the user moves the wand in the space, the closest point on the uninished wirerame to the tip is computed, and a dierent color is shown on the screen to indicate this point. In this way, the user can ind a position to draw without diiculty. When this position is ound, the user presses some key to let the system know it, and then the movement o the tip is considered as a new edge o the wirerame. Second, to distinguish a drawing stroke rom nondrawing movement o the wand, we use the keyboard to let the system know when a stroke begins and ends. Besides, we have also developed a number o sketching and editing operations to acilitate obect design, such as extrusion, moving, copying, rotation, and zooming. These 3D operations and gestures are summarized in Table 1. The keyboard is used to control the start and stop o a 3D gesture shown Z

4 Table 1. Gestures and operations deinedinthesystem. Keyboard Function m/r move/copy the selected part Let/Right rotate along Y-axis Up/Down rotate along X-axis +/- zoom in/out d a c/e mode 1: move the position o a vertex or a control point mode 2: sketch a patch by dragging a straight/curve edge mode 3: sketch a pyramid/cone structure by dragging a patch mode 4: sketch a rectangular/cylindrical volume by dragging a patch draw a straight line/curve mode 1: draw a circle/ellipse mode 2: draw a body o revolution rom a straight/curve edge q/w adust the scaling o the selected part z/x adust the curvature o curved strokes 1-4 change editing modes F1 switch between the curve mode and the straight-line mode by the movement o the wand. All the operations can be done by the wand and the keyboard, without resorting to the mouse, thus allowing a continuous design by moving thewandwithonehandandhittingthekeyboardwithanother. The system also allows the user to switch between two drawing modes: the curve mode and the straight-line mode. In the curve-mode, smooth Bezier curves are generated to it the path o wand movement when orming the wirerame. On the other hand, in the straight-line mode, the path inormation is discarded and only the start and the end positions o each stroke is used to generate straight lines. Compared with traditional sketch-based editing operations on a 2D plane, many 3D operations have their advantages. For example, i we want to draw a duct by the extrusion o a closed circle along an arbitrary open curve (see Fig. 7), a 2D system will encounter two problems: (a) whether the closed curve is a circle or ellipse and its orientation in 3D space are unclear; (b) the 3D trail o the open curve is impossible to determine. However, these problems do not exist in our system. 5. Surace Generation When we obtain the wirerame o an obect in the 3D space by using the sketching scheme described in the previous section, the next step is to generate the 3D surace rom the wirerame to inally reconstruct the obect. Surace generation can be divided into two steps. The irst is to identiy all the aces, i.e., the circuits in the wirerame which represent patches constituting the whole surace, and the second Fig. 7. Extrusion o the circle along the curve. step is to generate these patches, either planar or curved, rom their boundaries Face Identiication Given a wirerame, beore illing in it with surace patches, we have to identiy the circuits that represent these patches (aces). Since the wirerame may represent a maniold or non-maniold solid, a sheet o surace, or the combination o them, with or without holes, identiying the aces is not a trivial problem due to the combinatorial explosion in the number o circuits in the wirerame [13], [14], [15]. To solve this problem, we use the algorithm proposed in [15] to detect the aces o a wirerame. In our interactive system, the user can also select the edges o a ace or ace identiication manually, which can help ix wrongly detected aces by the algorithm occasionally Planar Surace Generation An obect may have many planar aces. In the system, a straight line replaces a stroke in the straight-line mode. It is reasonable to consider that a ace is planar i all its edges are straight lines. However, or a planar ace with more than three vertices, it is not likely or all the vertices to be located exactly on a plane in 3D space due to the inaccuracy o the measurement and the input during the sketching process. Filling in these circuits with triangular patches will make the obect distorted. In order to solve this problem, we propose an automatic line drawing correction algorithm to deal with this problem. Ater ace identiication rom a (partial) wirerame, we know the circuits representing planar aces. From the vertices o these circuits, a itting algorithm is used to ind a set o planes that best it these planar circuits. We represent a plane passing through ace by its normal vector = (a,b,c ) T and a scale d. Then, any vertex v =(x, y, z) T on this plane satisies the linear equation: a x + b y + c z d =0or v T = d. We hope that or each identiied ace, the corrected positions o its vertices should be as close to the itting plane as possible. Besides, or each vertex, its corrected position should not deviate too much rom its initial position. Let V be the set o the vertices o ace. The obective unction to be minimized is deined as ollows: Q(v 1, v 2,, v N, 1, 2,, M,d 1,d 2,,d M )= NX MX X kv i v 0 i k2 i + β d k 2, (8) k k 2 i=1 kvt =1 i V

5 where v 1, v 2,, v N are the corrected positions o N vertices; ( 1,d 1 ), ( 2,d 2 ),, ( M,d M ) are the parameters o M planar aces; v 0 1, v2, 0, v 0 N are the positions o the N vertices in the original sketching; β is a weighting actor. The goal o the optimization is to ind the corrected positions v i, i =1, 2,,N, and the itting planes (,d ), =1, 2,,M, such that Q is minimized. We solve this optimization problem in an iterative way. Let the set o v i, i = 1, 2,,N, the set o, = 1, 2,,M, and the set o d, =1, 2,,M be V = {v i } N i=1, F = { } M =1, andd = {d } M =1, respectively. Also let V n = {vi n}n i=1, F n = { n}m =1,andDn = {d n }M =1 be the optimization results ater the nth iteration. The optimization problem is divided into two iterative minimization steps, and a closed-orm solution can be achieved in each step. Step 1: Face itting. (F n+1,d n+1 ) = argmin Q(V n,f,d). (9) F,D Step 2: Vertex correction. V 0 = {vi} 0 N i=1, (10) V n+1 = argmin Q(V, V Fn+1,D n+1 ). (11) In Step 1, we do the plane itting on all the identiied planar aces using the updated positions o the vertices obtained in the previous iteration. The optimal itting is obtained as ollows. First, by solving Q d =0,wehave d = 1 X v T i, =1, 2,,M. (12) V i V Substituting (12) into (8), ater some algebraic manipulation, we can transorm the problem in (9) into the problem o minimizing the Rayleigh quotient T S with T respect P to each, = 1, 2,,M, where S = 1 V i V (v i v )(v i v ) T is the covariance matrix, and v = 1 V Pi V v i. Furthermore, minimizing T S T can be reduced to the ollowing eigen-problem: S = λ,min, =1, 2,,M, (13) with being the eigen vector corresponding to the minimum eigen value λ,min o S. From (12) and (13), we can obtain the closed-orm solution n+1 and d n+1, = 1, 2,,M,intermsovi n, i =1, 2,,N. Step 2 is done by minimizing Q, giventheitting planes obtained in Step 1. From Q v i = 0, i =1, 2,,N,we have Q =2(v i v v i)+2β X 0 (vt i d ) = 0, (14) i k k 2 F i where F i is the set o the aces containing vertex i. The closed orm solution to (14) results in the ollowing equation or computing vi n+1 : vi n+1 =(R n+1 ) 1 ˆv i n+1, i =1, 2,,N, (15) where R n+1 = I + β X n+1 n+1 k n+1 F i k, (16) 2 ˆv n+1 i = v 0 i + β X n+1 d k n+1 F i T n+1 k 2. (17) Our experiments have shown that the algorithm is eective and converges quickly within several iterations Smooth Curved Surace Generation Ater a curved stroke is inished,weuseabeziercurve to approximate it to obtain a smooth curve. Ater we have a (partial) wirerame with identiied aces (circuits) and curves represented by Bezier curves, we ill in the circuits denoting curved aces with smooth surace patches. BilinearlyblendedCoonspatches[6]areusedtodoit. 6. Experiments In this section, we show a number o examples to demonstrate the perormance o our system. Our system is implemented using Visual C++, running on a PC with 3.4 GHz Pentium IV CPU. The parameter β in(8)ischosentobe 10. Our experiments show that the system is insensitive to the parameters; very similar results are obtained when β changes in [5, 20]. The wand tracking and display module work in real time at a rate o 10 rames per second. The system can track the moving o the wand at a maximum speed o about one meter per second. A new user usually needs to take two or three hours o training to get adapted to simultaneous keyboard and wand operation in the system. In our system, strokes are preprocessed and the displayed wirerames are composed o straight lines and smooth curves. The ittering o the strokes by natural hand tremor is smoothed. Fig. 8 shows a set o wirerames created with our system. For each wirerame in the irst column, we give the corrected wirerames in the second column and the 3D reconstruction result displayed in two views in the third and ourth columns. The results show that the correction step (Section 6.2) is eective and the aces o the reconstructed obects are planar. Figs. 9 shows a set o more complex obects that include strokes o both straight-lines and curves. We can see that our system can handle complex wirerames sketched in the air and generate expected 3D obects. Thetimeusedortheaceidentiication and the generation o planar and curved suraces o a scene is between 3 and 19 seconds, depending on the complexity o the scene.

6 8. Acknowledgements The work described in this paper was ully supported by a grant rom the Research Grants Council o the Hong Kong SAR, China (Proect No. CUHK ). Fig. 8. Experimental results. The axes o the 3D coordinate system are also shown. Fig. 9. More experimental results. For instance, the system takes 3 and 19 seconds to reconstruct the second obect in Fig. 8 and the ourth obect in Fig. 9 respectively, ater the wirerames are obtained. 7. Conclusion and Future Work We have developed a novel 3D vision-based sketching system with a simple and inexpensive interace allowing users to sketch obects directly in the 3D space. A number o new techniques are proposed or working in this system, including input o obect wirerames, gestures or editing and drawing obects, and optimization-based planar and curved surace generation. Experiments have veriieditseicacy in designing 3D obects. Our system is still being improved. At the current stage, a new user usually needs to take two or three hours o training to get adapted to simultaneous keyboard and wand operation in the system. Our uture work includes: 1) developing more gestures and operations to handle complex obects; 2) improving the tracking algorithm to support more accurate 3D positioning o the wand movement. Reerences [1] Autodesk Inc. AutoCAD. [2] SolidWorks Corporation. SolidWorks. [3] A. Alexe, L. Barthe, M. Cani, and V. Gaildrat. Shape modeling by sketching using convolution suraces. Proc. Paciic Graphics, [4] R. Amicis, F. Bruno, A. Stork, and M. Luchi. The eraser pen: a new interaction paradigm or curve sketching in 3D. Proc. 7th Intl Design Conerence, 1: , [5] Y. Chen, J. Liu, and X. Tang. A divide-and-conquer approach to 3D obect reconstruction rom line drawings. ICCV, [6] G. Farin. Curves and Suraces or Computer Aided Geometric Design: A Practical Guide. Academic Press, [7] T. Grossman, R. Balakrishnan, and K. Singh. An interace or creating and manipulating curves using a high degreeo-reedom curve input device. SIGCHI Conerence, pages , [8] T. Igarashi, S. Matsuoka, and H. Tanaka. Teddy: a sketching interace or 3d reeorm design. SIGGRAPH, pages , [9] O. Karpenko and J. Hughes. Smoothsketch: 3D ree-orm shapes rom complex sketches. SIGGRAPH, pages , [10] D. Keee, D. Feliz, T. Moscovich, D. Laidlaw, and J. LaViola. Cavepainting: a ully immersive 3D artistic medium and interactive experience. Symposium on Interactive 3D Graphics, pages 85 93, [11] H. Lipson and M. Shpitalni. Optimization-based reconstruction o a 3D obect rom a single reehand line drawing. Computer-Aided Design, 28(8): , [12] J. Liu, L. Cao, Z. Li, and X. Tang. Plane-based optimization or 3D obect reconstruction rom single line drawings. IEEE Trans. PAMI, 30(2): , [13] J. Liu and Y. Lee. A graph-based method or ace identiication rom a single 2D line drawing. IEEE Trans. PAMI, 23(10): , [14] J. Liu, Y. Lee, and W. K. Cham. Identiying aces in a 2D line drawing representing a maniold obect. IEEE Trans. PAMI, 24(12): , [15] J. Liu and X. Tang. Evolutionary search or aces rom line drawings. IEEE Trans. PAMI, 27(6): , [16] S. Schkolne, M. Pruett, and P. Schroder. Surace drawing: creating organic 3D shapes with the hand and tangible tools. SIGCHI Conerence, pages , [17] A. Shesh and B. Chen. Smartpaper: An interactive and user riendly sketching system. Proc. Eurograph, [18] K. Sugihara. Machine Interpretation o Line Drawings. MIT Press, [19] R. Zeleznik, K. Herndon, and J. Hughes. Sketch: an interace or sketching 3D scenes. SIGGRAPH, pages , 1996.