A rich discrete labeling scheme for line drawings of curved objects

Transcription

1 A rih disrete labeling sheme for line drawings of urved objets Martin C. Cooer, IRIT, University of Toulouse III, Toulouse, Frane Abstrat We resent a disrete labeling sheme for line drawings of urved objets whih an be seen as an information-rih extension of the lassi line-labeling sheme in whih lines are lassified as onvex, onave, oluding or extremal. New labels are introdued to distinguish between urved and lanar surfae-athes, to identify orthogonal edges and to indiate gradient diretions of lanar surfae-athes. keywords: sene analysis, shae, line drawing analysis, line labeling, soft onstraints. 1 Regularities in man-made objets Most man-made objets have ertain harateristi shae features whih distinguish them from natural objets suh as roks, louds or trees. In this aer we onentrate on lanarity and orthogonality, although learly other regularities (suh as symmetry and isometry) an also rovide useful visual lues in the interretation of drawings of man-made objets. We say that an edge E, whether urved or lanar, is orthogonal if at eah oint on E the tangents to the two surfaes whih meet at E are orthogonal. Consider, as an illustration, the two drawings in Fig. 1. Of the 20 visible faes of these two objets, 13 are lanar. Of the 57 visible edges, 51 are orthogonal, the six non-orthogonal edges being marked by an 1

2 asterisk. Both of these drawings aear unambiguous to a human viewer. In artiular, we immediately identify surfae A in Fig. 1(a) as lanar and surfae B as urved, although most eole have great diffiulty in exlaining why they ame to this onlusion. This aer is onerned with setting down some basi loal geometrial rules whih will allow a omuter to automatially identify lanar surfaes and orthogonal edges. Possible aliations inlude sketh interretation [21], automati indexing of databases of line drawings [19] and 3D objet retrieval from the web [1]. We follow in the tradition of Kanade [9] who advoated the use of more information about the hysial world, thus avoiding overstrit onstraints based on unrealisti assumtions (suh as lanar surfaes meeting at trihedral verties [8, 2]) or the otimization of a fairly arbitrary objetive funtion. An imortant oint is that features that are ommon in manmade objets, suh as orthogonal edges or lanar faes, are essentially disrete roerties. For examle, 90 degree angles are ommon, but 85 degree angles are no more ommon than 65 degree angles. Thus an objetive funtion F, based on orthogonality (or other essentially disrete roerties), whih is a true refletion of the likelihood of a omlete 3D reonstrution should take on the same onstant value over most of the solution sae, sine the vast majority of solutions involve no orthogonality. Soft Constraint Satisfation [16] rovides a more aroriate method for otimizing F rather than searh methods in real-valued arameter sae [15, 12, 21]. In a revious aer we have shown how soft onstraint satisfation rovides a framework in whih we an ombine strit geometrial onstraints and referene onstraints [6]. An examle of a strit geometrial onstraint is that arallel 3D lines annot interset; an examle of a referene onstraint is that we refer a air of lines whih are arallel (within a given error tolerane) in the drawing to be rojetions of 3D arallel edges, sine arallel edges are ommon features of man-made objets. Our aroah is similar to that of Ding & Young [7] who used a Truth Maintenane System for the omlete 3D reonstrution of olyhedral objets from imerfet line drawings. We introdue new onstraints for the analysis of line drawings of urved objets, but we do not attemt hidden-art reonstrution. 2

3 * * P B E A * * (a) J * K * (b) Figure 1: Examles of drawings of urved man-made objets. 2 Line labels for urved objets Line-drawing labeling, in whih eah line is assigned a semanti label ( + for onvex, for onave or for oluding), was ioneered by Huffman [8] and Clowes [2]. At a onvex (onave) edge E, the external angle between the two visible surfaes whih interset at E is greater (less) than π; at an oluding edge, only one surfae is visible. They established the atalogue of legal labelings of juntions reated by the rojetion of trihedral verties. These onstraints, together with the Outer Boundary Constraint (whih simly says that the outer boundary of the drawing is neessarily an oluding line), were often found to be suffiient to uniquely determine the orret semanti labeling of the drawing of a olyhedral objet. Catalogues of labeled juntions have also been established for non-trihedral verties [20], but in this ase extra visual ues, suh as arallel lines, are required to avoid an exonential number of legal global labelings [6]. Malik [14] extended semanti labeling to line drawings of urved objets omosed of smooth surfae athes searated by surfae-normal disontinuity edges. His atalogue was refined and extended to inlude objets with tangential edges and surfaes [3, 4]. When surfaes are urved, the semanti label of a line may hange at any oint due to the resene of hantom juntions. For examle, transitions from oluding to onvex labels, suh as the oint P in Fig. 1(a), are ommon in line drawings of urved objets. Other transitions may our. For examle, as ointed out in [3], a onvex-onave transition an our if distint 3

4 + (a) (b) Figure 2: Illustration of (a) semanti line labels, and (b) lanarity labels. surfaes may be tangential to eah other. Under the Straight Edge Formation Assumtion, whih says that a straight edge is formed by the intersetion of two loally-lanar surfaes [5], the semanti label of a line L is invariant along any straight segment of L. In artiular, a straight line has a unique semanti label. Eah global semanti labeling an be individually heked for hysial realisability via Sugihara s linear rogramming aroah [18] generalized to urved objets with some linear features [5]. Unfortunately, although mathematially elegant, Sugihara s aroah does not hel us find the most likely interretation of a drawing. 3 Planarity onstraints Consider a 3D edge E formed by the intersetion of two surfaes S, T. We say that S is loally lanar along E if the tangent to S at eah oint of E is idential. Planar surfaes are neessarily loally lanar, but a urved surfae may also be loally lanar. As an examle, onsider surfae B in Fig. 1, whih is loally lanar along the edge E, sine it has an invariant tangent lane along the whole length of E. A surfae-normal disontinuity edge (a onvex, onave or oluding edge) is formed by the intersetion of two surfaes S, T. A line L whih is the rojetion of a surfae-normal disontinuity edge has a semanti label +,, or, as illustrated in Fig. 2(a). We also label eah side of L by a lanarity label or to indiate whether S, T are loally lanar or not ( for lanar and for urved ). At an oluding edge, one of the surfaes, say T, is invisible. The lanarity label of S is written on the side of the line into whih the surfae S rojets and the lanarity label of T on the other side. This is illustrated in Fig. 2(b): the rightmost label indiates that the hidden base of the ylindrial art of the objet is 4

5 L L urved = or L +/ / L straight = under the Straight Edge Formation Assumtion + + L or L = L 1 L 2 = x 1 x 2 x 1 = x 2 {,} orth(l 1 )=orth(l 2 ) = Figure 3: Planarity onstraints. loally lanar. An extremal edge is the lous of oints of intersetion of the line of sight with a urved visible objet surfae. The rojetion of an extremal edge, known as an extremal line, is labeled by a double-headed arrow as shown in Fig. 2(a). By onvention, both sides of an extremal line are always labeled, as shown in Fig. 2(b). Having introdued the new lanarity labels, we an now give the lanarity onstraints whih relate lanarity and semanti line labels. These are given in Fig. 3. We say that the drawing satisfies the General Viewoint Assumtion (GVA) if no small erturbation in the osition of the viewoint hanges the onfiguration of the drawing (juntion-tyes, resene of straight lines and arallel lines). Under the GVA, the rojetion of a 3D edge E is a straight line if and only if E is a straight edge. The first onstraint in Fig. 3 simly says that a urved line annot be the rojetion of the intersetion of two loally lanar surfaes. The seond onstraint is a translation of the Straight Edge Formation Assumtion. The last three lanarity onstraints in Fig. 3 (showing a hantom, a 3-tangent and a urvature-l juntion resetively) allow us to dedue that a surfae is urved. Consider a 5

6 3D oint P at whih a surfae-normal disontinuity edge E intersets an extremal edge E ext, and let S be the surfae in whih the extremal edge lies and T P the tangent-lane to S at the oint P. By the definition of an extremal edge, the viewoint lies in the lane T P. If S were loally lanar along E, then T P would be the tangent lane to this loally lanar segment of E, whih would ontradit the General Viewoint Assumtion. This reasoning allows us to dedue the labels shown in the last three lanarity onstraints in Fig. 3. There are three distint onstraints deending on whether the extremal edge E ext or art of the surfae-disontinuity edge E is oluded. The dot reresents a disontinuity of urvature between the rojetions of E and E ext. These lanarity onstraints are valid even in the ase of objets with tangential edges and surfaes [4]. Sine lanar surfaes are ommon in man-made objets, it is natural to try to maximize the number of lanar surfaes in our interretation of the drawing. However, simly maximizing the number N of labels is not always suffiient to determine the orret lanarity labeling: for examle, alying this riterion to the drawing in Fig. 1(a) does not allow us to distinguish between the two labelings and for the edge searating faes A and B. Considering the drawing as a lanar grah G, the faes of G are rojetions of visible (artial) surfae-athes of the 3D objet. A fae F of G an only be the rojetion of (art of) a lanar surfae if all its lanarity labels are. We say that F is loally lanar if all its lanarity labels are. Maximizing the riterion N + N lf, where N lf is the number of faes of G whih are loally lanar, allows us to refine the artial order defined by N and hene to find the orret lanarity labeling of the drawing in Fig. 1(a). 4 Constraints from orthogonal edges It is well known that identifying a labeled juntion as the rojetion of a ubi orner allows us to alulate the 3D orientation of the three faes that meet at the orresonding vertex [17, 10]. This setion shows that we an extend this to urved objets, by determining ertain information about the surfaes whih meet at viewoint-deendent verties, based only on the assumtion that the 3D edge E is orthogonal. Consider the first onstraint shown in Fig. 4, reading the imliation from left to right. 6

7 3-tangent + urvature-l or hantom + Figure 4: Gradient-diretion onstraints assuming that the surfae-normal disontinuity edges are orthogonal. The imliations are onsequenes of the GVA. It onerns a labeled 3-tangent juntion formed by the rojetion of a urved orthogonal edge E whih is the intersetion of a urved surfae S with a lanar surfae S. Let P reresent the 3D oint on E at whih the tangent-lane to S asses through the viewoint, and let T P reresent this tangent-lane. If n is the normal to the lanar surfae S, then by orthogonality of E, n is arallel to T P. It follows that the rojetion of n in the drawing is arallel to the extremal edge (whih is the rojetion of T P ). We symbolize the diretion of the rojetion of n by a short arrow next to the on the right hand side of Fig. 4. Similar onstraints exist, and are given in Fig. 4, for urvature-l juntions and hantom juntions. The arrowhead indiates the diretion of n. By onvention, the 3D orientation of n, the normal to the lanar surfae S, is always towards the viewoint. This onvention means that in the urvature-l onstraint of Fig. 4, the gradient diretion is the same whether E is a onave or an oluding edge. For the hantom juntion shown in Fig. 4, the gradient diretion an only be determined aurately if the osition of the hantom juntion is reisely known. However, this is an imortant onstraint when read from right to left, sine it allows us to loate hantom juntions at the oints where the gradient-diretion is tangential to the line. Under orthograhi rojetion, the gradient-diretion is invariant on a lanar surfae, and hene roagates through juntions and along loally-lanar surfaes. Under ersetive rojetion, the gradient diretions of a surfae meet at a vanishing oint; 7

8 (a) or = + or = / / (b) s αβ = π 2 < α < π 2 π 2 < β < π 2 s = + if π 2 < β < α < π 2 s {,, } if π 2 < α < β < π 2 Figure 5: The gradient-diretion/semanti-label onstraints: (a) assuming that the edge is orthogonal; (b) for any edge. at least two gradient diretions are required to loate the vanishing oint of the normals to a lanar surfae. For eah line segment L in the drawing we reate a boolean variable orth(l) whih takes on the value true if and only if L is the rojetion of an orthogonal edge segment. We then imose the hard onstraints orth(l) the gradient-diretion onstraints aly at eah 3-tangent, urvature-l and hantom juntion on L. together with the soft onstraint whih imoses a enalty of w o if orth(l) = false. Following the disussion in Setion 3, there is also a enalty of w h for eah hantom juntion, a enalty of w for eah lanarity label, a enalty of w lf for eah fae whih is not loally lanar, and a enalty w t for eah lanarity label transition ( to ) between the two ends of a line segment. The objetive funtion to be minimized is the sum of these enalties. Hard onstraints an be viewed as enalty funtions taking values in {0, } [16]. There are also gradient-diretion onstraints at rojetions of trihedral verties V at whih surfaes and edges meet non-tangentially. Let L 1, L 2, L 3 be the rojetions of the edges E 1, E 2, E 3 meeting at V and let F 12 be the fae bounded by E 1 and E 2. The normal n to F 12 is arallel to E 3 in 3D iff both E 1 and E 2 are orthogonal at V. Thus, invoking the 8

9 GVA whih says that arallel 2D lines are rojetions of arallel 3D lines, we an dedue that the gradient-diretion of F is arallel to L 3 iff both E 1 and E 2 are orthogonal at V. The gradient-diretion/semanti-label onstraints in Fig. 5(a) show the tight relationshi between the gradient diretion and the semanti label of a line L, whenever L is the rojetion of an orthogonal edge. The arrows reresent any gradient diretion whih oints away from L in the first onstraint and towards L in the seond onstraint. The line L is shown urved, but ould be straight or urved in the oosite diretion. At a oint P on an orthogonal edge E at whih surfaes S 1, S 2 interset, the normal n 1 (n 2 ) to surfae S 1 (S 2 ) is arallel to the tangent-lane to the surfae S 2 (S 1 ) at P. If E rojets into a onvex (onave) line L, it follows that the rojetions of n 1,n 2 oint away from (towards) L. Sine, by onvention, oluding lines have the same gradient-diretion labels as onave lines, this roves the orretness of the gradient-diretion/semanti-label onstraints given in Fig. 5(a). One onsequene of these onstraints is that a losed urve L whih is the rojetion of a loally-lanar orthogonal edge annot have the same semanti label around the whole of L. If L is not a losed urve but instead terminates at a 3-tangent or urvature-l juntion, then the gradient-diretion onstraints allow us to determine the gradient-diretion. This, in turn, rovides us with the semanti label at eah oint of L via the gradient-diretion/semantilabel onstraints. In Fig. 5(b), the surfaes S 1,S 2 are both loally-lanar and hene L is neessarily straight. The right-hand end of L is further away from the viewer than the left-hand end if and only if π < α < π and if and only if π < β < π. Furthermore, for π < α < π, the dihedral angle between the surfaes S 1 and S 2 is greater than π if α > β, equal to π if α = β and less than π if α < β. The onstraints of Fig. 5(b) follow immediately. 5 Exerimental trials To test our labeling sheme, we built u a test set of 28 drawings, all rodued by orthograhi rojetion: the four drawings in Fig. 1 and Fig. 6, the 15 drawings of urved objets from [3], the four drawings of urved objets from the test-set of [13] and the five drawings given in Fig. 2 of [1]. Any lines whih were not rojetions of visible edges were eliminated 9

10 (b) (a) Figure 6: Drawings of urved objets with orthogonal edges. in order to rodue erfet rojetions of oaque objets. In the orret interretations, 76.7% of the 395 edges are orthogonal, 72.9% of the 229 faes are lanar and 67.6% of the 1044 lanarity labels are. The GVA was satisfied in eah of the drawings. We alied the Outer Boundary Constraint together with the aroriate semanti labeling sheme [3, 6]. For simliity and reroduibility, we set all of w o, w h, w, w lf and w t to 1. We also alied the following simle geometrial onstraints: (1) two surfaes whih are tangential annot be orthogonal; (2) a viewoint-deendent juntion (3-tangent, urvature-l or hantom) annot be onave [3] if it is the rojetion of the intersetion of a lanar surfae with a urved surfae at an orthogonal edge. For eah drawing, we found, by exhaustive searh, the set of all otimal labelings. A line segment was onsidered to be orretly/ambiguously/inorretly labeled if it was assigned the orret label in all/some/none of the otimal labelings. Out of 395 orthogonality labels, 94.4% were orret, 2.3% ambiguous and 3.3% inorret. Out of 1044 lanarity labels, 95.2% were orret, 3.4% ambiguous and 1.4% inorret. Out of 229 visible faes, 96.1% were orretly lassified as loally lanar/urved, 3.5% were ambiguous and 0.4% were inorret (ambiguity tyially ourring when two regions are searated by a urved line but there is insuffiient evidene to determine whih of the two faes is lanar). If we set w = w lf = 0 (i.e. we do not try to maximize lanarity), the lanarity onstraints alone (Fig. 3) orretly identify only 77.6% of the lanarity labels and orretly lassify only 63.1% of faes as loally-lanar/urved. 10

11 X Y Figure 7: The drawing of Fig. 6(a) with lanarity labels and gradient diretions. To illustrate the strength of our onstraints, onsider the drawing in Fig. 6(a). Using the semanti labeling sheme [14], the lanarity onstraints and the gradient diretion onstraints, we obtain the labeling given in Fig. 7 (with semanti labels not shown to avoid luttering u the figure). This interretation simultaneously maximizes the number of orthogonal edges, the number of labels and the number of loally-lanar surfae athes. All surfaes are orretly labeled as lanar or urved and the diretions of the rojetions of the normals to the lanar surfaes have been orretly determined. This rovides a muh riher interretation than the semanti labeling alone. The hantom-juntion gradient-diretion onstraint (Fig. 4) allows us to reisely loate the two hantom juntions X, Y shown in Fig. 7 at whih there is a transition from an oluding label to a onvex label. To illustrate the limitations of our labeling sheme, onsider the drawing in Fig. 6(b). Using the semanti labeling sheme, the lanarity onstraints and the gradient diretion onstraints, we obtain the labeling illustrated in Fig. 8. To avoid luttering u the figure, labels, as well as semanti labels, are omitted. The line L 1, whih extends from juntion 11

12 J L 2 L 1 F K Figure 8: The drawing of Fig. 6(b) with lanarity labels and gradient diretions. J to juntion K, rovides an examle of lanarity-label transitions. Starting at juntion J, the atual lanarity label air hanges from to to. These hanges are not deteted by our onstraints, whih stritly seaking only inform us about lanarity labels in the viinity of juntions or along the length of straight line segments, with the result that fae F is inorretly lassified as loally lanar. The losed urve L 2 in Fig. 8, whih we see as a long hole running down the handle of the objet, rovides a hallenge for our onstraints, whih rodue an inorret semanti labeling of L 2. It would aear to be the skew symmetry of L 2, whose axis oinides with the axis of symmetry of the whole objet, whih allows us to identify L 2 as a hole in a lanar surfae rather than a searate objet lying on to of a larger objet. Our omlete branh-and-bound searh has worst-ase exonential time omlexity, but this is inevitable given that finding a single semanti labeling is NP-omlete [11]. 6 Conlusion We have seen that a rih labeling sheme together with simle loal geometrial onstraints allow us to obtain a onsiderable amount of information about the 3D objet deited in a 12

13 drawing. This information goes muh further than traditional semanti line labels (oluding, onvex, onave, extremal), but does not in itself rovide a omlete 3D reonstrution. For this, we an look for various other ommon features of man-made objets, suh as ubi orners, symmetry, arallel urves and isometry. Referenes [1] Cao, L., Liu, J. & Tang, X., 3D Objet retrieval using 2D line drawing and grah based relevane feedbak, MM 06, Santa Barbara, CA, USA (2006) [2] Clowes, M.B., On seeing things, Artifiial Intelligene 2 (1) (1971) [3] Cooer, M.C., Interretation of line drawings of omlex objets, Image and Vision Comuting 11 (2) (1993) [4] Cooer, M.C., Interreting line drawings of urved objets with tangential edges and surfaes, Image and Vision Comuting 15 (1997) [5] Cooer, M.C., Linear onstraints for the interretation of line drawings of urved objets, Artifiial Intelligene 119 (2000) [6] Cooer, M.C., Constraints between distant lines in the labelling of line drawings of olyhedral senes, Int. J. of Comuter Vision 73(2) (2007) [7] Ding, Y. & Young, T.Y., Comlete shae from imerfet ontour: a rule-based aroah, Comuter Vision and Image Understanding 70(2) (1998) [8] Huffman, D.A., Imossible objets as nonsense sentenes, in Mahine Intelligene 6, Meltzer, B. & Mihie, D. (eds.) Edinburgh University Press (1971) [9] Kanade, T., Reovery of the three-dimensional shae of an objet from a single view, Artifiial Intelligene 17 (1981) [10] Kanatani, K., The onstraints on images of retangular olyhedra, IEEE Trans. on Pattern Analysis and Mahine Intelligene PAMI-8 (4) (1986)

14 [11] Kirousis, L.M. & Paadimitriou, C.H., The omlexity of reognizing olyhedral senes, J. Comut. System Si. 37 (1) (1988) [12] Lison, H. & Shitalni, M., Otimisation-based reonstrution of a 3d objet from a single freehand line drawing, Comuter-Aided Design 28 (8) (1996) [13] Liu, J., Lee, Y.T., Cham, W.-K., Identifying faes in a 2D line drawing reresenting a manifold objet, IEEE Trans. PAMI 24 (12) (2002) [14] Malik, J., Interreting line drawings of urved objets, International Journal of Comuter Vision 1 (1987) [15] Marill, T., Emulating the human interretation of line drawings as three-dimensional objets, International Journal of Comuter Vision Vision 6 (2) (1991) [16] Meseguer, P., Rossi, F. & Shiex, T., Soft Constraints, in Handbook of Constraint Programming, eds. Rossi, F., van Beek, P. & Walsh, T., Elsevier (2006) [17] Perkins, D.N., Visual disrimination between retangular and nonretangular aralleleieds, Peretion and Psyhohysis 12 (5) (1972) [18] Sugihara, K. Mahine Interretation of Line Drawings, MIT Press: Cambridge, MA (1986). (Freely available on Kokihi Sugihara s website). [19] Syeda-Mahmood, T., Indexing of tehnial line drawing databases, IEEE Trans. on Pattern Analysis and Mahine Intelligene 2l(8) (1999) [20] Varley, P.A.C. & Martin, R.R., The juntion atalogue for labelling line drawings of olyhedra with tetrahedral verties, Int. J. of Shae Modelling 7 (1) (2001) [21] Varley, P.A.C., Martin, R.R. & Suzuki, H., Frontal geometry from skethes of engineering objets: is line labelling neessary, Comuter-Aided Design 37 (2005)