[151 A. K. Agrawala and T. G. Rauscher, Foundationg of Microprogramming, Architecture, Software and Applications. New York: Academic, 1976.

Transcription

1 MEE TRANSACTIONS ON COMPUTERS, VOL. C-27, NO. 9, SEPTENrBER C. [6] [7] [81 [9] [10] [11] [12] [13] [1] R. Clare, Designing Logic Sysens Using Sae Machines. New York: McGraw-HilL S. Lin, An Inroducion o Error Correcing Codes. Englewood Cliffs, NJ: Prenice-Hall, R. M. M. Obermann, Disciplines in Combinaional and Sequenial Circui Design. New York: McGraw-HilL 1970, ch. 15. V. T. Rhyne, "Serial binary o decimal and decimal o binary conversion," IEEE Trans. Compu., vol C-19, pp , Sep J. F. Couleur, 'BIDEC-A binary-o-decimal or decimal-o-binary converer," IRE Trans. Elecron. Compu., vol. EC-7, pp , Mar N. Cavlan and R. Cline, "Field-PLAs simplify logic designs," Elecron. Design, vol. 8, pp. 8-90, Sep D. E. Knuh, The Ar of Compuer Programming, vol. 1. Reading, MA: Addison-Wesley, 1969, p. 37. D. L. Diemeyer, Logic Design of Digial Sysems. Boson, MA: Allyn and Bacon, M. V. Wilkes, The Bes Way o Design an Auomaic Calculaing Machine, reprined in Compuer Design Developmen, A. Swaizlander, Ed. Rochelle Park, NJ: Hayden, S. S. Husson, Microprogramming: Principles and Pracices. Englewood Cliffs, NJ: Prenice-HalL [151 A. K. Agrawala and T. G. Rauscher, Foundaiong of Microprogramming, Archiecure, Sofware and Applicaions. New York: Academic, Mabo R. Io, for a phoograph and biography, please see p. 637 of he July issue of his TRANSACTIONS. Rober D. Cameron (S'76) was born in Edmonon, Canada, in He is currenly a graduae suden in he Deparmen of Elecrical Engineering, Universiy of Briish Columbia, Vancouver, B.C. His research ineress include logic design, compuer archiecure, and sofware engineering. Mr. Cameron is a member of he Associaion of Compuing Machinery and he ARRL. Compuer Descripion of Bodies Bounded by Quadric Surfaces from a Se of Imperfec Projecions RUTH SHAPIRA AND HERBERT FREEMAN, Absrac-This paper describes a compuer program for consrucing a descripion of solid bodies from a se of picures aken from differen vanage poins. The bodies are assumed o be bounded by faces which are planar or quadric, and o have verices formed by exacly hree faces I is assumed ha a preprocessor provides he program wih line and juncion informaion which i has exraced from he picures. The preprocessor is expeced o make misakes, Manuscrip received February 1, 1977; revised December 5, This research was suppored by he Direcorae of Mahemaical and Informaion Sciences, Air Force Office of Scienific Research, under Gran AFOSR R. Shapira is a 23 Sweden Sree, Dania, Haifa, Israel. H. Freeman is wih he Rensselaer Polyechnic Insiue, Troy, NY FELLOW, IEEE such as losing feaures or providing misinformaion abou heir naure. A echnique is presened for validaing doubful feaures as well as for maching corresponding feaures exraced from he differen picures. New grammar rules are developed for linedrawing projecions of curved and planar bodies and are used as a ool in he scene analysis process. Each picure's daa analysis is suppored dynamically by he resuls obained hus far in he oher picures' analysis. The analyzed daa from all picures are grouped ino ses, each corresponding o a single face (fla or curved), whose _'_ m _JA! mm ausio ieniuie. A U currusp[uu aiuue aneic scib afrgroupeu agaek o he differen bodies in he scene. The program wrien in PL/I has been esed successfully on several scenes. -i- 0- -lo _ A- -A AOAA A Index Terms-Arificial inelligence, compuer graphics, paern recogniion, picure grammars, scene analysis, hree-dimensional reconsrucion /78/ $00.75 (D 1978 IEEE

2 82 INTRODUCTION IN COMPUTER scene analysis a picure of a hreedimensional scene is convered ino an array of pixels, wih each assigned a number corresponding o he average grey value in ha local area of he picure. A common scheme for preprocessing he array for scene inerpreaion is o exrac lines ha are believed o correspond o he bodies' edges. To inerpre a perfec line descripion of a scene, a picure grammar mus be supplied ha is based on he common general properies of he scene's bodies. When he line descripion conains errors (as a resul ofpreprocessor limiaions), is parsing and inerpreaion becomes difficul if no impossible. In such a case undersanding of he scene can be gained by using a se of line srucures (each possibly defecive), exraced from picures describing he scene from differen vanage poins, a scheme which is also useful for obaining hree-dimensional informaion abou he bodies. The scene inerpreaion given in erms of edges (grouped firs according o faces and hen according o bodies), is mos suiable for polyhedra, bu can also be applied o curved bodies. This paper is resriced o he analysis of bodies wih planar or quadric faces. The echnique, however, could easily be modified o handle also bodies whose curved faces are- close o quadric surfaces, and his includes a big subse of man-made bodies. Many papers were published in recen years dealing wih polyhedral scene analysis, of which he papers by Robers [7], Guzman [5], Falk [3], Huffman [6], Clowes [2], and Walz [11] are represenaive of he line of research ha has been pursued. Some researchers conribued o he undersanding of he process of percepion by analyzing perfec (bu unfeasible) line drawings. Ohers used real daa (slighly edied) wih is imperfecions, bu resriced hemselves o a se ofknown bodies. Nearly all limied hemselves o a single picure of he scene. (A resricion ha led o many unnecessary difficulies.) The work by Underwood and Coaes [12] does uilize muliple picures in he analysis of scenes of a single convex polyhedron-no o overcome daa and analysis problems bu merely o "have a look" a all sides of he body. Ganapahy [] also used sereo pairs of views o reconsruc scenes conaining polyhedra. Aemping o exend he work from polyhedra o even a resriced family of curved bodies increases he difficulies o an exen which may no be apparen a firs glance. Such an effor has been described by Chien and Chang [1]. They analyzed perfec line drawings represening bodies wih planar, conic, or cylindric faces, where no inersecion ofwo curved faces was allowed. Some ineresing work in his area was also done by Turner [10] who exended Walz's mehods o he analysis of curved objecs. In his paper we shall show how a program can be made o "undersand" a se of bodies wih planar or quadric faces, uilizing defecive daa exraced from a se of muliple phoographs. No preknowledge of he bodies is assumed, bu cerain general properies of he scene-such as he exisence of precisely hree faces in every verex-are sipulaed. IEEE TRANSACTIONS ON COMPUTERS, VOL. c-27, NO. 9, SEPTEMBER 1978 BASIC DEFINITIONS AND ASSUMPTIONS Many of he following erms have been adaped from hose used by Woon [13] and Clowes [2]. A surface is eiher a quadric surface or a plane. An edge is all or par ofhe inersecion ofwo surfaces; he inersecion is bounded by one or wo oher surfaces or is closed on iself. A verex is he inersecion of hree or more edges. Aface is a porion of a surface bounded by edges or closed on iself. A boundary is a closed chain of edges bounding a face. A single face may have several boundaries. A body is a closed, conneced par of he 3D space, delimied by a finie number of faces. A scene is a se of bodies. A projecion is a cenral projecion of a 3D eniy ono he picure plane. A limb is he locus of he poins on a quadric surface ha are angen o he projecing rays and do no lie on an edge. A virual verex is he inersecion ofan edge and a limb. A region is a conneced, visible par of he projecion of a face. The projecion of a face may resul in zero, one, or more regions. A line is he projecion of an edge or a limb. The line projeced by a limb will also be called a limb when his will no cause any confusion. A line may be sraigh or curved. A juncion is he inersecion of wo or more lines, and is hus eiher he projecion of a verex or of a virual verex, or he resul of obscuring par ofan edge or a limb by a face. An objec is he se of regions, lines, and juncions corresponding o a single body. We impose he following resricions: 1) Every verex in he scene is formed by exacly hree surfaces and belongs o exacly hree edges; 2) Smooh ransiion beween wo differen faces is no allowed (i.e., he derivaive mus be disconinuous across he edge); 3) The camera posiion is assumed o be "general" (for example, he projecions of differen verices may no coincide in any picure); ) No limb passes hrough a verex. In accordance wih he above,juncions can have no more han hree lines. They can be classified ino he ypes W, Y, V, T, A, and S. An example of each ype is given in Fig. 1(a). Types Y and W mus be he projecions of verices. Type T resuls from he covering of par of an edge or a limb by a face. Types S and A are he projecions of virual verices. In a perfec line descripion V is also a projecion of a verex. The scene daa are assumed o be exraced visually from a se. of real phoographs. The daa include line informaion (as a se of poins) and juncion informaion consising of coordinaes and ypes. The daa are expeced o be (in addiion o he geomeric inaccuracies)defecive in he sense ha 1) visible lines or pars of visible lines may be missing, 2) visible juncions may be missing, 3) juncions may be repored incorrecly (V insead of Y, W, T, or S juncions). We call a juncion valid when i is a projecion of a verex. Thus all Y and W juncions are valid, all A, S, and T juncions are no valid, and he naure of he V juncions canno be decided unil more evidence is colleced (as will be explained laer). In hose cases in which a preprocessor has difficuly in disinguishing a T juncion from a fla Y or W juncion, hey can be included in he group ofhe undecided juncions.

3 SHAPIRA AND FREEMAN: BODIES BOUNDED BY QUADRIC SURFACES V Fig. 1. (a) and (b) Juncion ypes and cyclic order. CYCLIC ORDER PROPERTY As saed earlier, every verex belongs o exacly hree edges. We define for hese edges a cyclic order in he verex [8], [9], as he order induced by "walking around" he verex in a clockwise manner, and numbering he edges in he order 1 < 2 < 3 < 1. If we race ou he edges of a boundary, we consisenly change edges a he verices, eiher always in a decreasing cyclic order or always in an increasing cyclic order. Le us always choose o race edges in an increasing order. Then we shall raverse every edge in wo differen direcions as we walk around he wo faces ha share i, as illusraed in Fig. 1(b). When a verex's projecion has hree lines, heir cyclic order is deermined by he arrangemen of he lines in he picure. (The clockwise sense of he cyclic order is preserved in a projecion when he hree edges are visible.) However, when one line is invisible or missing, he cyclic order is no known. Le us define he relaion AB < AC o mean ha in juncion A he line AC follows he line AB immediaely in he cyclic order. Then if A is a wo-line valid juncion, we have eiher he relaion AB < AC or he relaion AC < AB. Knowledge of he cyclic order a a juncion is crucial o he assembly of he lines belonging o a single region. Therefore i is imporan o have means for deermining he cyclic order in a juncion when he order is no given naurally. We define a line assembly (LA) as he direced pah followed in racing ou he lines corresponding o a single boundary-or any coninuous par of i-in increasing (a) (b) cyclic order. We denoe an LA in a picure by he ordered se. of juncions Ai,, Ai,n visied in he course of he race. Now i follows from he definiion ha for every 1 < k < n we have Ai,k Ai,(k- 1) <Ai,k Ai,(k+ 1)* Two LA's are said o be disinc if hey race ou lines corresponding o wo disinc boundaries. Two LA's, Ai,1... A i,, and Aj,I... A j,m are disinc if 1) Ai,, -Ai,n and here is a leas one Ai differen from every A1, or 2) here are wo successive juncions in he wo LA's such ha Ai, -Aj,h and Ai,(k1) -Aj,(h+1), or 3) Ai,n= Aj1, and he relaion Aj,I Aj,2 <Ai,n Ai,(n- 1)* Two rules forcing a cyclic order in ajuncion can be saed now. 83 Rule 1: If we have an LA Ai,, Ain in which Ai, I_Ai,n hen we mus have in Ai,, he relaion Ai', Ai,(n 1) < A,1 Ai,2- For example in Fig. 2(a) we have he LA B, C, D, A, B. (The LA is shown as an arrowed line.) The lines in B are hus forced o have he relaion BA < BC. Rule 2: If we have wo disinc LA's, Ai,I... A i,, and Aj, 1...Ai, such ha Aj ma 1, and A (m- 1) # Ai,2 hen we mus have in A1,1 he relaion A1,1 Ai,2 < Aj,mAj,(m 1). For example in Fig. 2(b) we have he wo disinc LA's: E, F, C, B and B, A, G, E. They are disinc because of he cyclic order already esablished in juncion B by Rule 1. Now by Rule 2 he relaion EF < EG mus hold in E. (This in urn esablishes he LA M, F, E, G, N, 0.) A more complicaed sraegy may be adoped o force he correc order a a juncion when hese wo rules canno be applied direcly. One assumes a cyclic order in he juncion and ries o reach a conradicion by a sequence ofinducive seps. If a conradicion is reached, he cyclic order opposie o he one seleced is forced on he juncion. For example, in Fig. 2(c) we assume an order in juncion R such ha RQ < RN. Then we can esablish he LA's P, Q, R, N, G, A, D and D, C, F, M, P. They are disinc because of he exising naural order in D, and hence by Rule 2 force in P he relaion PQ < PM. A he same ime we have he wo LA's R, Q, 0 and 0, N, R which are disinc due o he assumed order in R. By Rule 2 he relaion ON < OQ is forced in 0. This in urn esablishes he LA M, F, E, G, N, 0, Q, P, M. Now by Rule 1 we mus have in M he relaion MP < MF, which conradics he naural order in M. This sequence of deducions was based -on he assumpion ha he relaion RQ < RN exiss in R. The conradicion reached les us conclude ha he opposie relaion, namely RN < RQ, mus exis in R. This esablishes he LA 0, N, R, Q,O and, by Rule 1, he relaion OQ < ON in 0. We have hus been able o find he cyclic order in of he original 5 unordered juncions. Juncion P remains unordered. The final ordering of he juncions is given in Fig. 2(d). A byproduc of he foregoing is a new class of impossible

4 8 IEEE TRANSACTIONS ON COMPUTERS, VOL. c-27, NO. 9, SEPTEMBER 1978 Fig. 3. The mach line. Fig. 2. Forced cyclic order. objecs in addiion o hose described by Huffman [6]. The objecs can be idenified as being impossible ifheir srucure is such as o lead o conradicory cyclic order a heir juncions [8], [9]. JUNCTION MATCHING To uilize he daa from all picures, we mus know he parameers ha relae each picure's coordinaes o hose of he oher picures and also o he universal coordinae sysem in which he final descripion should be given. These parameers are difficul o measure. Some of hem are measured indirecly and some are calculaed by minimizing he sum-of-squares differences beween he measured and calculaed projecions of a se of predeermined poins. A deailed descripion of he parameer deerminaion is given in [8]. In order o arrive a a rue descripion of he 3D scene in spie of he daa imperfecions, we compare he line srucures from he given picures (hree in our case) agains each oher and use he informaion found in one picure o verify he informaion found in anoher. Le us sar wih juncions. We have wo goals. One is o deermine hose V juncions for which here is enough evidence o assume ha hey are valid juncions. The second is o group, from he differen picures,juncions ha are he projecions of he same verex. From here on in his secion, whenever we use he erm "juncion," we shall mean a V, Y, or W juncion. If wo juncions in wo differen picures are projecions of he same verex, hey mus obey cerain geomeric rules. Le Pi, C,, and Ai,j be he picure plane, he cener ofprojecion, and he projecion of verex J, respecively, for picure i. Then for wo picures, i and j, C,, Cj, J, A,,J, and A J, are coplanar, and he lines Ci Cp, Ci Ai,J, and CjAj,, inersec any plane P. in hree colinear poins: C ijq, J ip and Jjq, respecively (Fig. 3). We shall call he line formed by hese poins a mach line. Thus, if we reprojec ino he plane Pq juncion Ai,j from Ci, juncion Aj,j from Cj, and C, from Cj, we can check wheher his condiion is saisfied. Two juncions from differen picures ha saisfy his condiion are said o be machable. Obviously, if a verex I is in he plane of Ci, Cj, and J, hen Ai,, and Aj,, are machable, and so are.ai,j and Aj,,. To resolve such ambiguiies, we refer o he hird picure, picure k. We require ha C,, Cj, Ck be no colinear. Then if Ai,J, Aj,J, and Ak,J are he hree projecions of verex J, he hree differen mach lines on plane P.form verices are Ji,s a riangle whose Jq,, and Jkq (Fig. ). We call he se of hree juncions, from differen picures, ha are machable in pairs, a riple. By forming riples we can eliminae many pairs ha are machable bu are no projecions of he same verex. However, even in he pure geomeric case we may have hree projecions of hree differen verices ha form a riple [8]. We are dealing here wih daa exraced from acual phoographs; hus we mus accep Ai,j and Aj,j as machable even if Jiq is disan from he mach line Cij,Jjq by an amoun less han some hreshold. Relaxing he colineariy requiremen on he mach lines increases he ambiguiy of maching. The larger he hreshold disance, he greaer he num-ber of ambiguous cases. Two machable juncions ha are he projecions of he same verex are said o mach each oher. A riple in which

5 SHAPIRA AND FREEMAN: BODIES BOUNDED BY QUADRIC SURFACES Fig.. The mach-line riangle for verex J. (Mach lines shown bold.) he juncions mach each oher is called a mach riple. To esablish maches, we find for every juncion all machable juncions in he wo oher picures. Then we form all he riples. To find he mach riples in he se of all riples, we mus use some picure conex. The kind of conex we shall use is line connecions beween juncions. If we have hree lines-each in a differen picure-whose ends form wo riples, he wo riples are each considered o be a mach riple. (The chance ofnaming a wrong riple as a mach riple sill exiss bu is now grealy reduced.) Since i is possible ha a verex's projecion will be missing in one or more picures, i is desirable o esablish maches beween machable pairs. Naurally, more conex mus be used o esablish a mach on he basis of only a wo-picure comparison. The pair mach is done in wo passes and only afer he riple mach phase is compleed. (I reduces he number of free candidaes and, herefore, he chance of error.) We selec a se of hree juncions in which one juncion-call i he middle-is conneced by lines o he oher wo and which is machable o a se of hree juncions conneced (in he same way) in anoher picure. In he firs pass we require ha he middlejuncion be ofype Y or W and ha he wo lines have he same cyclic precedence in he wo juncions. The hreejuncions are hen assumed o mach he hree juncions in he oher picure. In he second pass we allow one or boh of he middle juncions o be of ype V and do no check he cyclic precedence (i may be unknown for he Vjuncion). The second pass is a lile more hazardous since we are unable o use cyclic precedence, bu since i is carried ou afer mos juncions are already mached, here is only a small chance of making an error. If one juncion in a riple is mached o he oher wo by wo applicaions of a pairwise mach, we consider he riple o be a mach riple; ha is, we consider he remaining wo as maching also. Every Vjuncion ha is mached o ajuncion in anoher picure is marked as valid. The mach supplies addiional 85 evidence ha he juncion is a projecion of a verex. The validaed V juncions are assigned a cyclic order, when possible, by applying he cyclic order rules. In general, he need for conex suppor in he maching procedure decreases as more picures are used. Three picures of a scene is a good choice bu four or more picures may be even beer. LINE MATCHING AND DATA RECOVERY The maching of lines in differen picures poses difficulies because of daa imperfecions and because he number of lines beween wojuncions can vary from zero o hree. Maching is done for wo picures a a ime for he lines erminaing a mached juncions. There are hree cases o be considered: 1) The wo mached juncions are boh 3-line juncions, 2) one is a 3-line and he oher a 2-line juncion, and 3) boh juncions are 2-line juncions. Each case can be furher divided ino subcases. The scheme for line maching in each of he differen cases is shown in Fig. 5. The erm naural exension employed in his figure is defined as follows. Suppose ha we are given a juncion I and a line d wih ends J and K, where only J is a valid juncion. Now le us fi a sraigh line or a conic secion o he poins of d and I by using a minimum mean-square error echnique. If, for a sraigh-line fi, his minimum error is less han some hreshold, he fi is plausible. Oherwise a nondegenerae conic is considered if such a conic can be fied. Any fi found is rejeced if 1) he disance beween I and K is greaer han he disance beween J and I, 2) i is an ellipse and one of he axes is oo small, 3) i is an ellipse and he par of he ellipse beween K and I is greaer han a quarer of he ellipse, ) i is a hyperbola and I and he poins of d belong o wo differen branches. When a fi is found and no rejeced, juncion I is said o lie on he naural exension of d. When he wo juncions around which a mach of lines is aemped are boh cyclically ordered, a mach ofone pair of lines yields in urn he corresponding mach of he successors in he cyclic order. Also wo lines in wo picures found o mach o he same line in a hird picure are declared o mach each oher. When wo lines of a cyclically ordered juncion are mached o wo lines of an unordered juncion, he second juncion can be ordered because is cyclic order mus agree wih ha of he juncion o which i is mached. The line maching procedure makes i possible o deec missing connecions beween juncions. Two disinc siuaions are encounered: 1) juncion J should be conneced hrough is line d o juncion I, as can be clearly deduced for he configuraions of Fig. 5(c)II, (g)-(j), (m), (n), and (s)-(u), and 2) juncion J should be conneced o juncion I bu he line of connecion is missing in J, as is he case for he configuraions of Fig. 5(o), (r), (s), (v), and (w). Table I liss he appropriae acions o be aken, depending on he siuaion injuncion I. The lines around he juncions are assigned indices ha agree wih he cyclic order in ordered juncions, or reassigned o agree wih i in hose cases in which he cyclic order is deermined laer. A juncion is said

6 .~~~~~~~~~~~~~ i 00 as a, 00 (k) (1) k~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Picure i Picure J oin) g =(n) F8/-<~~~~~~~ JJ / (q) (s) SJ IJ G= =ai~~~~~(p (r) L ~~~~~~~~~~~~~~~~~I i <, +,> *I.-* Ii 0 z 09 vvalid juncion *-+ relaes Juncions h'a are already mached. Picure i Picure J z 0 naural exension. vcalid juncimi ha is alread.r cyclica]ly ordered. <_-~ naach is esablished beweeni he lines Legend (v) i~~~~~~~~~~~~~~ 0 0 Ji I~~~" oo Fig. 5. Differen configuraions for line maching.

7 SHAPIRA AND FREEMAN: BODIES BOUNDED BY QUADRIC SURFACES 87 TABLE I Sae of juncion I Juncion J has o be conneced No line in index r hrough d indexed r of juncion J Exend line d o juncion I, Creae an "empy" line s r or s give i index in I. I becomes beween I and J, give i a 3L juncion linked in 3 indices. index in I and r in J. J is linked in index r. Number of lines in I and I is a 2L juncion J increases by 1. J is linked in r and s. linked in r, I in. ui EI The line associaed wih index Exend he line associaed in I is combined wih d. I be- wih o J. Associae i s \ or s r comes linked in 3 indices. J is wih index r in J. I belinked in index r. comes linked in 3 indices. Number of lines in J increase l is a 31 juncion by 1. J linked in r. linked in r and s. One of he wo lines associaed Ou of he wo lines assoc r wih r and, he one wih he iaed wih r and ake he Ig poins of smaller average error one ha has naural exens wih respec o he naural ex- sion o J, if here is any. ension of d, is combined wih d. Exend he line o J. No. I is linked in ha index. J of lines in J increases by 1. I is a 3L juncion linked in r. J becomes linked in r, and I linked in index s. in 2 indices. Exend line d o I and give i Creae an "empy" line be-,fli index or r. I becomes a 2L ween I and J. Give i index juncion linked in 2 indices. r in J and index r or in I. J is linked in index r. I becomes a 2L juncion linked in 2 indices. No. of lines I is a 11 juncion in J increases by 1. J linked linked in index s. in r. If he average error of he If J is on he naural exenf ;~ ~ he poins of he line assoc- sion of he line associaed iaed wih, wih respec o wih, exend he line o J. I he naural exension of d, is I becomes linked in 2 indices less han some hreshold, com- No. of lines in J increases I is a 2L juncion bine he line wih d. I becomes by 1. J is linked in r. linked in index r. linked in and J in r. Oher No Acion No Acion o be linked in is index if he corresponding line has a valid juncion on is oher end. An "empy" line is a line whose only known poins are is end juncions and whose form has no ye been deermined. The imporance of filling in missing connecions o a juncion, in addiion o providing a beer undersanding of he scene, lies in he fac ha i improves he maching abiliy around he juncion and faciliaes furher applicaion ofhe rules for cyclic ordering. Juncion recovery is also par of he daa correcion process. For example, he abiliy o conver configuraion Fig. 5(1) ino configuraion Fig. 5(m) leads o an addiional line maching and in urn provides he basis for furher recoveries. Consider he subconfiguraion of Fig. 5(1), consising ofjuncions Ji, Jj, Ij and lines di and Ij Jj. Given ha Ii has a mach Ik in picure k bu no mach in picure i, here are wo possibiliies: 1) There exiss a riple (Ii, Ij, IJ) and Ii is no ye mached; hen if Ii lies on he naural exension of di, he riple is declared a mach riple. 2) No such riple exiss; a juncion Ii is synhesized by aking a poin in 3D space ha has he minimal sum of absolue disances from he lines Cj Ij and Ck Ik and projecing i from C, on o Pi. This poin is declared he mach of Ij and Ik* If Ii lies on he naural exension of di, i is declared a 1-line juncion. The sequence of line maching, daa recovery acions, and cyclic ordering ofjuncions is repeaed over and over again unil finally a sae is reached a which no new resuls are obained. Noe ha he line maching echnique does no apply o lines wih end juncions differen from hose described, i.e., closed lines, lines wih wo ype-a end

8 88 EEE TRANSACTIONS ON COMPUTERS, VOL. c-27, NO. 9, SEPTEMBER 1978 (a) Fig. 6. Maching lines by range. (b) juncions, ec. For hese, addiional ools for line maching are needed. Thus we define for a line gi (one ha is no a projecion of a limb) lying in picure i, range -limis wih respec o picure j. These are wo sraigh lines ha have poins in common wih gi and pass hrough Cij wih he maximum angle beween hem. (Lines a and b in Fig. 6.) Taking wo picures i andj, we creae in each picure, for all qualified lines, pairs of range limis wih respec o he oher picure. If we now projec boh picures on a plane Pq we obain all he range limis from boh picures as a se oflines going hrough C,jq. The range limis of gi when projeced on Pq form some angle a, and hose ofhjform some angle #, wih y being he overlap angle (Fig. 6). When y/(x + #) is greaer han some hreshold, g9 and hj are said o be machable by range. When hree lines gi, hj, andfk, in hree picures, are pairwise machable by range, hen hey are said o be mached by range. This echnique makes i possible o mach lines which could no be mached by he regular line maching echnique [8]. OBJECT FORMATION The collecion of lines from all picures ha correspond o he boundaries of a single face will be referred o as aface group. Clearly, every line ha is no a limb belongs o wo face groups. We build a face group by selecing a line, in any picure, ha does no ye belong o wo face groups. We hen follow he LA in a direcion no previously followed, adding o he group every line raced and is maches in he oher picures. If he race reurns us back o he line from which we sared, he race is erminaed-and his face-group generaion is complee, assuming ha he face has only one boundary. Oherwise when we reach a line for which no nex line in he LA is available, we check wheher an LA race can be coninued from he line's mach in anoher picure. The face-group generaion may hus proceed by "jumping" from one picure o anoher. If he process is blocked in all Fig 7. N (c) (a)-(c) Three configuraions for second-level daa recovery. picures, we reurn o he saring poin and ry o follow he LA in he opposie direcion, again "jumping" from one picure o anoher if necessary, unil he process is blocked again in all picures. All S and A juncions encounered in he race are remembered. When he process erminaes, he limbs a hese juncions are checked and ifa limb's oher end juncion belongs o a line no ye in he face group (implying ha anoher boundary of he same face has been deeced), he race resumes here. The limb lines are also added o he face group. When all lines have been raced wice, furher daa recovery acion is called for. This second-level daa recovery can be carried ou for he following hree configuraions, illusraed in Fig. 7. 1) Two lines I and k in he same picure, wih a valid juncion in only one end of each, have been assembled ino wo face groups M and N, k, 1 e M and k, 1 e N. Acion: Combine 1 and k ino one line. (See Fig. 7(a).) 2) A line lwih one valid end juncion has been assembled ino wo face groups M and N, and here is in he same picure a 2-line valid juncion J wih lines m and n, m E M and n E N. Acion: Exend he line I o juncion J. (See Fig. 7(b).) 3) There are wo 2-line validjuncions Iand J in he same picure wih lines k, 1 and h, g, respecively, where k and h

9 89 SHAPIRA AND FREEMAN: BODIES BOUNDED BY QUADRIC SURFACES have been assembled ino face group M, and I and g have been assembled ino face group N. Acion: Creae an "empy" line beween he wo juncions I and J. (See Fig. 7(c).) These daa recovery acions uilize he propery ha wo componens assembled simulaneously ino wo face groups mus correspond o he inersecion of he associaed faces. There is, of course, he possibiliy ha he inersecion is secioned ino several separae edges and ha he wo componens correspond o wo differen secions; bu a his sage of he recogniion phase, wih no evidence o he conrary, we assume ha hey belong o he same secion. Afer any second-level daa recovery acions have been performed, he process of maching lines is ried again, using he newly gained informaion. Then he possibiliy for second-level daa recovery is invesigaed again, and so on unil no furher line maching is possible. The face groups are updaed each ime new resuls are obained. Each face group corresponds o a face of one ofhe bodies in he scene. Afer he face-group assembly and second-level daa recovery is compleed, we are ready for he descripion of he objecs. Each objec is a se of face groups, wih no common elemens beween wo ses. Every face group mus belong o some se. A new objec is formed by aking a face group ha is as ye unassigned and recursively adding o i every unassigned face group ha has a line in common wih any of he face groups already in he se. We also wish o deermine hose face groups ha correspond o curved faces. Clearly, every curved face mus manifes iself somehow, or we have no way for recognizing i as curved. We know ha whenever wo face groups share, in a leas one picure, a curved line, a leas one ofhem mus correspond o a curved face. I is also obvious ha every face group ha conains a limb mus correspond o a curved face. Thus o deermine he curved face, we build, by means of a ree search, a minimal se of faces conaining iniially all hose faces whose face groups have a limb, such ha a leas one parner (acually is corresponding face), from each pair sharing a curved line, is presen in i. A procedure for exracing informaion abou he naure of he faces and for geing he faces' equaions is described in [8]. (a) (b) (c) Fig. 8. (a)-(c) Three phoographs of scene 1. Noe ha juncions (9,; 10J), (272; 282), (252; 262) were no validaed nor were juncions (12; 152), (12; 162), (63; 73), and (173; 183). An example of a riple ha was formed bu no idenified as a mach riple is ((61; 71), (102; 112), (313; 323; 333)) as well as he riple ((251; 281; 291), (292; 302; 32), (313; 323; Neverheless (25,; 28,; 29J) and (292; 302; 32) are 333)). EXPERIMENTAL RESULTS pairwise mached in he second pass. In Fig. 8(a)-(c) hree picures of a scene are shown. Each The juncions ha were cyclically ordered in he firs shows he scene from a differen vanage poin. Fig. 9(a)-(c) applicaion of Rules 1 and 2 are (151; 161), (131; 11), (291; gives a schemaic descripion' of he picures. 301), (102; 112), (122; 132), and (312; 32). Through he line In he juncion maching phase he following Vjuncions maching process and he new condiions creaed in he line were validaed (he juncions are denoed by he pairs of maching process, more juncions were cyclically ordered: lines ha form hem, wih he subscrip indicaing he (11; 1), (31; 81), (1; 61), (61; 71), (21; 91), (32; 2), (52; picure number): 62), (12; 62), (12; 22), (82; 92), (83; 93), (3; 53), (23; 33), (11; J), (3,; 8J), (1; 61), (61; 71), (151; 161), (21; 91), (23; 93), (153; 163), (193; 203), (203 ;213), (213; 223), and (131; 1J) (291; 301), (191; 201), (102; 112), (122; 132), (233; 23). (162; 172), (32; 2), (52; 62), (12; 62), (12; 22), (82; 92), The daa recovery acions done hrough he "image (312; 32), (83; 93), (3; 53), (23; 93), (23; 33), (153; 163), process (Fig. 10(a)-(c)) resuled in connecundersanding" (183; 193), (193; 203), (203; 213), (213; 223), (233; 23). ing juncions (1; 6I) and (31; 81) by an "empy" line, and connecing juncions (12; 62) and (82; 92), juncions (23; 93) ' The wo nonlimb lines of an S juncion, which are acually wo pars of he same line, are denoed differenly in he inpu daa, bu are merged and (3; 53), and juncions (203; 213) and (233; 23). Line251 laer ino a single line named afer one of hem. was exended o juncion (291; 301), and line 292 was

10 850 IEEE TRANSACTIONS ON COMPUTERS, VOL. c-27, NO. 9, SEPTMBER 1978 Li X I ((s)a) 92BII (a) 1 (a) (a) LI 'I I6 25 i8 ±6 Is (b) (b) 33 zs 30 (c) Fig. 9. (a)-(c) Schemaic descripion of he phoographs of Fig. 8. exended o juncion (312; 32). A synheic juncion was generaed in Fig. 10(a) o which lines 71, 81, and 91 were exended. This juncion was also cyclically ordered. The final descripion repored by he program is (lines are arranged in riples of mached lines): Body 1: Face grbup {(331, 262, 283). Limbs: 31, 282, 263} labeled curved, Face group {(331, 262, 283)}. Body 2: Face group {(311,22, 103); (231,212, 133). Limbs: 221,211, 232, 252, 13, 113} labeled curved. Face group {(311, 22, 103)}. Face group {(231, 212, 133)}. Body 3: Face group {(301, 312, -); (251, 292,-). Limbs: 27,, 3321 labeled curved. Face group {(301, 312,-); (291, 32,-)} Face group {(291, 32, (251, 292,- Body : Face group {(201, -, 193); (131, 132,-); (-, 12, -)}. Face group {(171, 192, 213); (11, 202,-); (131, 132,); (121, 122, 203)}, (c) Fig. 10. (a)-(c) Resuls of he analysis of scene 1. (The circled juncions are cyclically arranged. The arrows poin a daa recovery resuls. The dashed lines indicae "empy lines," for which only he end poins are known.) Face group {(171, 192, 213); (111, 112, 33); (161, 102, 233); (151, 182, 223)}. Face group {(161, 102,233); (101, -, 23); (-,-163); 172, 153)}. Face group {(151, 182, 223); (-, 172, 153); (11, 202,-); (-, 12,-)}. Face group {(121, 122, 203); (201, -, 193); (111, 112, 33); (101, 23); (-, -, 173)}. Body 5: Face group {(91, 2, 83); (71, 52, -); (-, -, 63)}. Face group {(81, 92, 93); (351, 352, 23); (61, 62, -); (71, 52, Face group {(81, 92, 93); (91, 2, 83); (21, 32, 53); (31, 82, 353)}. Face group {(61, 62, -); (k, 12, 33); (- 9-73)}. Face group {(1, 12, 33); (351, 352, 23); (31, 82, 353); (11, 22, 3)}. Face group {(21, 32,53); (-,-,63); (1, 22,3); (-,-, 73)}- The second example consiss of he scene shown in Fig. 1 1(a)-(c). The inpu daa for his scene are shown in Figs The resuls of he daa recovery process are shown in Fig. 15(a)-(c). The final descripion repored by he program is as follows.

11 851 SHAPIRA AND FREEMAN: BODIES BOUNDED BY QUADRIC SURFACES (a) (b) (c) Fig. 11. (a)-(c) Three phoographs of scene 2. ~ o +f j ^ 1*7 *+., * +2 ', 3q.+ 2 *, " A, - +. Fig. 12. Scene-2 inpu daa: Fig. 11(a). Body 1: Face group {(571, 52, 573); (561, 622, 83); (551,I -)2 Face group {(61, 92, 533); (651, 522, 563); (531,22, -); labeled curved. (601, 502, ); (591, 512, 553); (561, 622, 83); {(551, 2, -); (51, 3-2 Face group - Face group (51, 3,)}* {(531, 22,3-); (52k, 53,)}. Face group {(61, 92, 533); (521, 532, -); (601, 502, Body 2: (581, 72, 503). Limbs: 611, 82, 513} labeled curved. Facegroup{(501, 52, 313). Limbs: 511, labeled curved. Face group {(591, 512, 553); (571, 52, 573); (651,522,563); Face group {(501, 52, 313)}. (581, 72, 503)}. 562, 572, 303}

12 852 IEEE TRANSACTIONS ON COMPUTERS, VOL. c-27, NO. 9, SEPTEMBER * +~~~~ +05 < 2 1 * (9 ++ -r $ + 17 is *.&~~~~ * P + * 2a3.. *1 * + f + 'o sq + If *.r *k + +I ' r ~~'.+, I 31 6 ib' +0 :21 $6 * r r 'r 2&.33 X53. & + +,~ ~~5 Is..b XG s * * "1~~~~~~~~~~~~1 +:6. 5lq * ~ * 2 ir # L&.. ~ r* +* * Fig. 13. Scene-2 inpu daa: Fig. 11(b). * *sy 5 v+*'* as* ~~~~~~~~~1 *. 3 3 S' * Pb~~~~~~~~1.. 3Y 7 + * B i if + *6,+. fd s v. 1. sq,~~~1 T '.1e F Fig. 1. Scene-2 inpu daa: Fig. 11(c). Body 3: Face group {(91, 282, 163); (71,-, 183); (371, 332, 23); (11, -, 233); (01,-, 203). Limbs: 35k, 361, 292, 63, 1731 labeled curved. Face group {(91, 282, 163)). Face group {(81, 322, 603); (51, 12, 613); (71, -, 183); (6i, 02, 593)). Face group {(8,, 322, 603); (391, 352, 263); (21,362,273); (381, 32, 253)) labeled curved. Face group {(61, 02, 593); (01,-, 203); (31, 392, 283); (391, 352, 263)). Face group {(51, 12, 613); (381, 32, 253); (1,382,293); (37D, 332, 23)1. Face group {(1, 382, 293); (21, 362, 273); (31, 392,283); (11, -, 233))- Body : Face group {(331,222, 333); (291, 192, 363). Limbs: 321,311, 202, 212, 33, 353) labeled curved. Face group {(331, 222, 333)). Face group {(291, 192, 363)).

13 SHAPIRA AND FREEMAN: BODIES BOUNDED BY QUADRIC SURFACES 853 '7 (a) (b) (c) Fig. 15. (a)-(c) Resuls of analysis of scene 2. (For explanaions of markings see capion o Fig. 10.) Body 5: Face group {(281, 232, 383)) labeled curved. Face group {(281, 232, 383)}.2 Body 6: Face group {(21, -, 53); (221, -, 653); (211, 602, 583); (191, -,33); (181, 252, 13). Limbs: 261, 271, 393, 03) labeled curved. Face group {(21,-, 53); (231,-, 63); (211, 602, 583); (201, -,73)} Face group {(231,-, 63); (221,-, 653)). Face group {(201,-, 73); (191,-, 33)). Face group {(181, 252, 13)). Body 7: Face group {(11, 12, 53); (21, 152, 73); (-, 162, ); (-, 172, -)} Face group {(11, 12, 53); (-, 132,-); (661, 102, 63); (13k, 122, -); (15, I-,) Face group {(131, 122, -); (16,-, -);(121, -,23); (61, 72, -;( ); (1 II, I1 25- )1- Face group {(121,-, 23); (51, 32, 93); (-, 22,-); (-, 1, - ); (,162, - )}. Face group {(111, 112, -); (101, 652, -); (-,92,-); (-, 12, -); (21, 152, 73); (661, 102, 63)). Face group {(101, 652, 62, -);(671,2, -);(8152, Face group {(81, 52,-) (1, 82, 153); (-, 22,-); (-, 92, Face group {(6,, 72, - (51, 32, 93); (1, 82, 153); (671, 2, 2 For he case of a one-face-body (an ellipsoid for example) no limb informaion is given in he inpu daa (no S or A juncions). The program inerpres i as half an ellipsoid. Of paricular ineres in his example are he curved-face labelings in Body 1 and Body 3. CONCLUSION The objecive of he research described here was o invesigae echniques for he compuer undersanding of picures of scenes. Three principles guided he research: 1) No complee preknowledge of he scene's bodies should be assumed since ha would limi he reperoire of bodies. 2) Insead he number of bodies should be unresriced and merely some properies of he bodies should be prespecified; he properies should be sufficienly general o permi inclusion of all bodies likely o be encounered. 3) Since he inpu is o be goen from phoographs, he presence of a limied amoun of daa imperfecions such as missing -lines and juncions, wrong juncions, and geomeric inacrquracies should be permied. To achieve he objecive i was assumed ha a se of muliple phoographs of he scene, aken from differen vanage poins, would be available and would provide he (possibly defecive) inpu daa. A se of new grammar rules was formalized. The rules apply o boh planar-faced and curved-surface bodies and faciliae he analysis of he picures. A procedure was devised for esablishing maches beween juncions in he differen picures and deermining he validiy of doubful juncions. The procedure is heurisic; i firs esablishes maches among juncions for which he evidence of a mach is sronges and hen progressively uses he informaion gained a each sep o esablish maches for juncions for which he mach evidence is weaker. I includes also a process of line maching, filling in missing connecions beween juncions and missing juncions and cyclic ordering ofjuncions o verify he arrangemen of he lines enering hem. This process is ieraed unil no furher resuls can be achieved. This is hen followed by a procedure which assembles he analyzed daa ino ses, each describing

14 85 5.EEE TRANSACTIONS ON COMPUTERS, VOL. c-27, NO. 9, SEPTEMBER 1978 single body of he scene. The procedure deermines he edges bounding each face and esablishes wheher he faces are planar or curved. A compuer program based on his approach was wrien. The program was able, given real inpu daa, o "undersand" he phoographed scene and o yield a plausible descripion ofi. Alhough he procedure is heurisic and can,. of course, no guaranee a correc scene inerpreaion, good performance has been achieved for phoographs of scenes conaining a moderae, realisic Universiy. Technion. amoun of imperfec daa. a REFERENCES [1] R. T. Chien and Y. H. Chang, "Recogniion of curved objecs and objecs.assembly," in Proc. 2nd In. Join Conf Paern Recogniion, IEEE Publ. 7 CH0885-C, Copenhagen, pp , Aug [2] M. B. Clowes, "On seeing hings," Arificial Inelligence, vol. 2, no. 1, pp , [3] G. Falk, "Compuer inerpreaion of imperfec line daa as a hreedimensional scene," Ph.D. disseraion, AD , Dep. Compu. Sci., Sanford Univ., Sanford, CA, Aug [] S. Ganapahy, "Reconsrucion of scenes conaining polyhedra from sereo pairs of views," Al Memo 272, Arificial Inelligence Laboraory, Sanford Univ., Sanford, CA, Dec [5] A. Guzman, "Compuer recogniion of hree-dimensional objecs in a visual scene," Ph.D. disseraion, AD , MIT, Cambridge, MA, Dec [6] D. A. Huffman, "Impossible objecs as nonsense senences," in Machine Inelligence, vol. 6, B. Melzer and B. Michie, Ed. New York: American Elsevier, 1971, pp [7] L. G. Robers, "Machine percepion of hree-dimensional solids," in Opical and Elecroopical Informaion Processing, J. T. Tippe e al., Ed. Cambridge, MA: MIT Press, 1965, pp [8] R. Shapira, "Compuer reconsrucion of bodies bounded by quadric surfaces from a se of imperfec projecions," Ph.D. disseraion, AD-A , NYU, New York, NY, Sep [9] R. Shapira and H. Freeman, "A cyclic-order propery of bodies wih hree-face verices," IEEE Trans. Compu., vol. C-26, pp , Oc [10] K. J. Turner, "Compuer percepion of curved objecs using a elevision camera," Ph.D. disseraion, School of Arificial Inelligence, Edinburgh Univ., Edinburgh, Scoland, 197. [11] D. A. Walz, "Generaing semanic descripion from drawings of scenes wih shadows," Ph.D. disseraion, AD , Arificial Inelligence Lab., MIT, Cambridge, MA, Nov [12] S. A. Underwood and C. L. Coaes, Jr., "Visual learning from muliple views," IEEE Trans. Compu., vol C-2, pp , June [13] P. Woon, "A compuer procedure for generaing visible line drawings for golids bounded by quadric surfaces," Ph.D. disseraion, AD 72 7, Dep. Elec. Eng., New York Universiy, New York, NY, Dec Ruh Shapira was born in Tel-Aviv, Israel, in 190. She received he B.Sc. and M.Sc. degrees in mahemaics from he Technion, Israel Insiue of Technology, Haifa, Israel, in 1962 and 196, l respecively, and he Ph.D. degree in compuer science from New York Universiy, New York, NY, i From 1963 o 1970 she was on he Mahemaics Faculy of he Technion, and from 1971 o 197 wih he Deparmen of Elecrical Engineerng and Compuer Science, New York From 197 o 1976 she pursued research work a he Herber Freeman (A'7-M'52-SM'5-F'67) received he B.S.E.E. degree from Union College, Schenecady, NY, in 196, and he M.S. and Dr.Eng.Sc. degrees in elecrical engineering from Columbia Universiy, New York, NY, in 198 Hed156, respecively. H joined he Sperry Gyroscope Company, Grea Neck, NY, in June 198, as a Projec Engineer. In he early 1950's he designed Sperry's firs digial compuer, he SPEEDAC. In February 1957 he was promoed o Head of he Advanced Sudies Deparmen, and in December 1959 o Head of he Miliary Daa Processing Deparmen. In Sepember 1960 he joined he Deparmen of Elecrical Engineering, New York Universiy, New York, NY, and served as Chairman of he deparmen from 1968 o Since 1975 he has been Professor of Compuer Engineering a Rensselaer Polyechnic Insiue, Troy, NY. He was a Visiing Professor a MIT ( ), he Swiss Federal Insiue of Technology in Zurich (1966), and he Compuer Science Research Insiue in Pisa, Ialy (1973). He is he recipien of a NSF Senior Pos Docoral Fellowship (1966) and a Guggenheim Fellowship (1973). He is he auhor of over 50 echnical papers and he book Discree Time Sysems, and currenly also serves as Edior of he inernaional journal Compuer Graphics and Image Processing, and Associae Edior of he journals Calcolo and Paern Recogniion. Dr. Freeman is a Regisered Professional Engineer in he Sae of New York and member of Associaion for Compuing Machinery, Sigma Xi, he Sociey for Informaion Display, he Paern Recogniion Sociey, and he New York Academy of Sciences. He has been acive in he Inernaional Federaion for Informaion Processing, was Programme Chairman for IFIP Congress 7, held in Sockholm in Augus 197, and has received he IFIP Silver Core Award. Since 1976 he has been Treasurer of he Inernaional Associaion for Paern Recogniion. He was a General Chairman for he IEEE Compuer Sociey Conference on Paern Recogniion and Image Processing (1977), and is currenly Chairman of he IEEE Compuer Sociey Commiee on Machine Inelligence and Paern Analysis.