The official electronic file of this thesis or dissertation is maintained by the University Libraries on behalf of The Graduate School at Stony Brook

Transcription

1 Stony Brook University The offiial eletroni file of this thesis or dissertation is maintained by the University Libraries on behalf of The Graduate Shool at Stony Brook University. Alll Rigghht tss Reesseerrvveedd bbyy Auut thhoorr..

2 3D Shae Measurement Based on the Phase Shifting and Stereovision Methods A Dissertation Presented by Xu Han to The Graduate Shool in Partial fulfillment of the Requirements for the Degree of Dotor of Philosohy in Mehanial Engineering Stony Brook University August 2010

3 Coyright by Xu Han 2010

4 Stony Brook University The Graduate Shool Xu Han We, the dissertation Committee for the above andidate for the Dotor of Philosohy degree, hereby reommend aetane of this dissertation. Dr. Peisen Huang, Advisor Professor, Deartment of Mehanial Engineering Dr. Jeffrey Q. Ge, Chairman Professor, Deartment of Mehanial Engineering Dr. Yu Zhou, Member Assistant Professor, Deartment of Mehanial Engineering Dr. Hong Qin, Outside Member Professor, Deartment of Comuter Siene This dissertation is aeted by the Graduate Shool. Lawrene Martin Dean of the Graduate Shool ii

5 Abstrat of the Dissertation 3D Shae Measurement Based on the Phase shifting and Stereovision Methods By Xu Han Dotor of Philosohy in Mehanial Engineering Stony Brook University 2010 Strutured light systems have been used in inreasingly more aliations for 3D shae measurement due to their fast measurement seed, good auray, non-ontat harateristi, and ortability. This dissertation is foused on imroving the erformane of the 3D shae measurement systems based on digital fringe rojetion, hase shifting and stereovision tehniques. New amera and rojetor models and alibration algorithms as well as a novel system design based on a ombined hase shifting and stereovision method are introdued. The first art of this dissertation introdues systems based on digital fringe rojetion and hase shifting tehniques. In this researh, a olor fringe attern is generated by software and rojeted onto the objet being measured by a digital-lightroessing (DLP) rojetor working in the blak and white (B/W) mode. The fringe images are atured by a high seed CCD amera, whih is synhronized with the iii

6 rojetor by software. The 3D model is reonstruted by using every three onseutive fringe images. The reviously develoed linear alibration method does not take lens distortion into onsideration and as a result, has very limited measurement auray. In this researh, the effet of lens distortion on both the amera and rojetor is modeled based on areful alibration. Radial and tangential distortion arameters of different orders are analyzed and the right ombination of arameters is hosen to rovide an otimal erformane. Exerimental results show that the measurement auray has been imroved by more than 75 erent (the RMS from 1.4 mm to 0.35 mm) after the imlementation of the roosed nonlinear alibration method. The roosed nonlinear alibration method is imlemented in the real-time system to ahieve higher auray. To enhane the erformane of the system, a new real-time system is designed and exerimented, whih ahieves a maximum seed of 60 Hz. A quality ma guided hase unwraing algorithm is develoed as well to address the hase ambiguity roblem of the revious hase unwraing algorithm. As a result, hase unwraing errors aused by disontinuous features are eliminated in most ases, thus signifiantly enhane the reliability of the system. In the seond art of this dissertation, a novel design, whih ombines the hase shifting and stereovision tehniques, is roosed to eliminate errors aused by inaurate hase measurement, for examle, eriodi errors due to the nonlinearity of the rojetor s gamma urve. This method uses two ameras, whih are set u for stereovision and one rojetor, whih is used to rojet hase-shifted fringe atterns onto the objet twie with the fringe atterns rotated by 90 degrees in the seond time. Fringe images are taken by iv

7 the two ameras simultaneously, and errors due to inaurate hase measurement are signifiantly redued beause the two ameras rodue hase mas with the same hase errors. One side effet of this method is that the rojetor alibration is not neessary, whih simlifies the alibration of the entire system. The use of a visibility-modulated fringe attern is roosed as well to redue the number of images required by this ombined method. This new fringe attern is sinusoidal in the horizontal diretion as in a onventional fringe attern, but is visibilitymodulated in the vertial diretion. With this new attern, we an obtain the hase information in one diretion and fringe visibility information in the other diretion simultaneously. Sine no attern hanging is neessary during the image aquisition roess, the image aquisition time an be redued to less than half of the time reviously required, thus making the measurement of dynamially hanging objets ossible. A olor system is designed to further imrove the seed of this system. Color ameras and olor rojetor are introdued in this system. By utilizing these olor devies, one olor fringe image is suffiient to reonstrut a 3D model instead of three blak and white fringe images. The three hase shifting fringe atterns are enode into the R, G and B hannels of the olor attern whih is rojeted onto the objet. And the olor fringe images taken by olor ameras an be searated into three blak and white images. By using this tehnique, we an further imrove the seed of the struture light system, and the system will be more resistant to fast moving objets. Finally, a ortable 3D measurement system based on the ombined hase shifting and stereovision method is roosed, whih an be used to measure large objets. During the whole measurement roedure, the rojetor is used to rojet a visibility-modulated v

8 fringe attern on the objet and is relatively fixed to the objet. The ameras are moved to as many ositions as needed to ature loal views. These loal views an then be transformed into the same global oordinate system to form the whole 3D model of the big objet whih an not be measured by one take. vi

9 Table of Contents LIST OF FIGURES... IX LIST OF TABLES... XIII ACKNOWLEDGMENT... XIV CHAPTER 1 INTRODUCTION APPLICATIONS AND MOTIVATIONS Medial system Comuter grahis and entertainment Reognition Roboti vision RELATED WORKS Otial 3D measurement tehniques Stereovision Time-of-flight Photogrammetry Laser sanner Moiré Other tehniques Strutured light tehniques Real-time system Camera model and alibration Aroahes to measure 360 degree shae of objets OBJECTIVES DISSERTATION STRUCTURES CHAPTER 2 3D SHAPE MEASUREMENT BASED ON THE PHASE SHIFTING TECHNIQUE PHASE SHIFTING TECHNIQUE Fundamental onet Three-ste algorithm Phase unwraing SYSTEM SETUP LINEAR CALIBRATION AND COORDINATE RECONSTRUCTION SUMMARY CHAPTER 3 NONLINEAR CALIBRATION FUNDAMENTAL CONCEPTS OF LENS DISTORTION NONLINEAR CALIBRATION OF CAMERA NONLINEAR CALIBRATION OF PROJECTOR SUMMARY CHAPTER 4 REAL-TIME SYSTEM BASED ON THE NONLINEAR CALIBRATION METHOD vii

10 4.1 REAL-TIME SYSTEM WITH AN ENCODED COLOR PATTERN HZ SYSTEM SUMMARY CHAPTER 5 QUALITY MAP GUIDED PHASE UNWRAPPING ALGORITHM PHASE AMBIGUITY QUALITY MAP GUIDED PHASE UNWRAPPING SUMMARY CHAPTER 6 THE COMBINED PHASE SHIFTING AND STEREOVISION METHOD PHASE ERRORS OF THE PHASE SHIFTING METHOD COMBINED PHASE SHIFTING AND STEREOVISION METHOD Priniles and system setu Pixel mathing and 3D model reonstrution Exerimental results USE OF A VISIBILITY-MODULATED FRINGE PATTERN Priniles Exerimental Results SUMMARY CHAPTER 7 COLOR SYSTEM BASED ON THE COMBINED PHASE SHIFTING AND STEREOVISION TECHNIQUE MOTION ERROR OF B/W SYSTEM COLOR BASED APPROACH Color-enoded fringe attern Color imbalane and olor ouling Use of a olor visibility-modulated fringe attern EXPERIMENTS System setu Comensation methods Exerimental results Disussion SUMMERY CHAPTER 8 PORTABLE 3D MEASUREMENT SYSTEM PRINCIPLES System setu and measurement strategy Data Registration EXPERIMENTAL RESULTS SUMMARY CHAPTER 9 CONCLUSIONS AND FUTURE WORKS CONCLUSIONS FUTURE WORKS BIBLIOGRAPHY viii

11 List of Figures Figure 2.1: The variation of intensity with the referene hase Figure 2.2: The onversion of the hase to modulo 2π Figure 2.3: Phase unwraing roess Figure 2.4: Shemati diagram of the system layout Figure 2.5: Inside look of the system Figure 2.6: Enoded olor fringe attern Figure 2.7: System timing hart Figure 2.8: Examle of 3D reonstrution. (a)-() Three fringe images. (d) Wraed hase ma. (e) Reonstruted 3D model. (f) Reonstruted 3D model with texture Figure 2.9: Camera oordinate system and world oordinate system Figure 2.10: Chekerboard used for alibration. (a) Red/Blue hekerboard. (b) Chekerboard image Figure 2.11: World oordinate system in (a) A CCD hekerboard image (b) A DMD hekerboard image Figure 2.12: Eight images used to determine CCD and DMD ixel orresondene Figure 2.13: Relationshi between the three oordinate systems Figure 3.1: Examle of radial and tangential lens distortion. (a) Radial distortion. (b) Tangential distortion Figure 3.2: Chekerboard image before and after lens distortion orretion. (a) Before nonlinear orretion. (b) After nonlinear orretion Figure 3.3: Error mas of measured flat board. (a) 3D model of a flat board at a near osition. (b) Error ma when the flat board is at the near-osition. () Cross setion of the near-osition error ma at the 450th row. (d) Error ma when the flat board is laed at a further osition Figure 3.4: Error mas after the amera nonlinear alibration algorithm is alied. (a)-(b) The error at a near osition. ()-(d) The error ma at a further osition Figure 3.5: Error mas of the flat board measured in different orientations Figure 3.6: Error mas of a flat board measured with different alibration methods: (a) Linear alibration method. (b) Nonlinear alibration on amera only. () Nonlinear alibration on both amera and rojetor with the eight-image ix

12 method. (d) Nonlinear alibration on both amera and rojetor with the iterative method. (e) Nonlinear alibration on both amera and rojetor with the ubi equation method Figure 3.7: 3D reonstrution results obtained by using the nonlinear ubi equation method Figure 4.1: Flow hart of the multi-thread real-time 3D shae measurement system Figure 4.2: (a) Ideal fringe attern. (b) Modified fringe attern with higher intensity. () Fringe atterns with a short enterline marker. () Fringe atterns with a wide enterline marker Figure 4.3: Markers in the fringe images Figure 4.4: Interative marker-searhing window. (a) Default window. (b) Adjusted window Figure 4.5: Data modulation of the image with a rosshair marker. (a) 2D image. (b) Data modulation Figure 4.6: Automati searhing for the rosshair marker by using eiolar line Figure 4.7: 3D data texture ma. (a) Old texture ma. (b) New texture ma Figure 4.8: Seleted 3D models of a real-time measurement sequene of human fae exression Figure 4.9: Timing hart of the 60 Hz System Figure 4.10: Front and bak views of the high seed amera Figure 4.11: Human fae models sanned by the 60 Hz system Figure 5.1: Phase ambiguity examle. (a) 2D fringe image with hase disontinuity on the lower edge of the ste whih will ause hase ambiguity. (3) 3D model with hase jum ause by hase ambiguity Figure 5.2: Quality ma of the board with a ste Figure 5.3: Quality ma guided flood-fill hase unwraing algorithm Figure 5.4: 3D model of the ste on a board reonstruted by the quality ma guided flood-fill algorithm Figure 5.5: 3D model of a human fae. (a) 2D image. (b) 3D model reonstruted by the old algorithm. () The quality ma. (d) 3D model reonstruted by the quality ma guided flood-fill algorithm Figure 6.1: Examles of hase errors. (a) Error aused by nonlinear rojetor gamma. (b) Error aused by olor ontrast Figure 6.2: Shemati diagram of the system layout Figure 6.3: Absolute hase mas of Zeus statue. (a) Horizontal inreasing absolute hase ma. (b) Vertial inreasing absolute hase ma Figure 6.4: Eiolar onstraint used in this method x

13 Figure 6.5: The setu of the ombined hase shifting and stereovision system. (a) Cameras set u for stereovision. (b) Projetor used to rojet attern is searated from the ameras Figure 6.6: 3D models of (a) Zeus statue and (b) ard box reonstruted by the ombined hase shifting and stereovision method Figure 6.7: Error ross setions of a flat board 3D model. (a) Measured by hase shifting method. (b) Measured by the ombined hase shifting and stereovision method Figure 6.8: Results omarison of modified atterns. (a) Results measured by traditional hase shifting method. (b) Results measured by the ombined hase shifting and stereovision method Figure 6.9: (a) Visibility-modulated fringe attern. (b) Sinusoidal waveforms at two different vertial ositions Figure 6.10: Stereo mathing roedure. (a) Using the hase mas to find the same hase urve (b) Mathing the visibility values to loate the right ixel Figure 6.11: Results measured with visibility-modulated fringe attern Figure 6.12: Cross setions of a flat board 3D model measured with the visibilitymodulated fringe attern Figure 6.13: 8 seleted 3D models of dynamially hanging faial sequene atured by using the ombined hase shifting and stereovision method with the visibility-modulated fringe attern Figure 7.1: Shemati diagram of the error aused by objet motion Figure 7.2: Color imbalane of the three olor hannels Figure 7.3: Color ouling aearane in a green fringe image Figure 7.4: Measurement results of (a) the olor hase shifting method, and (b) the roosed ombined hase shifting and stereovision olor system Figure 7.5: Measurement results of objets in motion by ombined hase shifting and stereovision systems. (a) Results obtained by B/W system. (b) Results obtained by olor system Figure 7.6: Red hannels of two olor images taken by different olor ameras. (a) Image taken by single-ccd amera. (b) Image taken by 3-CCD amera Figure 8.1: Shemati diagram of the ortable system Figure 8.2: Finding orresonding ixel airs for two loal views Figure 8.3: Pattern with multile referene lines Figure 8.4: Exerimental results of a laster statue. (a) The first loal view. (b) The seond loal view. () Loal views before oordinate translation. (d) Merged loal views after oordinate translation Figure 8.5: Measurement results of a metal art xi

14 Figure 8.6: Measurement results of a laster seahorse attahed on a flat board. (a) Point louds of 9 loal views. (b) The whole 3D model. () The entral loal view. (d) The entral loal view in the whole model Figure 8.7: Exerimental results of a fender. (a) The whole 3D model with olored loal views. (b) The whole 3D model shown in one olor xii

15 List of Tables Table 2.1: 2π hase orretion Table 3.1: Camera intrinsi arameters with different distortion models Table 3.2: Camera lens distortion oeffiients Table 3.3: Camera extrinsi arameters Table 3.4: RMS errors of the flat board measured in different orientations. RMS (L): Linear alibration; RMS (CNL): Camera nonlinear alibration Table 3.5: Projetor intrinsi arameters with different distortion models Table 3.6: Projetor lens distortion oeffiients Table 3.7: Projetor extrinsi arameters Table 3.8: RMS errors of the measured flat board in different orientations xiii

16 Aknowledgment First I would like to exress my sinere gratitude to Professor Peisen Huang, my aademi advisor, for his ontinuous guidane and suort during the ast a few years. Without his atient instrutions and brilliant ideas this dissertation an not be finished so smoothly. Many thanks to the ommittee members, Professor Jeffery Ge, Professor Yu Zhou and Professor Hong Qin, for their reious time sent on this dissertation and their valuable suggestions. I give my seial thanks to Song Zhang, who gave me a very helful guidane at the beginning of my researh even after his graduation. And I would like to give my thanks to all my olleagues and friends in the deartment of Mehanial Engineering and in the Stony Brook University. Finally, I want to thank my arents and all my families for their onstant suort and understanding.

17 Chater 1 Introdution In reent years, 3D shae measurement tehniques have found wide aliations in manufaturing, on-line insetion, biomedial engineering, entertainment, quality ontrol, reverse engineering, et [1]. Real-time 3D shae measurement, as a new tehnology breakthrough in this area, has even more otential aliations in many areas due to its aability of measuring moving objets [2]. This dissertation resents a major effort in imroving the erformane of 3D shae measurement systems mainly based on digital fringe rojetion, hase shifting and stereovision tehniques. A novel nonlinear alibration method is develoed and utilized in the real-time system to ahieve highauray, real-time 3D shae measurement. Systems based on a ombined hase shifting and stereovision tehnique is designed as well to eliminate systemati errors. Finally, this ombined tehnique is used in a ortable system whih an be used for measurement of large size objets. 1.1 Aliations and Motivations Medial system 3D measurement systems an hel in automati medial systems, for examle, laser debridement system. Traditionally, dotors need to oerate the system and do all the tedious works themselves to san the whole wounded areas line by line with a laser head. This reeating work an be highly automated by integrating a real-time 3D sensor to 1

18 trak the motion of human bodies and loate the wounded areas with 2D image roessing and 3D information. The laser head an be alibrated with the 3D sensor, and an ommuniate with the 3D sensor to obtain the urrent oordinates for sanning and san the arrears automatially, suh that releases the dotors from the tedious work and ahieve a higher reliability Comuter grahis and entertainment Comuter animation whih is widely used in entertainment industry is inreasingly generated by means of 3D omuter grahis. 3D models of real world objets and human beings are reated when making films and video games. 3D measurement system an hel to generate 3D models in an easier way, rovide higher resolution and make the models more realisti. Motion data sequenes suh as human being s body language, ostures and faial exressions an also be reorded by real-time 3D measurement systems. These raw data an then be transmitted onto the digital 3D model to imitate human beings behaviors Reognition Current faial reognition and seeh reognition tehniques are mostly based on 2D image roessing. However, a main roblem of these tehniques is that 2D images rovide different features when the images are taken from different ersetives. 3D tehniques are more stable in this aset that, regardless of the ersetives, the final 3D models onsist in features. Thus, more aurate and more stable reognition results are 2

19 aomlishable by using 3D sensors whih rovide both 2D images and the orresonding 3D geometries Roboti vision 3D sensors an also be built in assistive roboti systems suh as resue robots to obtain the surrounding features and strutures [3]. With the information rovided by 3D sensors, robot navigation an be realized in luttered dusty environment like urban searh and resue tasks. Vitim identifiation an also be ahieved by human gesture reognition and human emotional state lassifiation with 3D shae information. 1.2 Related Works Traditional Coordinate Measuring Mahines (CMMs) are oint-by-oint measuring systems. CMMs an not meet all the demands of 3D shae measurement beause they are usually slow and an ause otential damage to objet surfaes. With the latest develoment of tehnologies in digital imaging, digital video rojetion, and digital image roessing, vision-based otial metrology tehniques are being utilized more and more extensively. Many methods have been develoed based on the stereovision, hotogrammetry, time-in-flight, interferometry, and various tyes of oded strutured light tehniques, et. The stereovision method and strutured light method based on a hase shifting tehnique are mainly used for this dissertation researh. In this setion, we will give an overview of several well-known otial metrology tehniques as mentioned above and will introdue strutured light tehniques with more details. We will also inlude some related works involving real-time system, alulation 3

20 of absolute hase value, amera models and alibration, and aroahes for full body measurement Otial 3D measurement tehniques Stereovision Stereovision is one of the most studied tehniques [4], in whih two ameras take itures of the same sene, but searated by a distane - simulating the human eyes. It is widely used in mobile robotis for obstales detetion and navigation. By omaring the two images, the best math arts are used to alulate the distane. However, finding the homologous oints is a hallenging roblem and omutationally exensive. Several tehniques have been develoed to rovide more effiient and faster stereo mathing, suh as multi-resolution tehniques and orrelation based tehniques [5]. The stereo mathing roblem is also brought into time domain by a tehnique alled sae-time stereo to redue mathing ambiguity [10] Time-of-flight The time-of-flight method measures the time that it takes for a artile or objet to reah a measurement sensor while traveling over a known distane [11]. Time-of-flight based systems have the advantages of real-time, robust, and aable of oerating over very long distanes. However, this kind of system usually has a limited resolution of millimeter. High-resolution tehniques were also reorted an ahieve submillimeter resolution [12]. In addition, tyial resolution of time-of-flight ameras is usually no more than , whih is muh lower than modern ameras. 4

21 Photogrammetry Photogrammetry tehnique, whih tyially emloys stereo tehnique for 3D measurement, an rovide a very high auray as one art in 100,000 or even 1,000,000 [16]. The reonstrution is usually imlemented by a least square roedure based on the rinile of bundle adjustment [17], [18]. This rinile an be used to realize system self-alibration [19], in whih ase, the 3D oordinates and the system arameters are determined at the same time. However, in a tyial hotogrammetry system, markers are required to be laed at various loations of the objet surfae. The onet of virtual landmarkers was roosed by G. Notni, et al [22], to solve this roblem, whih eliminated the need of hysial markers by using an extra onneting amera relatively fixed to the objet. The hase values of the ixels in the onneting amera were used as the virtual landmarkers. Some other methods suh as defousing, saling and shading are also used for hotogrammetry tehnique Laser sanner Laser sanners are ative sanners whih emloy either time-of-flight tehnique or triangulation relationshi in otis. The triangulation laser sanner emits a laser on the objet and utilizes a amera to trak the laser dot loation. Deending on the distane between the laser emitter and the objet, the laser dot aears at different laes in the amera s field of view. The so alled triangulation relationshi is referring to the triangle formed by the laser dot, the amera and the laser emitter. In order to seed u the aquisition roess, a laser strie, instead of a single laser dot, is often used and swet aross the objet. Triangulation laser sanners usually have a very high measurement 5

22 frequeny of 40 khz or higher [23], [24]. The tyial measurement range is ±5 to ±250 mm, and the auray is about 1 art in 10, Moiré Moiré tehnique has been widely alied as a nonontat method for 3D shae ontouring [25]. Moiré fringes are formed by the suerosition of two gratings. Aording to the rinile used, Moiré tehnique an be lassified into rojetion moiré and shadow moiré [33], [34]. The key to the former method is two searated gratings, one is a master grating and the other is a referene grating. In the latter method, only one grating is used. However it is usually diffiult to aly it for the measurement of large objets. Sna shot or multile image moiré systems have been develoed for overoming environmental erturbations, inreasing image aquisition seed, and emloying hase shift methods for the fringe attern analyzing. Multile moiré fringe atterns with different hase shifts are aquired simultaneously using multile ameras or imageslitting methods [35]. The tyial measurement range of the hase shifting moiré method is from 1 mm to 0.5 m with the resolution at 1/10 to 1/100 of a fringe [38] Other tehniques Other widely used and well studied otial 3D measurement tehniques inlude interferometry [39], Laser Sekle Pattern Setioning [42], shae from shading [45], shae from fous/ defous [46], [47] and strutured light tehniques, whih are emloyed in this researh and introdued in details in the following setion. 6

23 1.2.2 Strutured light tehniques Strutured light tehnique is an ative triangulation tehnique for measuring the 3D shae of an objet by using rojeted light atterns and a amera system. It is similar to the stereovision tehnique, exet that one of the ameras is relaed by an image rojetion devie [48]. Although many other variants of strutured light rojetion are ossible, atterns of arallel stries are widely used. By rojeting a oded fringe attern onto a 3D shaed surfae, the deth information is enoded in the deformed fringe attern. The shae of the surfae an be diretly deoded from the deformed fringe attern images taken by an image aquisition devie [1]. The most well-known strutured light tehnique is hase shifting tehnique, for examle, three-ste hase shifting tehnique [49] used in this researh. For real-time 3D shae measurement, a three-ste algorithm is the otimal hoie beause it requires the minimum number of fringe images. In most ases, three fringe atterns with 120 degrees hase shift to eah other are rojeted onto the objet by a rojetor. At the meanwhile, a amera whih is alibrated with the rojetor is used to grab fringe images of the objet. Then a hase ma, whih is used to deode the 3D shae, an be alulated from every three fringe images by hase wraing and unwraing algorithms. The details of threeste hase shifting tehnique will be exosed in Chater 2. Binary oding is another well-known tehnique whih extrats deth information by rojeting multile binary oded strutured light atterns [52], [53]. In this tehnique, only two illumination levels, 0 and 1, are ommonly available. Multile strie atterns are rojeted, and for every ixel its orresonding value in eah image is either 0 or 1. A odeword omosed of 0s and 1s is obtained for every ixel when the rojetion 7

24 sequene is omleted. 3D shae information an be retrieved based on deoding the odeword. This tehnique is robust to noise. However, the resolution annot be very high. In order to inrease resolution more atterns ould be rojeted whih, in another aset, inreases the aquisition time. In general this method is not suitable for high-resolution real-time measurement. Tehniques of multi-level grey oding are develoed, in whih multile intensity levels are used to redue the number of required fringe atterns. Referenes an be found in [54]. The N-ary methods signifiantly redue number of fringe images; however, they usually require additional threshold referenes to obtain high resolution. Binary ode tehnique is atually a seial ase of N-ary ode when N equals 2. Methods using olor-enoded fringe atterns have been roosed for high seed measurement by reduing the required number of fringe images, sine a single olor image inludes more information than a single grey level image. Harding roosed a olor enoded moiré tehnique that an reonstrut the 3D shae by a single snashot [58]. Zhang and Curless develoed a olor strutured light tehnique, whih rojets a attern of stries with alternating olors and an reover the 3D shae from one or more images [59]. Jeong and Kim resented a olor grating rojetion moiré method, whih an erform 3D ontouring with only a single oeration [60]. Huang et al. roosed a olor hase shifting method, in whih three hase-shifted fringe atterns are enoded in the Red, green, and blue hannels to form a olor fringe attern [61]. Sine only one single olor fringe image is suffiient for 3D reonstrution, this tehnique signifiantly inreases the measurement seed. However, due to hase errors aused by the use of olor, the ahieved measurement auray was limited. Pan et al. ontinued the work by 8

25 introduing both hardware and software based methods to redue hase errors aused by the use of olor [62]. Signifiant imrovement was ahieved, but the results were still not ideal. There are various other strutured light tehniques have been develoed and reorted, suh as the use of Fourier transform [63], [64], intensity ratio [65], [66], random atterns [67], traezoidal hase shifting atterns [68], et Real-time system Real-time 3D shae measurement tehnique is attrating inreasingly more attention in the researh ommunity lately due to its ability of aturing the 3D shaes of moving objets. So far a number of aroahes have been roosed, whih an be lassified into two basi ategories: single-attern-based tehniques and multile-atternbased tehniques. Single-attern-based tehniques, as the name suggests, use only one attern for 3D shae aquisition. This kind of methods usually use olor attern [58][69], as well as Furious transform [70]. As for multile-attern-based tehniques, multile atterns are swithed quikly so that images of these atterns an be taken in a relatively short time eriod [2][71], [72]. Obviously, single-attern tehniques have the advantage of higher ahievable measuring seed. However, multile-attern tehniques usually rovide better resolution Camera model and alibration A roer alibration of elements used in any strutured light system is the key to aurate 3D reonstrution [73]. Methods based on different aroahes have been 9

26 develoed, suh as absolute hase [74], bundle adjustment [19][75][76] and neural networks [77], et. For any strutured light system that uses ameras and rojetors, lens distortion, eseially the radial distortion, is a ommon issue. It has to be dealt with if more aurate measurement is to be made. Elaborate lens distortion models with high order terms have been investigated in many ases [78]. However, a lens distortion model with only two radial terms is usually suffiient. In most ases, the lens distortion is atually dominated by the first radial omonent [81], [82]. A novel alibration method is develoed by Zhang and Huang. In this method, the rojetor is treated as a amera, and the alibration roedure is muh simlified [83]. A nonlinear alibration based on this tehnique is further develoed by Huang and Han [84] Aroahes to measure 360 degree shae of objets To measure objets with large sizes or ondut 360 degree measurement, either a single system taking measurements sequentially at different loations or multile systems taking measurements simultaneously from different ositions are neessary [1]. One these measurement data or loal views are obtained, a registration method is used to transform these loal views from their own oordinate systems to a global oordinate system and then stith them together to form a omlete 3D model [85], [86]. Several aroahes ould be used to determine the transformations between the loal views. One is to have the system fixed on a high auray mehanial ositioning system, whih is quite straightforward [87], [88]. However, suh a system usually requires equiment with a high ost in order to ahieve high auray. Another aroah is hotogrammetry, whih an rovide a high auray without the need for exensive 10

27 equiment [89]. However, in a tyial hotogrammetry system, markers are required to be laed at various loations of the objet surfae. During the measurement, every loal view needs to inlude some of these markers to establish its relationshis with its neighboring views. The use of hysial markers makes the roedure time-onsuming and omlex. To solve this roblem, G. Notni roosed the onet of virtual landmarkers [22], whih eliminated the need of hysial markers by using an extra onneting amera. Multile ameras and rojetors are used in his system and the bundle adjustment tehnique is used to obtain the system arameters and objets oordinates simultaneously. 1.3 Objetives This dissertation researh is foused on imroving the erformane of the 3D shae measurement systems based on digital fringe rojetion, hase shifting and stereovision tehniques. Some issues need to be addressed in order to make these methods even more aeted. The objetives of this researh are listed as following: 1) Emloy a nonlinear alibration method for ameras and rojetors used in the hase shifting system to redue the error aused by lens distortion. 2) Imlement the nonlinear alibration method, absolute hase alulation, quality ma guided hase unwraing algorithm and new-generation high seed ameras in the real-time system to simlify the system setu and build u a new real-time measurement system with higher seed and better auray. 3) Develo new systems based on a ombined hase shifting and stereovision method to eliminate the errors aused by the rojetor s nonlinear gamma urve. Realize the 11

28 measurement of moving objets by alying a novel visibility-modulated fringe attern. 4) Develo a system omosed of olor devies and use olor visibility-modulated fringe attern to ahieve further imrove the measurement seed. 5) Develo a system for large objets measurement based on the ombined hase shifting and stereovision method. 1.4 Dissertation Strutures Chater 2 introdues the basi theory of three-ste hase shifting tehnique and desribes the setu of a tyial 3D shae measurement system based on this tehnique. This hater also introdues the linear alibration method reviously develoed, whih is the basis of the nonlinear alibration method. In Chater 3, some fundamental onets of lens distortion are first introdued. Then a nonlinear amera alibration method is roosed, followed by a nonlinear rojetor alibration method. The results on the measurement auray ahieved by both the linear and nonlinear alibration methods are omared. Chater 4 desribes the imlementation of the nonlinear alibration method in real-time measurement system. This hater desribes a new method for obtaining the absolute hase information. As a result, the roosed nonlinear alibration method an be utilized in the real-time measurement system to signifiantly imrove the auray of the system. This hater also introdues the next-generation real-time system, whih has a 3D shae aquisition seed of 60 Hz. Chater 5 introdues a new quality ma guided hase unwraing algorithm, whih is designed to eliminate hase unwraing ambiguities. The hosen quality ma 12

29 and the new flood-fill algorithm is evaluated and some imroved hase unwraing results are resented. Chater 6 introdues a new method based on a ombined hase shifting and stereovision tehnique to eliminate the hase errors aused by the rojetor s nonlinear gamma urve. Two ameras are set u for stereovision, and the rojetor is used to rojet fringe atterns for stereo mathing. Sine the two ameras rodue hase mas with the same hase error, systemati errors are signifiantly eliminated. The use of a new designed visibility-modulated fringe attern is also roosed to redue the required number of fringe images, so that the system an be used to measure moving objets. In Chater 7, a olor system based on the ombined hase shifting and stereovision tehnique is roosed to further imrove the seed of this system. Color ameras and olor rojetors are utilized in this system. The seial issues, suh as olor imbalane and olor ouling, are dealt with by roer methods. Measurement results of both stati and dynami objets are omared with the results taken by the blak and white system and the traditional hase shifting system. Chater 8 rooses a ortable 3D measurement system based on the ombined hase shifting and stereovision method whih an measure large objets. This system is very simle in struture, and the only mobile art is the two ameras onneted by a light metal frame whih is easy to handle. Effiient registration method is designed for quik and aurate 3D model generating. Chater 9 resents the onlusions of this researh and disusses the future works. 13

30 Chater 2 3D Shae Measurement Based on the Phase Shifting Tehnique 2.1 Phase Shifting Tehnique The hase shifting tehnique used in this researh has its root in hase shifting interferometry (PSI), whih is a fringe analysis method widely used in otial metrology. PSI simly reords a series of interferograms, in whih the wavefront hase is enoded, while the interferometer referene hase is hanged and reovers the hase just by a oint-by-oint alulation. The need to loate the fringe enters is thus disarded, simlifying fringe analysis and imroving its auray. PSI has been widely known by many names inluding hase measuring interferometry, fringe sanning interferometry and real-time interferometry. But all of them desribe the same basi tehnique Fundamental onet The fundamental onet behind PSI is the introdution of a time-varying hase shift between the referene wavefront and the test wavefront in the interferometer. As a result, a time-varying signal, in whih the relative hase between the two wavefronts is enoded, is rodued at eah measurement oint in the interferogram [51] [90]. The referene and test wavefronts in the interferometer an generally be reresented by the following exressions: w ( x, y, t) r a ( x, y) e i[ φr ( x, y) δ ( t)] = r

31 and w ( x, y) t iφt ( x, y) = at ( x, y) e 2.2 where a r ( x, y) and a t ( x, y) are the wavefront amlitudes, φ ( x, y) and φ ( x, y) are the wavefront hases, and δ (t) is a time-varying hase shift introdued into the referene beam. The resulting intensity attern is r t I ( x, y, t) = w ( x, y, t) w ( x, y) 2.3 r + t 2 or I ( x, y, t) = I'( x, y) + I"( x, y)os[ φ ( x, y) φ ( x, y) + δ ( t)] 2.4 where I'(x,y)= a 2 r (x, y)+ a 2 t (x,y) is the average intensity and I" ( x, y) = 2a ( x, y) a ( x, y) is the fringe or intensity modulation. If we define φ( x, y) to be the wavefront hase differene φ ( x, y) φ ( x, y), we obtain t r t I ( x, y, t) = I' ( x, y) + I"( x, y) os[ φ ( x, y) + δ ( t)] 2.5 whih is the fundamental equation for PSI. This result an be illustrated by lotting the intensity as a funtion of δ (t), as shown in Figure 2.1. The onstant term I '( x, y) r r t is the intensity bias, I" ( x, y) is half the eak-to-valley intensity modulation, and the unknown hase φ( x, y) is related to the temoral hase shift. 15

32 I(x, y) I (x, y) I (x, y) 2π Ф (x, y) 2π Phase Shift δ (t) Figure 2.1: The variation of intensity with the referene hase Three-ste algorithm All of the ommonly used algorithms for PSI share the same harateristi: a series of interferograms are reorded as the referene hase is varied. The wavefront hase is then alulated at eah oint as the artangent of a funtion of the interferogram intensities measured at that oint. The final wavefront ma is obtained by unwraing the hases to remove the 2π hase disontinuities. Sine there are three unknowns ( I ( x, y), I"( x, y), φ ( x, y)) in Eq. (2.5), the minimum number of intensity values that is required to reonstrut the wavefront at eah measurement oint is three. For real-time 3D shae measurement, a three-ste algorithm 16

33 is the otimal hoie beause it requires the minimum number of fringe images. Assuming equal hase stes of size α in Eq. (2.5), orδ i = α,0, α, where i = 1, 2, 3, we have and [ φ( x, α] I ( x, y) = I' ( x, y) + I"( x, y) os ) y [ φ( x, )] I 2 ( x, y) = I' ( x, y) + I"( x, y) os y 2.7 [ φ ( x, +α] I ( x, y) = I'( x, y) + I"( x, y)os ) y Solving these equations, we an easily find the wavefront hase at eah oint as: 1 1 os( α) I1 I 3 φ ( x, y) = tan 2.9 sin( α) 2I 2 I1 I 3 and the average intensity I '( x, y) and intensity modulation I "( x, y) are I1+ I 3 2I 2 os( α) I '( x, y) = [1 os( α)] and 2 2 1/ 2 {{[1 os( α)]( I1 I 3)} + [sin( α)(2i 2 I1 I 3 )] } I ''( x, y) = sin( α)[1 os( α)] resetively. The ratio of these two intensities defines the so alled data modulation: I ''( x, y) γ ( x, y) = 2.12 I'( x, y) The most ommonly used hase stes for the three-ste algorithm are π/2 and 2π/3. In our ase, α = 2π/3 is used, whih results in the following equations for the hase and intensity modulation: 1 I1 I 3 φ ( x, y) = tan I 2 I1 I 3 17

34 and 2 2 3( I1 I 3 ) + (2I1 I 2 I 3 ) γ ( x, y) = 2.14 I + I + I Phase unwraing One more oeration must be erformed before the hase result is ready to be used for wavefront reonstrution. Sine artangent is defined only over the range of -π/2 to π/2, the disontinuities that our in the hase alulation must be orreted. The signs of sine and osine of the hase are known indeendently in addition to the tangent value. Table 2.1 shows how to onvert artangent values to hase values between 0 and 2π and Figure 2.2 illustrates this roess. After the above roess, the hase ma obtained still has 2π disontinuities, whih need to be removed as well to finally reonstrut the wavefront. This seond roess is usually alled hase unwraing and the rinile is shown in Figure 2.3. Table 2.1: 2π hase orretion. Sine Cosine Correted Phase Φ(x, y) Phase Range Φ(x, y) 0 to π/2 + 0 π/2 π/2 + - Φ(x, y) + π π/2 to π 0 - π π - - Φ(x, y) + π π to 3π/2-0 3π/2 3π/2 - + Φ(x, y) + 2π 3π/2 to 2π 18

35 Calulated Phase 2π π/2 -π/2 π 2π Atual Phase Figure 2.2: The onversion of the hase to modulo 2π. Phase Unwraed Phase 4π 2π Wraed Phase Position Figure 2.3: Phase unwraing roess. 19

36 2.2 System Setu The 3D shae measurement system desribed in this hater is based on the digital fringe rojetion and hase shifting tehnique. A digital rojetor (PLUS U5-632 with a resolution of and a brightness of 3000 lumens) is utilized to rojet fringe atterns generated by a ersonal omuter onto the objet being measured. A CCD amera (Dalsa CA-D with a resolution of ) is used to grab fringe images of the objet, whih are then roessed by the omuter for 3D shae reonstrution. Figure 2.4 shows a shemati diagram of the system layout. Both the rojetor and the amera ommuniate to the same omuter, whih has dual dislay oututs. One dislay outut is used for the omuter monitor, while the other used for reating the fringe attern to be rojeted by the rojetor. Figure 2.5 is a hotograh of the atual system. Fringe attern R Data Projetor Comuter G B Objet CCD Camera Figure 2.4: Shemati diagram of the system layout. 20

37 Power suly Timing iruit Projetor Camera Figure 2.5: Inside look of the system. Based on the rojetion mehanism of the rojetor, we enode the three haseshifted fringe atterns into the three olor hannels: red, green, and blue. The rojetor rojets a olor image hannel-by-hannel eriodially and at any time, only one hannel/olor is rojeted. To revent the measurement result from being affeted by the surfae olor of the objet, the rojetor is set in blak and white (B/W) mode. By using the rojetor in this way, we an rodue a dynamially shifting B/W fringe attern at a frequeny of 360 Hz. If we further use a higher seed amera and synhronize it with the rojetor, we an ature three hase shifted fringe atterns at a maximum seed of 120 Hz for real-time 3D shae measurement. In this hater, the system atures the three fringe images at a lower frequeny of 30 Hz due to the limited frame rate of the amera being used. The hase ste α is 2π/3, so the fringe intensity funtions for eah olor hannel are: 21

38 22 + = os 1 ), ( π π x a y x I r = x a y x I g π 2 os 1 ), ( 2.16 and + + = os 1 ), ( π π x a y x I b 2.17 where is the fringe ith and a is the fringe amlitude. Figure 2.6 shows the fringe attern generated by the omuter. It should be noted that when rojeted by the aforementioned rojetor, the fringe atterns lose their olor and beome B/W. The fringe images atured by the amera an be reresented by the following equations: + = 3 2 ), ( ) os, ( '' ), ( ' ), ( π φ y x y x I y x I y x I r 2.18 )], ( )os[, '( ' ), '( ), ( y x y x I y x I y x I g φ + = 2.19 and + + = 3 2 ), ( )os, ( '' ), ( ' ), ( π φ y x y x I y x I y x I b 2.20 Figure 2.6: Enoded olor fringe attern. Fringe Pattern Sinusoidal Intensity Profile 2π/3 2π

39 The fringe atterns are rojeted onto the objet sequentially and eriodially at a high seed. When grabbing the fringe images, the most imortant thing is to make sure every single image atured by the amera should have the fringe information of only one hannel. Otherwise hase ma reonstrution will not be orret. For this reason, the system needs to be synhronized and the exosure time of the amera needs to be set exatly the same as the duration of the rojetion time for eah hannel. The CCD amera utilized in this hater is ontrolled by a frame grabber (Matrox Meteor II/Digital). By using Matrox Imaging Library, the amera an be set in the trigger mode, whih means only when a trigger signal is generated and sent to the frame grabber, an the amera grab an image. To generate the trigger signal, a simle iruit was designed, whih takes the video refresh signal of the rojetor as the inut signal. Channels rojeted (120 Hz) R G B R G B R G B R G B Video refresh signal (60 Hz) Trigger signal to amera (90 Hz) R G B Channels atured Figure 2.7: System timing hart. 23

40 Figure 2.7 shows the system timing hart. For every single refresh ulse, the rojetor rojets Red, green, and blue hannels twie. For every four Red, green, and blue yles, the amera atures three fringe images, whih are used to reonstrut one hase ma based on the three-ste algorithm. Sine the video refresh rate is set to 60 Hz, the rojeting frequeny is 120 Hz and the system an reonstrut 3D shae of an objet at a frame rate u to 30 frames er seond. Figure 2.8 shows an examle of 3D measurement roedure by using this system. (a) (b) () (d) (e) (f) Figure 2.8: Examle of 3D reonstrution. (a)-() Three fringe images. (d) Wraed hase ma. (e) Reonstruted 3D model. (f) Reonstruted 3D model with texture. 24

41 2.3 Linear Calibration and Coordinate Reonstrution System oordinate reonstrution onverts the hase ma to x, y, and z oordinates of the objet surfae. The method desribed in this setion is based on a linear amera alibration method that uses a Matlab toolbox develoed by Jean-Yves Bouguet. For rojetor alibration, a new method is introdued to make a rojetor ature images as a amera. With a series of hekerboard images atured by the rojetor, we an use the same Matlab toolbox to alibrate the rojetor. A tyial inhole amera model has several intrinsi arameters inluding foal length, rinile oint, skew oeffiient and distortions, as well as extrinsi arameters suh as the rotation and translation matries. f y w x w [u 0,v 0 ] y x z w o w CCD o z Objet Figure 2.9: Camera oordinate system and world oordinate system. Figure 2.9 shows the relationshi between the amera oordinate system [ x y z ] o and the world oordinate system[ o x y z ]. f is the foal length and [ u 0 v 0 ] is the rinile oint. Based on this model, the onversion between a w w w w 25

42 oint on an objet and the orresonding rojetion on the amera sensor an be exressed as: si ] w = A[ R, t X 2.21 where I [ u v 1] T = is the image oordinate matrix, w X is the world oordinate matrix, A is intrinsi arameter matrix, and [ R t] is the extrinsi arameter matrix. The intrinsi arameter matrix has the following form f u A= 0 α 0 u0 v 1 0 f v where fu and f v are foal lengths along the u and v axes of the image lane resetively, u 0 and v 0 are oordinates of the rinile oint, and α is the rodut of skew oeffiients defining the angles between the u and v ixel axes and f u. The extrinsi arameter matrix is omosed of the following rotation and the translation matries: Rotation matrix r11 r12 r13 R = r21 r22 r r 31 r32 r33 and Translation matrix t= [ t 1 t 2 t 3 ] T 2.24 It should be noted that a linear alibration method does not onsider lens distortion. A nonlinear alibration method, whih takes lens distortion into onsideration, will be disussed in Chater 3. To obtain the intrinsi and extrinsi arameters of the amera using the Matlab toolbox, a series of hekerboard images are atured. The hekerboard used in this 26

43 researh onsists of mm squares in alternating blue and red olors. Figure 2.10 shows the hekerboard attern and a hekerboard image atured by the amera. (a) (b) Figure 2.10: Chekerboard used for alibration. (a) Red/Blue hekerboard. (b) Chekerboard image. z y x z y x (a) (b) Figure 2.11: World oordinate system in (a) A CCD hekerboard image (b) A DMD hekerboard image. 27

44 After extration of orner oordinates for every hekerboard image, the intrinsi arameters an be obtained in the unit of ixels. Extrinsi arameters are alulated based on an additional hekerboard image. The extration of orner oordinates needs to be done one more to determine the extrinsi arameters exressed as the rotation and translation matries. The world oordinate system in the CCD and DMD images are shown in Figure The x-y lane is the hekerboard lane and the z-axis is erendiular to the hekerboard lane ointing outward. Projetor alibration is usually more laborious than amera alibration beause the rojetor annot ature hekerboard images diretly as a amera does. However, after a method for absolute hase measurement is introdued to make the rojetor ature a series of images, the rojetor alibration roedure beomes muh simler. In order to alulate the absolute hase, an additional enterline image is atured. This enterline is loated in the middle of the DMD hi. One this enterline image is atured by the amera, the loation of the ixels that reresent the enterline in the image is first deteted. Then, in the hase ma reonstruted from the three fringe images, the hase values at the enterline ixels are averaged and this average hase value is subtrated from the hase value of every ixel in the hase ma. Thus, an absolute hase ma, in whih the hase values at the enterline ixels are zero, is generated. This absolute hase ma allows us to determine the orresondene between the ixels on the CCD sensor and the ixels on the DMD hi. For eah ixel on the CCD sensor, a line, either vertial or horizontal deending on the diretion of the rojeted fringe attern, on the DMD hi an be identified. To determine the ixel-to-ixel orresondene between the CCD sensor and DMD hi, a total of eight atterns with four in the vertial diretion 28

45 and four in the horizontal diretion are needed to be rojeted as shown in Figure By establishing suh a one-to-one ma between the CCD sensor and the DMD hi, every ixel in a CCD image is orrelated to a DMD ixel. In other words, if we already have a CCD hekerboard image, using the method desribed above, a orresonding DMD image an be generated, whih is regarded as an image atured by the rojetor. Figure 2.12: Eight images used to determine CCD and DMD ixel orresondene. After the roedure introdued above, the intrinsi and extrinsi arameters of the rojetor an be estimated by using the same toolbox used for the amera. It should be noted that in order to establish the oordinate relationshi between the amera, the rojetor, and the world oordinate system, the DMD image used to determine the extrinsi arameters of the rojetor has to be the DMD image that orresonds to the CCD image used to determine the extrinsi arameters of the amera. Figure 2.13 shows 29

46 the relationshi between these three oordinate systems. One the intrinsi and extrinsi arameters of both the amera and the rojetor are obtained, we an use them to reonstrut the 3D shae of the objet. While eight hekerboard images are required during the alibration roess, only four images (three fringe images lus a enterline image) are needed for normal measurement. v v CCD [u 0,v 0 ] u f y x o w z w y w x w o z Otial Axis f y x DMD o z Objet v v [u 0,v 0 ] u Figure 2.13: Relationshi between the three oordinate systems. 30

47 where For oordinate reonstrution, let us first rewrite Eq. (2.21) in the following form: A, R and T [ u v ] A [ R, t ][ X Y Z ] T s w w w 1 1 = 2.25 t reresent the intrinsi and extrinsi arameters of the amera. For any arbitrary ixel, u and v are its ixel oordinates and are known. A and R are 3 3 matries and unknowns: s, X w t is a 3 1 matrix. There are three equations in Eq. (2.25) with four, Y w, and Z w equation that is similar to Eq. (2.25): where A, R and. For the rojetor side, we an write the following T [ u v ] A [ R, t ][ X Y Z ] T s w w w 1 1 = 2.26 t reresent the intrinsi and extrinsi arameters of the rojetor. There are also three equations in Eq. (2.26) but with additional three unknowns: and s, u, v. Combining Eqs. (2.25) and (2.26), we have six equations with seven unknowns. Therefore, an additional equation is needed in order to solve these equations for X w, Y w and Z. This additional equation an be found by using the absolute hase method. That w is u (when vertial fringe atterns are used) or v (when horizontal fringe atterns are used) an be determined from the absolute hase value of ixel[ u v ]. In our system, the fringe attern is rojeted vertially due to hardware design. As a result, determined by the orresonding hase value. With equations and six unknowns left ( s, X w, Y w, and therefore the equations an be solved. Z w, u is u determined, there are now six s, and v ) in Eqs. (2.25) and (2.26) 31

48 2.4 Summary This hater introdued the fundamental theory of hase shifting algorithm and rovided an overview of our system setu. In Setion 2.3, the linear alibration method and the 3D reonstrution method were desribed in details. This linear alibration method is the basis of the nonlinear alibration method to be exlained in details in Chater 3. 32

49 Chater 3 Nonlinear Calibration System alibration is usually a omliated and time-onsuming roedure, eseially when a rojetor is involved. Unlike amera alibration, rojetor alibration is less well understood and established. The method introdued in Chater 2 makes rojetor alibration muh faster and more systemati than reviously roosed methods. However, this alibration method uses only a linear model and does not onsider lens distortion when omuting the intrinsi and extrinsi arameters and reonstruting 3D oordinates. To further imrove system auray, lens distortion, as the most dominant error soure of the system, has to be onsidered in alibration and oordinate reonstrution. This hater fouses on the nonlinear alibration of amera and rojetor. Lens distortion arameters are first determined using the Matlab toolbox and then used in oordinate reonstrution. 3.1 Fundamental Conets of Lens Distortion Lens distortions usually an be modeled as radial and tangential distortions, as illustrated in Figure 3.1. Eah arrow reresents the effetive dislaement aused by lens distortion. Figure 3.2 shows an examle of the original hekerboard image together with the same image after lens distortion is orreted by software. Cheking the line and the irle arefully, we notie that lens distortion is most serious in the orner areas. In most ases, lens distortion an be adequately desribed by the first two terms of radial distortion. 33

50 Pixel (a) Pixel Pixel (b) Pixel Figure 3.1: Examle of radial and tangential lens distortion. (a) Radial distortion. (b) Tangential distortion. 34

51 (a) Before orretion (b) After orretion Figure 3.2: Chekerboard image before and after lens distortion orretion. (a) Before nonlinear orretion. (b) After nonlinear orretion. To onstrut a nonlinear amera model that takes lens distortion into onsideration, assume P to be a oint on the objet surfae with a oordinate vetor [ X Y Z ] X = in the amera oordinate system. Let n rojetion on the image lane. Then, in a linear model: x be the normalized x n X = Y / Z x = / Z y If we assume r = x + y, after inluding the lens distortion the new normalized oordinate x d is defined as follows: xd ( 1) xd = = + k r + k r + k r xn + dx x (1 (1) (2) (5) ) d (2) 3.2 where dx is the tangential distortion defined as: 35

52 2 2 2k(3) xy+ k(4)( r + 2x ) dx = 2 2 k(3)( r + 2y ) + 2k(4) xy 3.3 In Eqs. (3.2) and (3.3), k is a vetor that ontains both radial and tangential distortion oeffiients. One distortion is alied, the ixel oordinates [ u v] of the rojetion on the image lane is: u fu ( x = v d (1) + αx f x (2) + v v d d (2)) + u or u v = 1 xd (1) A xd (2) where A is the amera intrinsi arameter matrix as shown in Eq. (2.22). Table 3.1 omares the intrinsi arameters when different numbers of distortion oeffiients are used. From this table we an see that the intrinsi arameters have an obvious differene between the linear model and the nonlinear model. But between the two nonlinear models (one uses only one distortion oeffiient and the other uses four), the differene is not that obvious. Atually, in the latter two models, the arameter flutuations are within the unertainties of the estimated arameters. Table 3.2 shows the first four distortion oeffiients along with their estimation unertainties. The seond and the third oeffiients are aroximately equal to or even larger than the unertainties, whih makes them unreliable. The fourth oeffiient is too small omared to the first one, although it is two times larger than its unertainty. Therefore, this amera nonlinear model an be best desribed by only the first distortion 36

53 oeffiient, beause the inlusion of any additional oeffiient not only would not hel but also would ause numerial instability [23]. Table 3.1: Camera intrinsi arameters with different distortion models. Linear Nonlinear k(1) only Nonlinear k(1) to k(4) f u f v u v α k(1) Table 3.2: Camera lens distortion oeffiients. k(1) k(2) k(3) k(4) Coeffiient Unertainty (±)

54 3.2 Nonlinear Calibration of Camera The lens used on the amera is a 16-mm otial lens. From Eqs. (2.25) and (3.5), the following relationshi is obtained: xd (1) xd (2) = 1 1 s T 1 [ R, t ][ X Y Z 1] = [ X Y Z ] T w w w s 3.6 where [ Y Z ] X is the oordinates of a oint on the objet surfae in the amera oordinate system. s is a onstant used to normalize these oordinates. Obviously, in this ase s = Z and the new exression is: xd (1) X / Z x d (2) = Y / Z 1 1 = x n 3.7 where x n is the same vetor as shown in Eq. (3.1). Aording to Eq. (3.2), only when k(n) = 0, n = 1, 2, 3, 4, 5, Eq. (3.7) is tenable. It is onsistent with the fat that the alibration method we used before is a linear method. To evaluate the effet of lens distortion, a flat white board was sanned from different ersetives. Using a least-square lane fitting method, we found the error ma, whih is the deviation of the measured data from the fitted lane. Two examles of the error ma are shown in Figure 3.3. In these two examles, the flat board is set in the same orientation (aroximately erendiular to the otial axis of the amera), but at different distanes from the amera. Notie that the error in the seond ma is muh larger than the first one. This is beause the lens distortion effet is roortional to the field-of-view of the amera, whih in turn is roortional to the distane between the 38

55 objet and the amera. The longer the distane is, the larger the error is. Inside the reommended measurement range of our system, the error aused by lens distortion is usually in a range from 5 to 10 mm if a linear model is used. The error is most severe in the orner areas of the amera image. mm Pixel Pixel (a) (b) mm mm () Pixel Pixel (d) Pixel Figure 3.3: Error mas of measured flat board. (a) 3D model of a flat board at a near osition. (b) Error ma when the flat board is at the near-osition. () Cross setion of the near-osition error ma at the 450th row. (d) Error ma when the flat board is laed at a further osition. 39

56 To eliminate the error aused by lens distortion, a reasonable lens distortion model needs to be seleted with the right ombination of lens distortion oeffiients. The seond-order symmetri radial distortion model, whih uses only the first radial distortion oeffiient, is a viable model. It is often used when low distortion otial systems are used, or when only a few hekerboard images are available for alibration. Another ommonly used lens distortion model is the 4 th order symmetri radial distortion model with no tangential omonent. In our ase, Table 3.2 shows that the tangential omonents are signifiantly smaller than the radial omonents and the estimation unertainties of the 4 th order omonents are even larger than their orresonding error oeffiients. Therefore, the 4 th order radial distortion model is not reliable and should not be used. Due to these reasons, the seond-order symmetri radial distortion model is seleted for this researh. Aording to this model, the normalized oordinates of Eq. (3.2) beome: xd ( 1) 2 x d = = (1+ k(1) r ) x n 3.8 xd (2) One the distortion model is determined, the imortant issue is how to integrate this distortion model into the reonstrution algorithm. Obviously, Eq. (2.25) annot be alied diretly. Instead, a new relationshi an be found from Eqs. (3.1), (3.5) and (3.8). For onveniene, Eq. (3.1) is rewritten into the following exression: xn(1) X Z xn (2) Y = = w w w 1 1 Z [ R, t ][ X Y Z ] T 3.9 where R and t are the rotation matrix and the translation matrix given in Table

57 Table 3.3: Camera extrinsi arameters. Translation matrix t = [ ] Rotation matrix R = Using the intrinsi arameter matrix A and the ixel oordinates on the image lane [ v 1] u, through Eq. (3.5), [ x (1) x (2) 1] x = an be uniquely determined. d d By alying Eq. (3.8) to solve for the x n value, Eq. (3.9) an be used to alulate world oordinates[ X Y Z ]. w w w In Eq. (3.9) there are four unknowns but only 3 linear equations. Therefore, one more equation is needed to solve for the oordinates. Let us first assume that the lens distortion of the rojetor is not onsidered. Then Eq. (2.26) an be used to solve the equations. Assuming [ X w Y w Z w ] T = X w and x n = [ x n (1) x n (2) 1] and using the exressions in Eqs. (2.23) and (2.24), we have the equations on the amera side: n w [ R, t ][ X, ] T d Z x = Z x + w n = R X t 3.11 Z x ( i) R X + t, i = 1, 2, n 3 = k= 1 On the rojetor side, let [ u v 1 ] = I : ik w k i w [ R, t ][ X, ] T s I A 1 = 3.13 w s I = A R X + A t

58 w s I = E X + F 3.15 where E = A R, F = A t 3 w s I = E X + F, i = 1, 2, i k= 1 From Eqs. (3.12) and (3.16), we have: where H is a 3 3 matrix and b is a 3 1 matrix defined as: ik k i HX w = b 3.17 H ij = R3 x ( i) R, i = 1, j n ij H 3 j E3 j I1 F1 j = 3.19 bi ti xn ( i) t3 =, i = 1, bi F1 I1 F3 = 3.21 Then, X w 1 = H b 3.22 By using the new algorithm above that takes lens distortion into onsideration, the reonstrution of the world oordinates beomes muh more aurate. Figure 3.4 shows the error mas after the nonlinear algorithm is alied to the same data as that in Figure 3.3. The error aused by lens distortion has been signifiantly redued. When the flat board is not erendiular to the otial axis, some arts on the flat board are loser to the amera than other arts. Beause the oordinate error hanges with the distane from the amera to the objet, the error mas show in different shaes when the flat board is set in various orientations, as shown in Figure 3.5. Table 3.4 shows the RMS errors of the six 3D models before and after the amera nonlinear alibration algorithm is alied. 42

59 mm mm Pixel (a) Pixel (b) Pixel mm mm Pixel Pixel () (d) Pixel Figure 3.4: Error mas after the amera nonlinear alibration algorithm is alied. (a)-(b) The error at a near osition. ()-(d) The error ma at a further osition. From the analysis above, solving linear equations is still the last ste of the nonlinear alibration roedure. This is beause we inluded the lens distortion information in the normalized oordinate vetors, but not in the reonstrution alulation. This aroah searates the nonlinear art from the linear art, so that the relationshi between the amera and the rojetor is as simle as that of the linear model and the algorithm still has a low omlexity. 43

60 mm mm mm Pixel mm Pixel Pixel Pixel Figure 3.5: Error mas of the flat board measured in different orientations. Table 3.4: RMS errors of the flat board measured in different orientations. RMS (L): Linear alibration; RMS (CNL): Camera nonlinear alibration. 3D model RMS (L) RMS (CNL) 3D model RMS (L) RMS (CNL) #1 # #3 # #5 #6 44

61 3.3 Nonlinear Calibration of Projetor Projetor lens has the same lens distortion roblem as the amera lens. The fringe attern is distorted by the lens after being rojeted. The distortion has already been formed before the fringe reahed the objet. That is a reverse roedure omared to the amera. Taking the same idea introdued in the ase of amera lens, equations of the same format an be written. Combined with the equations for the amera as shown in Eq. (3.12), the rojetor lens distortion an be orreted in the same way. To alulate the intrinsi and extrinsi arameters, the method desribed in Chater 2 is used to ature more than ten rojetor images of the hekerboard. Then the Matlab alibration toolbox is used to estimate the arameters. The intrinsi arameters with different distortion models are shown in Table 3.5 and the distortion oeffiients are listed in Table 3.6. Table 3.5: Projetor intrinsi arameters with different distortion models. Linear Nonlinear only with k(1) Nonlinear with k(1) to k(4) f u f v u v α k(1)

62 Table 3.6: Projetor lens distortion oeffiients. k(1) k(2) k(3) k(4) Coeffiient ( ) Unertainty (±) ( ) When hoosing different distortion models, the distortion oeffiients obtained would not be exatly idential. In the ase of the amera, this situation is not signifiant and we ould easily hoose the otimal distortion model. However, the rojetor lens distortion oeffiients hange signifiantly in different models. When the distortion model with four oeffiients is hosen, even the first oeffiient is not reliable beause the unertainty is more than 50 erent. For the distortion model with just one oeffiient, we an obtain a reliable k(1) value, whih only has a unertainty of about 25 erent. Therefore, the same seond-order symmetri radial distortion model (only k(1) = is utilized) used for the amera lens is alied to the rojetor lens as well. Table 3.7 lists the extrinsi arameters from this model. Table 3.7: Projetor extrinsi arameters. Translation matrix t = [ ] Rotation matrix R =

63 Following the same roedure alied to the amera, through Eq. (3.5), oordinate vetor with distortion [ x (1) x (2) 1] Eq. (3.23), we obtain the normalized oordinate vetor system. x = is alulated first. Then from d d d x n in the rojetor oordinate xd ( 1) 2 x d = = (1+ k(1) r ) x n 3.23 xd (2) Using extrinsi arameters to write Eqs. (3.24)-(3.26) as following and ombining rojetor side equation Eq. (3.26) with the amera side equation Eq. (3.12), the world oordinates an be reonstruted. n w [ R, t ][ X, ] T Z x = Z + w xn = R X t 3.25 Z x ( i) R X + t i = 1, 2, n 3 = k= 1 ik w k i In order to alulate the normalized oordinate vetor in Eq. (3.5), ixel oordinates in both u and v diretions are required. Even though in the final ste only one omonent of the normalized oordinate vetor is useful, both of them have to be known in advane. Therefore, eight images are needed to solve for the ixel oordinates, four images for the u diretion and four for the v diretion. Figure 3.6 is the final error ma of a flat board with nonlinear alibration on both sides, as omared to results of linear alibration and results of nonlinear alibration only on the amera side. The error due to lens distortion is aarently further redued after alying nonlinear alibration to both the amera and rojetor lenses. The drawbak is that too many images are required and 47

64 the fringe atterns need to be swithed just for a single 3D data. Therefore, this aroah is not feasible for real-time measurement of slowly moving objets. One way to eliminate the need for these four additional images is to use an iterative reonstrution. With this aroah, only four images are atured, three fringe images lus one enterline image. When using Eq. (3.5), only one omonent of the normalized oordinate vetor, or the ixel oordinate in one diretion, an be determined. Instead of using four more additional images, we just assume that the unknown ixel oordinate is at the enter of the DMD image along the same diretion. For instane, in this researh the resolution of the rojetor is ixels, if we do not know the ixel oordinates in the diretion whih has 768 ixels, we just assume the value to be 384 no matter what it really is. This estimation will definitely introdue reonstrution error to the world oordinate result X w. However, the w X obtained from this first iteration is not the final result. Substitute [ X Y Z ] X = and then normalize w X into the following equation to find X to obtain a new x n : X Y Z = [ R, t ][ X Y Z 1] T w w w 3.27 Using x n in Eqs. (3.5) and (3.23) inversely, the ixel osition of this oint an be reovered in both u and v dimensions. Using these new ixel oordinates to alulate world oordinates again, a new w X is obtained. After several suh iterations, the result will onverge. This aroah uses an initial guess and borrows the onstraint from the amera side to determine the final world oordinates iteratively. Aording to our exeriene, iterating just one is adequate to ahieve an aetable auray. 48

65 Figure 3.6 (d) is the error ma obtained after a single iteration. Comaring with the result reonstruted by eight images, neither the error ma nor the world oordinates show obvious differenes. In this ase, however, the rojetor lens only has a small distortion. Otherwise, only iterating one may not be suffiient to generate an aetable result. The first aroah needs too many images and the seond aroah is not that effiient. To solve the roblem systematially with no more than four images, a novel aroah is roosed, in whih a ubi equation is used to relae those linear equations. Rewrite Eq. (3.4) to obtain the relationshi between the normalized oordinate vetor and the ixel oordinates as follows: x d x x (1) ( u = (2) ( v u ) / f = d 0 u d v0 )/ fv 3.28 Substituting Eq. (3.28) into Eq. (3.23), we have where r + u u f ( 0 ) / u 2 (1 k(1) r = + ( v v0 ) / fv ) x = ( X / Z ) ( Y / Z ) and x n = X /Z Y /Z n [ ] T, [ Y Z ] 3.29 X are the oordinates of a oint on the objet in the rojetor oordinate system. With four images, only one dimension of the ixel oordinate is known. Using the u-axis, Eq. (3.29) beomes a ubi equation: u e= u f u X Y X = 1 + k(1) Z Z Z Eq. (3.30) has three unknowns: X, Y and Z, whih are related to X w, Y w and Z w through the extrinsi arameters of the rojetor. Combining Eqs. (3.11), (3.25), and 49

66 (3.30), the oordinates of the objet oint an be solved in the rojetor oordinate system. Sine the extrinsi arameters are already known, the result an be easily onverted into oordinates in the world oordinate system by oordinate transformation. To show how to solve the equations for the oordinates of the objet oint, we first rewrite Eq. (3.27) to obtain the relationshi between Solving for X w, we have X + w X and X. That is w = R X t 3.31 X w ( R ) X ( R ) t 1 1 = 3.32 Substituting Eq. (3.32) into Eq. (3.11) yields Z x = R + n 1 1 ( R ) X R ( R ) t t 3.33 Let C=R (R ) 1 and D= R (R ) 1 t + t, where C is a 3 3 matrix and D is a 3 1 matrix. We have Z xn (1) X xn (2) = C Y 1 Y + D 3.34 Eq. (3.34) has 3 linear equations and 4 unknowns. Eliminate reorganize the other 3 unknowns in the following format: Z first and then X Y a' = a" Z b' + b" 3.35 where a' = [( 33 xn (2) 23 )( 32 xn (1) 12 ) ( 32 xn (2) 22 )( 33 xn (1) 13 )] / [( 32 xn n n n 21 (2) )( x (1) ) ( x (1) )( x (2) )] 3.36 b' = [( d1 d 3 xn (1))( 32 xn (2) 22 ) ( d 2 d 3 xn (2))( 32 xn (1) 12 )] / 50

67 [( 32 xn n n n 21 (2) )( x (1) ) ( x (1) )( x (2) )] 3.37 a" = [( 33 xn (2) 23 )( 31xn (1) 11) ( 31xn (2) 21 )( 33 xn (1) 13 )] / [( 31xn n n n 22 (2) )( x (1) ) ( x (1) )( x (2) )] 3.38 and unknown b" = [( d1 d 3 xn (1))( 31xn (2) 21 ) ( d 2 d 3 xn (2))( 31xn (1) 11 )] / [( 31xn n n n 22 (2) )( x (1) ) ( x (1) )( x (2) )] 3.39 Substitute Eq. (3.35) into Eq. (3.30) to form a ubi equation with only one Z. Rearranging this ubi equation, we obtain a ubi equation of a ommon form, with one ubi, quadrati, and linear term eah as well as a onstant term: where and 3 2 OZ + PZ + QZ + T = O= a' + ( a' ) k(1) + a' ( a") e P = b' + 3( a' ) b' k(1) + 2a' a" b" + ( a") b' Q = 3a'( b') k(1) + a'( b") + 2a" b' b" 3.43 T = ( b' ) k(1) b' ( b") 3.44 This ubi equation an be solved by Cardano s method. One an be substitute into Eq. (3.34) to solve for Z is obtained, it X andy. The last ste of this aroah is to use Eq. (3.32) to onvert the oordinates in the rojetor oordinate system [ X Y Z ] X = bak to the oordinates in the world oordinate w system [ X Y Z ] X =. w w w 51

68 Figure 3.6(e) shows the result of a flat board reonstruted using the ubi equation method. Aording to these omarisons, the reonstrution auray does not hange signifiantly no matter whih one of these three aroahes is alied. The ubi equation method is our final hoie for rojetor nonlinear alibration. Table 3.8 lists the RMS errors when the flat board is set in different orientations. From the linear alibration method to the nonlinear alibration method, the RMS error has been redued signifiantly from more than 1.4 mm to less than 0.35 mm. Figure 3.7 shows several 3D sanning data of some other statues. Table 3.8: RMS errors of the measured flat board in different orientations. Position linear Camera Iteration Cubi 8 images nonlinear nonlinear nonlinear nonlinear # # # # # # #7(8image)

69 mm mm (a) Pixel () Pixel mm mm (b) Pixel (d) Pixel mm (e) Pixel Figure 3.6: Error mas of a flat board measured with different alibration methods: (a) Linear alibration method. (b) Nonlinear alibration on amera only. () Nonlinear alibration on both amera and rojetor with the eight-image method. (d) Nonlinear alibration on both amera and rojetor with the iterative method. (e) Nonlinear alibration on both amera and rojetor with the ubi equation method. 53

70 Figure 3.7: 3D reonstrution results obtained by using the nonlinear ubi equation method. 3.4 Summary In this hater, nonlinear alibration methods were introdued to orret lens distortion for both the amera and the rojetor. Results obtained from different alibration models were omared to show the imrovement of the system auray. The eak-to-eak error aused by lens distortion was redued signifiantly from more than 10 mm to less than 3 mm. The RMS error was also redued from more than 1.4 mm to less than 0.35 mm, a redution of more than 75 erent. Due to hardware limitation, lens distortion an not be orreted omletely. However, the remaining error an be further redued by error omensation. This researh fouses on establishing the nonlinear alibration method. All the results shown in this dissertation are the results after nonlinear alibration with no other error omensations alied. The auray enhanement is entirely due to the introdution of the nonlinear alibration method. 54

71 Chater 4 Real-Time System Based on the Nonlinear Calibration Method Real-time 3D shae measurement has a huge otential in seurity, entertainment, manufaturing, et. due to its aability of sanning moving objets. Generally, there are two aroahes for real-time measurement: one is to use a single attern, a olor attern in most ases, and the other is to use quikly swithing multile atterns. In our researh, the real-time system is based on the seond aroah. A olor attern based on the hase shifting tehnique is rojeted by a DLP rojetor. When the rojetor swithes the red, green and blue olor hannels raidly, multile fringe images an be atured in a short eriod of time for real-time shae reonstrution. In order to reah a high seed, the system reviously used a fast reonstrution algorithm, whih was not as aurate as even the linear algorithm introdued in Chater 2. This researh fouses on the imlementation of the most aurate nonlinear algorithm in the real-time system to ahieve high auray while maintaining the high seed aability. 4.1 Real-Time System with an Enoded Color Pattern The flow hart of the multi-thread software for the real-time system is shown in Figure 4.1. The first thread grabs fringe images at a seed of 90 frames er seond. The seond thread retrieves the hase ma from the grabbed fringe images. The third thread reonstruts the 3D oordinates and then dislays it in on the sreen. 55

72 Aquisition 1 Aquisition 2 Aquisition 3 Phase wraing & unwraing 1 Phase wraing & unwraing 2 Phase wraing & unwraing 3 Reonstrution & dislay 1 Reonstrution & dislay 2 Reonstrution & dislay 3 Figure 4.1: Flow hart of the multi-thread real-time 3D shae measurement system. The enterline image is indisensable when the absolute hase ma needs to be determined. However, the rojeted attern annot be swithed between the fringe attern and the enterline attern fast enough beause it takes time for the rojetor to generate a stable rojetion. To aly the nonlinear alibration method in real-time measurement, the first issue that needs to be addressed is how to eliminate the need for the enterline attern while keeing the information it rovides, so that no attern swithing is required. The most straightforward idea is to embed the enterline in the fringe attern diretly. The highlighted enterline an be easily found, but the hase values at the ixels on the enterline annot be determined beause the intensity on these ixels does not hange. To kee the hase shifting information, in the meanwhile, highlight the enterline ixels, a short enoded enterline marker is used. Reall the definition of data modulation in Eq. (2.12), whih is the ratio of the intensity modulation and the average intensity or the intensity bias. Data modulation is useful for evaluating the quality of the data that has been olleted. A data modulation 56

73 near one is good and a data modulation near zero means a bad quality. In order to loate enterline ixels without losing their hase information, the data modulation value of the enterline ixels are intentionally redued by an aroriate amount so that they an be identified by their relatively lower data modulation values and in the mean time, their hase values an still be retrieved with suffiiently high signal-to-noise ratio. and Rewrite Eqs. (2.6)- (2.8) in the following forms: [ φ( x, α] I ( x, y) = [ I'( x, y) + δi ] + [ I"( x, y) δi]os ) y [ φ( x, )] I 2 ( x, y) = [ I' ( x, y) + δi ] + [ I"( x, y) δi]os y 4.2 [ φ( x, α] I ( x, y) = [ I' ( x, y) + δ I] + [ I"( x, y) δi]os ) y where δiis a onstant. In the new exression, the maximum intensity is still I+ ' I", but the minimum intensity is inreased by 2δ I and the average intensity is inreased byδ I. Consequently, the data modulation value at these ixels beomes lower. When data modulation is below some threshold, signal-to-noise ratio will suffer and as a result, the hase information may not be reliably retrieved. To avoid this roblem, the data modulation at the ixels on the marker needs to be maintained at a reasonable level to ensure that these ixels an be distinguished from other ixels and still allow for reliable hase retrieval. Figure 4.2 shows the enoded atterns being used. Figure 4.2 (a) is the original ideal olor attern, whih has a modulation near one. Figure 4.2 (b) is the modified attern based on Eqs. (4.1)-(4.3). Figure 4.2 () shows the short marker with 1 20 ixels. Figure 4.2 (d) shows a long marker with a width of 5 ixels and a length the same as that of the fringe attern. These markers an obviously be notied in the fringe images as shown in Figure

74 I'(x, y) I"(x, y) i'(x, y) i"(x, y) (a) (b) i'(x, y) i"(x, y) I"(x, y) I'(x, y) Centerline osition () (d) Figure 4.2: (a) Ideal fringe attern. (b) Modified fringe attern with higher intensity. () Fringe atterns with a short enterline marker. () Fringe atterns with a wide enterline marker. Figure 4.3: Markers in the fringe images. 58

75 To searh for the loation of the marker, data modulation at eah ixel needs to be alulated. Atually, aording to the relationshi between the amera and the rojetor obtained from system alibration, the marker, whih is loated at a fixed osition of the rojetor DMD hi, an be mathematially rojeted onto the amera image lane. Eqs. (2.25) and (2.26) onnet the amera and the rojetor with the world oordinate system. Using this relationshi, the marker an be maed to a line on the amera image lane (if the nonlinear model is used, the line would be a urve). This line is usually alled an eiolar line. By searhing along this line, we an avoid searhing through the entire image, thus imroving the seed as well as reliability. When imlementing the searhing roedure for the marker, we tried two different aroahes. The first aroah uses an interative searhing window. Sine on a real objet there may be areas with low surfae refletivity, shadow, and saturation, where data modulation an be as low as, if not lower than, that at the loation of the marker, an interative searhing window avoids otential searhing error and imroves searhing auray and reliability. Figure 4.4 shows this interative searhing roedure. In the default window, there may be some noise oints. If these oints are regarded as art of the marker, the loation of the marker annot be determined aurately, thus introduing error into the alulation of the absolute hase. The seond figure shows the searhing window after adjustment. The window is now smaller and the nose oints are all eliminated. Searhing the marker in suh a small area also enhanes the searhing seed. 59

76 (a) (b) Figure 4.4: Interative marker-searhing window. (a) Default window. (b) Adjusted window. (a) (b) Figure 4.5: Data modulation of the image with a rosshair marker. (a) 2D image. (b) Data modulation. 60

77 Another aroah is to use a small rosshair marker. As shown in Figure 4.5, this rosshair an be learly found in the data modulation ma even on human models. Beause a rosshair marker has a seial shae, it an be easily deteted by a seially designed filter. The searh area is also narrowed down to a long stri along the aforementioned eiolar line. From Eqs. (3.2) and (3.5), we an obtain the normalized oordinate vetor x n of the rosshair enter. Substituting it into Eq. (3.33), we have X Y Z = R x (1) n 1 ( R ) xn (2) Z 1 R ( R ) t + t After we obtain the oordinates in the amera oordinate system, using Eqs. (3.1), (3.2), and (3.5), the ixel osition [ v ] u of the enter of the rosshair marker on the CCD image an be found as funtions of Z. Figure 4.6 shows the marker searhing roedure. The brighter stri is the searhing area, the ross marker is always loated in this area. The texture ma needs to be modified beause the marker is visible in the texture ma obtained by averaging the three fringe images. To remove the marker from the texture ma, the summation of the average intensity and intensity modulation is used instead of the average of the three fringe images to generate the texture ma. From the result in Figure 4.7, we an see that the marker is omletely removed in the new texture ma. Figure 4.8 resents some seleted 3D models of a sequene of human fae sanning data to show the system s aability of measuring moving objets. Smooth faial exression hange is atured ontinuously. These data are taken in a frame rate of 30 frames er seond. 61

78 Figure 4.6: Automati searhing for the rosshair marker by using eiolar line. (a) (b) Figure 4.7: 3D data texture ma. (a) Old texture ma. (b) New texture ma. 62

79 Figure 4.8: Seleted 3D models of a real-time measurement sequene of human fae exression. 63

80 Hz System The highest otential sanning seed of this real-time system is 120 Hz, whih is the refresh rate of the rojetor. The seed of our urrent system is limited at 30 Hz due to the seed of the amera used. In this setion, we desribe the design of the next generation system, whih is simler and has a higher sanning seed of 60Hz. This new system uses a new amera with a VGA resolution of ixels and a new frame grabber (Matrox Solios) for ontrolling the amera. This new amera has many advantages when omared to the old one. First it an sustain a 180 frames/se image aquisition seed, whih means the ahievable 3D reonstrution seed an be u to 60 Hz. Figure 4.9 shows the timing hart of the 60 Hz system with this new amera. The system seed has been doubled as an be seen by omaring this timing hart with that of the 30 Hz system shown in Figure 2.7. Channels rojeted (120Hz) R G B R G B R G B R G B Trigger signal to ameras (180Hz) 2.78 ms Channels atured R B G R B G 13.9ms Figure 4.9: Timing hart of the 60 Hz System. 64

81 This amera also makes the onnetion interfae with the omuter muh simler. In Figure 4.10, the new amera and the old amera are shown together. It an be seen from the bak side of the ameras that the new amera only has one able, in addition to the ower able, while the old one has four. Further more, with this new amera, the trigger signal an be generated by software, thus eliminating the need for the miroroessor-based timing signal iruit. Finally, the ower suly is also muh simler than the old one. These advantages allow us to build our next generation system into a faster, simler, and more reliable system. Figure 4.11 shows human fae models taken by this new system. 4.3 Summary The real-time measurement auray was imroved signifiantly by emloying the new reonstrution algorithm based on the nonlinear alibration method. A marker embedded in the fringe attern was used for absolute hase measurement. This design eliminated the need for the additional enterline image reviously required, thus making it ossible to ahieve real-time seed at a higher auray. Even though the new algorithm is signifiantly more omlex than the old one, we managed to ahieved the same 30 frames er seond sanning seed through otimization of the algorithm. A new generation 60 Hz system is also designed. With new high-seed amera and synhronization method, the real-time measurement seed is doubled and the system design beomes simler, smaller and more reliable. 65

82 Figure 4.10: Front and bak views of the high seed amera. Figure 4.11: Human fae models sanned by the 60 Hz system. 66

83 Chater 5 Quality Ma Guided Phase Unwraing Algorithm The revious hase unwraing algorithm we used is a simle raster san like algorithm. When the measured objet has disontinuous features suh as shar edges, stes, et., the roblem of hase ambiguity will arise, whih leads to disontinuous surfae jums on the reonstruted 3D model. In this hater, a new hase unwraing algorithm, namely quality ma guided hase unwraing algorithm, is develoed to remove hase ambiguities for orret 3D reonstrution. 5.1 Phase Ambiguity To reate an objet, whih auses otential reonstrution errors due to hase ambiguity, we attah a two-inh high ste to a flat board. This ste has three sides erendiular to the board and the fourth side in a sloe. When a fringe attern is rojeted onto this objet as shown in Figure 5.1(a), hase disontinuity is reated, whih results in hase ambiguity. In this artiular osition, the uer side of the ste an still be seen with fringe attern, even though it has a shar surfae. In other words, the flat board and the ste are onneted on the uer side and the fringe attern image is ontinuous. However, on the lower side, the fringe attern is disontinuous, whih auses hase ambiguity and further reonstrution error as shown in Figure 5.1(b). Somewhere along the lower edge, the hase hanges are more than 2π, whih makes the hase unwraing algorithm unable to reover the hase orretly. The result is a severe jum in the 3D model reonstruted. 67

84 Continuous Disontinuous (a) (b) Figure 5.1: Phase ambiguity examle. (a) 2D fringe image with hase disontinuity on the lower edge of the ste whih will ause hase ambiguity. (3) 3D model with hase jum ause by hase ambiguity. 5.2 Quality Ma Guided Phase Unwraing Quality mas are arrays of values that desribe the goodness of ixels in a hase ma. Many features an be used as quality mas, suh as the orrelation ma, hase derivative variane, maximum hase gradient, et. In this researh, the quality ma used is the hase derivative, whih is defined as follows: Q i, j u 2 i, j v 2 i, j = u where i, j v and i, j are the hase differenes along the u and v axes resetively in the CCD image. Figure 5.2 shows the quality ma of the ste and the board. From this ma, we an see that the lower edge of the ste is highlighted with a higher hase differene, whih reresents a relatively lower quality. 68

85 Figure 5.2: Quality ma of the board with a ste. After the quality ma has been generated, all the ixels an be sorted into grous aording to their quality values during the hase unwraing roedure. The hase unwraing an be done starting from ixels with the highest quality and ending with ixels with the lowest quality. The flood-fill algorithm will sort and unwra the hase values at the same time. Figure 5.3 illustrates the quality ma guided flood-fill algorithm. At the beginning, all the ixel grous are emty. A starting ixel is seleted and the hase value of this ixel is regarded as the first unwraed hase value. This starting ixel is then stored in one of the ixel grous aording to its quality value. After this, the algorithm roesses the remaining ixels in an iterative way, eah time taking a ixel out from the ixel grou with the highest quality value, unwraing its four neighboring ixels if they have not been unwraed yet and utting eah of them into one of the ixel grous aording to their quality values. When the algorithm selets an unwraed ixel, it 69

86 always starts from the nonemty ixel grou with the highest quality value. If there is no ixel in this grou, it moves to the ixel grou with the seond highest quality value, and so on. Eventually the ixel grous will beome all emty again, whih means all the ixels have already been unwraed. By use of this algorithm, the low quality ixels, for examle, ixels along the lower edge of the ste, as shown in Figure 5.2, are bloked from being roessed until all the other ixels are roessed. As a result, low quality ixels would not affet high quality ixels and the hase ambiguity roblem is solved effetively. Figure 5.4 is the 3D model of the ste and the board reonstruted by using the new algorithm. Figure 5.5 shows another exerimental result of a human fae model. It is obvious from these results that the new hase unwraing algorithm is highly effetive in solving the hase ambiguity roblem, thus making 3D reonstrution more aurate. Starting Pixel New unwraed ixel Pixel Grou #1 Piked Pixel Pixel Grou #2 New unwraed ixel Pixel Grou #n Figure 5.3: Quality ma guided flood-fill hase unwraing algorithm. 70

87 Figure 5.4: 3D model of the ste on a board reonstruted by the quality ma guided flood-fill algorithm. (a) (b) () (d) Figure 5.5: 3D model of a human fae. (a) 2D image. (b) 3D model reonstruted by the old algorithm. () The quality ma. (d) 3D model reonstruted by the quality ma guided flood-fill algorithm. 71

88 5.3 Summary In this hater, a new quality ma guided flood-fill hase unwraing algorithm was introdued to eliminate the hase ambiguities aused by hase disontinuities larger than 2π. The reonstrution results obtained by using this new algorithm are shown in two examles. By emloying this new algorithm, the roblem of erroneous hase jums is suessfully solved in most ases. 72

89 Chater 6 The Combined Phase Shifting and Stereovision Method A tyial strutured light system based on the hase shifting method onsists of one amera and one rojetor. As an ative method, the hase shifting method does not require any markers on the objet surfae. This makes it ossible to aurately measure objets without muh of textural features, whih is usually diffiult with the traditional stereovision method. However, there are a few issues with the hase shifting method that need to be addressed before aurate measurement an be made. One is the sensitivity of its measurement auray to the nonlinearity of the rojetor s gamma urve. Many omensation methods have been develoed to solve this roblem. Zhang and Huang roosed a method that uses a look-u table (LUT) to intentionally distort the sinusoidal attern generated in the omuter so that the attern will beome sinusoidal one rojeted by the rojetor [91]. Zhang and Yau also roosed a method to use a LUT to omensate for the hase error diretly [92]. However, in order to build the look-u table, either the rojetor s gamma urve or the hase error due to the nonlinear gamma urve needs to be alibrated arefully with a time-onsuming roedure. Another issue with the hase shifting method is that the rojetor needs to be alibrated to determine its intrinsi and extrinsi arameters before measurement, whih is tyially more omliated than the alibration of a amera. Even though a systemati 73

90 alibration method for rojetor has been develoed by Zhang and Huang [83], rojetor alibration is still not as simle and aurate as amera alibration. In this Chater, a new aroah to 3D shae measurement is roosed, whih ombines the hase shifting and the stereovision methods [93], [94]. The aim is to redue the errors aused by inaurate hase measurement, for examle, the eriodi errors due to the nonlinearity of the rojetor s gamma urve, and also to eliminate the need for rojetor alibration. 6.1 Phase Errors of the Phase Shifting Method In a tyial hase shifting system, the orresondenes between the amera and the rojetor are found by mathing the hase values alulated from the fringe images taken by the amera and the hase values of the ideal fringe attern generated by the omuter before rojeted by the rojetor. However, due to various reasons, suh as the nonlinear gamma urve of the rojetor, the fringe atterns rojeted by the rojetor and the fringe atterns atured by the amera are not ideally sinusoidal anymore. Commerial rojetors are all intentionally designed to have a nonlinear gamma urve for visually leasing results. Using a rojetor with a nonlinear gamma urve, the ideal sinusoidal attern generated in the omuter will be distorted when rojeted onto the objet. As a result, errors will be introdued to the hase ma, whih in turn auses errors in the reonstruted 3D model. Figure 6.1 (a) shows an examle of a reonstruted 3D model, whih has eriodi vertial noises aused by the nonlinear gamma urve of the rojetor. Phase measurement errors are also ourred when the objet surfae has various olors eseially for surfaes with large olor ontrast. As shown at the lower right orner of Figure 6.1 (b), the test 74

91 objet is a ardboard box with blue bakground and white letters. On the reonstruted 3D model, we an learly see the errors around the edges of the letters. It should be noted that all the results obtained from tyial hase shifting method in this hater are measured and reonstruted without using any error omensation method. (a) (b) Figure 6.1: Examles of hase errors. (a) Error aused by nonlinear rojetor gamma. (b) Error aused by olor ontrast. 6.2 Combined Phase Shifting and Stereovision Method A new method, whih ombines the hase shifting and stereovision tehniques, is roosed for more aurate 3D shae measurement. This method uses two ameras and one rojetor and an eliminate errors aused by inaurate hase alulation suh as the eriodi hase errors due to the nonlinearity of the rojetor s gamma urve. 75

92 Fringe images Camera Fringe attern Digital Projetor 3D model Objet Camera Fringe images Figure 6.2: Shemati diagram of the system layout Priniles and system setu The shemati diagram of the system layout is shown in Figure 6.2. The basi system set u inludes two digital ameras, one rojetor, and one omuter. Sinusoidal fringe atterns are generated by the omuter and rojeted onto the objet via a digital rojetor. For better resolution, the hase shifting method with the three-ste algorithm is used, whih has been introdued in Setion Three fringe atterns are rojeted horizontally and then vertially. Two enterlines, the horizontal and vertial enterlines, are also rojeted for absolute hase alulation. Fringe images are taken from two different diretions simultaneously by the two ameras arranged as a stereovision air. 76

93 Both ameras are re-alibrated and their intrinsi and extrinsi arameters are known. The fringe images are then sent to the omuter for roessing via a frame grabber. Phase wraing and unwraing algorithms are alied to obtain the horizontal and vertial hase mas based on the three-ste hase shifting algorithm. These hase mas are then used to assist stereo mathing at the ixel level. The oordinates of the objet surfae are alulated based on triangulation. Sine the hase value at eah ixel is only used as a referene to assist stereo mathing, it does not have to be aurate. Thus the rojetor does not need to be alibrated, whih simlifies the system alibration. Errors due to inaurate hase values are signifiantly eliminated beause the two ameras rodue hase mas with the same hase errors and these errors are anelled after the mathing. Two absolute hase mas of a Zeus statue, one horizontal and one vertial, are shown in Figure 6.3. (a) (b) Figure 6.3: Absolute hase mas of Zeus statue. (a) Horizontal inreasing absolute hase ma. (b) Vertial inreasing absolute hase ma. 77

94 6.2.2 Pixel mathing and 3D model reonstrution In order to obtain aurate ixel mathing and eliminate errors due to hase inauray, we math ixels between two ameras in this roosed method instead of between one amera and one rojetor as in a tyial hase shifting method. The rojetor no longer artiiates in 3D reonstrution. As an be seen in Figure 6.3, the hase value is monotonially inreasing leftwards in the horizontal hase ma, and downwards in the vertial hase ma. Sine both absolute hase mas are monotoni, it is not hard to find the orresonding oints between the two ameras without any ambiguities. For a ertain ixel in one amera image, we an simly narrow down the searhing field on the other ameras image from 2D area to 1D array by using the horizontal hase mas, and then use the vertial hase values to loate the orresonding ixel in that array. In a tyial stereovision system, assuming the ameras are inhole ameras, the geometry relations between the two ameras at distint ositions lead to onstraints between the image oints. This geometry is so alled eiolar geometry. A very useful eiolar onstraint is shown in Figure 6.4. For an objet oint C, if the rojetion of C on amera1 is P 1, and the rojetion of C on amera2 is P 2, then P 2 must lie on a artiular line, whih is alled an eiolar line. Assume the intrinsi arameter matries for the two ameras are A 1 and A 2, and the extrinsi arameters, rotation matries and translation vetors, are R 1, t 1 and R 2, t 2 resetively, then the linear amera models for amera1 and amera2 an be written as and 1 1 T [ u v 1] A[ R, t ][ X Y Z ] T 1 s =

95 2 2 2 T s [ u v 1] A [ R, t ][ X Y Z 1] T 1 1 X Y Z is the oordinates for the objet oint, [ v 1] where[ ] oordinates of P 1. Solve [ Y Z] the equation for eiolar line on amera2. = 6.2 u is the rojetion X from Eq. (6.1) and substitute into (6.2) we obtain T T [ u, v,1] = A R R A s [ u, v,1] A R R t + A t s 6.3 Where s 1 2 is the indeendent variable and[ u 2 v 1] is a oint on the eiolar line based on the value of s 1. Using this linear equation (6.3), the searhing area for a ertain ixel an be onstraint to the adjaent 3 to 5 rows. In this researh, the amera resolution is ixels. By alulating the eiolar lines, the searhing area an be redued by roughly a hundred times. For better auray, the nonlinear seond-order symmetri radial distortion model, whih has been introdued in Chater 3, is used for ameras. And the reonstrution roedure is the same roedure as introdued in Chater 3, exet that this time we use the orresondene between two ameras, not one amera and one rojetor. C Eiolar line P 1 P 2 O 1 O 2 Camera 1 Camera 2 Figure 6.4: Eiolar onstraint used in this method. 79

96 6.2.3 Exerimental results The exerimental setu is omosed of two blak and white (B/W) ameras (Pulnix TM6740CL) with a resolution of ixels and one B/W rojetor (Plus Vision PLUS-U5-632) with a resolution of ixels. The two ameras are searated by a ertain angle and fixed on a metal frame. They are onneted to the omuter (Dell reision 690) via a frame grabber (Matrox SoliosXCL). This frame grabber has two amera link onnetors and allows for simultaneous ontrol of the two ameras. The ameras are re-alibrated and their intrinsi and extrinsi arameters are obtained by using a Matlab amera alibration toolbox. The rojetor is searated from the ameras and is not fixed on the frame. During measurement, the rojetor is set at a osition lose to the ameras to make sure that the objet is well illuminated by the rojeted light. And the rojetor is ket still during the measurement. Figure 6.5 shows two hotograhs of the atual system. (a) (b) Figure 6.5: The setu of the ombined hase shifting and stereovision system. (a) Cameras set u for stereovision. (b) Projetor used to rojet attern is searated from the ameras. 80

97 (a) (b) Figure 6.6: 3D models of (a) Zeus statue and (b) ard box reonstruted by the ombined hase shifting and stereovision method. Figure 6.6 shows the results of the Zeus statue and the ardboard box measured by the new system. Comaring with results shown in Figure 6.1, the errors aused by rojetor nonlinearity and olor variations are almost totally eliminated. In order to give a more quantitative omarison, a flat board was measured by both methods. Figure 6.7 (a) shows the error ross setion of reonstruted 3D model of the flat board measured by the hase shifting method and Figure 6.7 (b) shows the orresonding result measured by the ombined hase shifting and stereovision method. Sine the board surfae is smoother than the measured results show, the variations shown in the lots an be onsidered as mostly from measurement errors. The alulation of the RMS values shows an error redution of almost 9 times from mm to mm with the new method. The remaining error seems to be more like random error based. 81

98 (a) (b) Figure 6.7: Error ross setions of a flat board 3D model. (a) Measured by hase shifting method. (b) Measured by the ombined hase shifting and stereovision method. 82

99 To show the robustness of the new method, the fringe atterns were intentionally modified to make the intensities of the three hannels imbalaned and the intensity rofiles of the atterns non-sinusoidal, whih is a simulation of a severe nonlinear gamma urve. Figure 6.8 (a) shows the results obtained by the hase shifting method and Figure 6.8 (b) shows the results obtained by the ombined hase shifting and stereovision method under the same ondition. From left to right, the first set of results were obtained by fringe atterns with imbalaned intensities of 0.8I 1, I 2 and I 3, the seond with imbalaned intensities of 0.8I 1, 0.6I 2, and I 3, the third with imbalaned intensities of 0.8I 1, 0.6I 2, and 0.4I 3, and the fourth with randomly introdued nonlinearity. (a) (b) Figure 6.8: Results omarison of modified atterns. (a) Results measured by traditional hase shifting method. (b) Results measured by the ombined hase shifting and stereovision method. 83

100 In Figure 6.8, omare the first and the seond olumns, the errors are almost omletely eliminated by use of the ombined method. In the latter two olumns, there are some eriodi errors left, but they are muh smaller in omarison. In normal situations the errors due to intensity imbalane and gamma urve nonlinearity will not be as signifiant. The above results nevertheless show the robustness of the new method in terms of measurement reision. Besides, the fat that the new method does not require areful alibration of the rojetor makes it muh easier to imlement the method for 3D shae measurement. 6.3 Use of a Visibility-Modulated Fringe Pattern The ombined hase shifting and stereovision method introdued in the last setion an effetively eliminate errors aused by inaurate hase alulation. The disadvantage of this ombined method, however, is the doubling of the number of required images for 3D reonstrution, whih slows down the image aquisition roess and, as a result, makes it diffiult to measure dynamially hanging objets. In this setion, we roose to use a new visibility-modulated fringe attern to address the aforementioned shortoming of the ombined method. The aim is to redue the number of required fringe images to half, so that the ombined method an still be alied to measuring dynamially hanging objets Priniles The layout of the 3D measurement system introdued in this setion is the same as shown in Figure 6.2, exet that the fringe attern used is a novel visibility-modulated fringe attern. 84

101 As introdued in Chater 2, the ratio of the intensity modulation and average intensity defines the fringe visibility or data modulation. Fringe visibility an be used to hek the quality of data at eah ixel. When its value is lose to zero, the fringe has a low visibility, whih indiates low data quality. In ontrast, when its value is lose to one, the fringe has a high visibility, whih indiates high data quality. In this setion, the fringe visibility is modulated to hange eriodially for the urose of stereo mathing. The fringe attern shown in Figure 6.9 (a) is the newly roosed visibilitymodulated fringe attern. Along the horizontal diretion, this olor attern ontains the three hase-shifted fringe atterns in its red, green, and blue hannels. Along the vertial diretion, the fringe visibility is modulated in a triangular shae. Figure 6.9 (b) shows the sinusoidal waveforms at two different vertial ositions A-A and B-B as indiated in Figure 6.9 (a). The uer waveform shows a lower average intensity but a higher intensity modulation, whih ensures a higher fringe visibility. In ontrast, the lower waveform has a higher average intensity but a lower intensity modulation, whih means a lower fringe visibility. Theoretially the hase and visibility of a fringe attern are indeendent to eah other. Therefore, we an alulate the hase value orretly regardless the value of the fringe visibility. In reality, however, lower fringe visibility in general means redued signal-to-noise ratio. Therefore, the deth of visibility modulation for the fringe attern needs to be arefully seleted, so that it is large enough for stereo mathing, in the mean time, not too large to ause signifiantly inreased noise level. With this new attern, we an obtain the hase information in one diretion and fringe visibility information in the other diretion simultaneously for stereo mathing, so that the seond grou of fringe images is no longer required. 85

102 A A Phase jum I (x, y) I (x, y) B B A-A I (x, y) Phase jum I (x, y) (a) B-B (b) Figure 6.9: (a) Visibility-modulated fringe attern. (b) Sinusoidal waveforms at two different vertial ositions With hase wraing and unwraing algorithms, a horizontally inreasing hase ma an be obtained from the fringe images. In order to use hase values for stereo mathing at the ixel level, the hase mas need to be transformed into absolute hase mas. This means that there have to be some kind of ommon referenes in both hase mas. Previously, we used an extra enterline image to rovide the referene, whih slows down the image aquisition seed. In hater 4 a rosshair marker with higher data modulation value is used for referene. However, in this new visibility-modulated fringe attern, the data modulation is already used for other urose, so the same aroah in Chater 4 is not aliable. 86

103 Center line Center line Curve with the same hase value A Searhing area for A (a) Visibility modulation hanging eriod A A Searhing area for A (b) Figure 6.10: Stereo mathing roedure. (a) Using the hase mas to find the same hase urve (b) Mathing the visibility values to loate the right ixel. 87

104 In this researh, the enterline is embedded in the fringe attern as a 30-degree hase jum, as shown in Figure 6.9 (b). During the hase wraing roess, this hase jum is deteted in a small window as shown in Figure 6.10 (a). By defining the hase values at the hase jum lines to be zero, a ommon referene in the hase mas from both ameras is established, whih makes it ossible to onvert the hase mas into absolute hase mas. As a result, only three images are required for 3D model reonstrution, whih makes ossible the measurement of moving objets. With the absolute hase mas available, for a ertain hase value of a ixel A in the image of one amera we an find a urve with the same hase value on the image of the other amera. To avoid mathing ambiguity and inrease the mathing seed, the eiolar geometry of the ameras is used to narrow down the searhing area to a narrow stri, whih is narrower than the eriod of the visibility modulation, as shown in the dotted-line-irled areas in Figure 6.10 (a) and 6.10 (b). One the urve is found in the hase ma, the visibility mas are alied to identify the orresonding ixel Exerimental Results The exerimental setu is the same as introdued in setion 6.2.3, exet for the fringe attern used. The reonstrution roedure has not been hanged neither and the amera model used is still the seond-order symmetri radial distortion model. To demonstrate the erformane of the roosed method in terms of error redution, the Zeus statue and a ardboard box were measured by using the visibilitymodulated fringe attern. Figure 6.11 shows the results. Comare to Figure 6.1, most of the errors are eliminated by using this new attern. 88

105 Figure 6.11: Results measured with visibility-modulated fringe attern. Figure 6.12: Cross setions of a flat board 3D model measured with the visibilitymodulated fringe attern. 89

106 The flat board is also measured with this new attern and a ross setion is shown in Figure The RMS error is mm. Comaring to Figure 6.7 (a), the error is redued by 5 times whih is not as good as Figure 6.7 (b) (whih redued the error by 9 times), however, the aquisition time for reonstruting a single 3D model is largely redued to 13.9 ms as the timing hart shown in Figure 4.9. To show the system s ability of measuring dynamially hanging objets, we measured a data sequene of human fae exression at a frame rate of 60 frames er seond for 8 seond, whih resulted in a total of 480 reonstruted 3D models. Figure 6.13 shows 8 seleted frames of all these 3D models, whih learly demonstrate the aability of the roosed method in aturing dynamially hanging faial exressions. Figure 6.13: 8 seleted 3D models of dynamially hanging faial sequene atured by using the ombined hase shifting and stereovision method with the visibility-modulated fringe attern. 90

107 6.4 Summary This Chater has introdued a new method, namely the ombined hase shifting and stereovision method, for more aurate 3D shae measurement. This method requires the use of two ameras and one rojetor. By using this method any errors due to inaurate hase measurement ould be signifiantly redued. Exerimental results showed that the tyial eriodi error due to rojetor nonlinearity was almost omletely eliminated. The use of visibility-modulated fringe attern has also been roosed. By using this new fringe attern, the image aquisition time for three ontiguous fringe images ould be redued to 13.9ms, thus made the measurement of moving objets ossible. 91

108 Chater 7 Color System Based on the Combined Phase Shifting and Stereovision Tehnique Real-time systems have been develoed in revious researhes as introdued in Chater 4 and Chater 6, whih demonstrate the aability of measuring slowly moving objets, suh as human faial exressions. However, when measuring raidly moving objets, these B/W systems showed signifiant measurement errors due to their sequential nature of fringe image aquisition. As introdued in Setion 1.2.2, methods using olor-enoded fringe atterns have been reorted reviously for use in measuring raidly moving objets. In this hater, we further the work on the olor hase shifting method by adoting the ombined hase shifting and stereovision method. In artiular, we roose to use a olor visibilitymodulated fringe attern and olor devies to rea the auray advantage offered by the ombined hase shifting and stereovision method, while keeing the seed advantage of a olor hase shifting system [95]. 7.1 Motion Error of B/W System Sine multile images are required for the hase shifting method, the objet is usually needed to be motionless. If the images an be atured fast enough, slowly moving objets an be measured. In the real-time system, three hase-shifted fringe atterns are enoded in the red, green, and blue hannels to form a olor fringe attern. 92

109 Figure 7.1: Shemati diagram of the error aused by objet motion. When this attern is rojeted onto the objet by a digital-light-roessing (DLP) rojetor working in B/W mode, the three olor hannels are rojeted in graysale sequentially and eriodially. By synhronizing the amera and rojetor, the three fringe images an be atured in 13.9ms. At this seed, slowly moving objets, suh as human fae with exression hanges, an be reorded as shown in revious haters. However, for fast moving objets, errors due to objet motions still aear on the reonstruted 3D models. Figure 7.1 shows the shemati diagram of the ausing of motion errors in the ombined hase shifting and stereovision method roosed in Chater 6. In order to 93

110 reover the hase ma, three fringe images are needed for eah amera. As illustrated, for the measurement of a ertain oint C, due to the objet osition hanging, the atually oints taken by amera1 are C, C and C ", and the orresonding oints taken by ' 1 1 amera2 are C, ' 2 C and C " resetively. The traditional hase shifting system also has 2 this roblem, exet that that kind of system only uses one amera. 7.2 Color Based Aroah Color-enoded fringe attern The roblem aused by objet motions an be solved by using a olor liquidrystal-dislay (LCD) rojetor and one or more olor ameras. Different from DLP rojetors, LCD rojetors rojet the three olor hannels simultaneously. The three hase-shifted fringe atterns an be enoded in the red, green, and blue olor hannels to form a olor fringe attern and rojeted at the same time. Then only one olor image needs to be taken by the olor amera. The red, green, and blue omonents of this olor image an be searated to generate the three fringe images for hase ma alulation. Sine only one image is atured by the amera, the image aquisition time is signifiantly redued and therefore objet motions will have muh less effet on the measurement results Color imbalane and olor ouling The seed advantage of the olor system omes with roblems assoiated with the use of olor, whih need to be solved before aurate measurements an be made. One of 94

111 the major roblems is olor imbalane. Figure 7.2 shows the ross setional intensities of three olor fringe images. In this ase, the single olor fringe atterns were generated with the same intensity and rojeted onto a white board via an LCD olor rojetor. The fringe images were atured by a olor amera. As an be seen in this figure, the green fringe image has a muh larger intensity over the red and blue fringe images. This intensity disreany is alled olor imbalane, whih exists in both the amera and rojetor. The olor imbalane roblem an result in signifiant hase errors in the measurement results. Comensation methods have been develoed in the ast. For examle, Pan and Huang roosed a method that used a look-u-table (LUT) to solve this roblem [62]. Color ouling, whih is aused by the setrum overlaing of the olor hannels, is another issue that needs to be addressed. The olor hannels are usually intentionally designed to have setrum overlas in order to revent olor-blind regions. As a result, the red, green, and blue hannels of a olor fringe image annot be searated omletely. Color ouling is usually severe between green and red hannels as well as green and blue hannels. It is relatively weaker between the red and blue hannels. Figure 7.3 shows the ross setions of a green fringe image. As an be seen in the figure, the red and blue omonents also have intensities that are not negligible. The olor ouling roblem an be solved by using either omensation algorithms or seially designed olor filters to hange the olor setra of the amera [61], [62]. 95

112 Figure 7.2: Color imbalane of the three olor hannels. Figure 7.3: Color ouling aearane in a green fringe image. 96

113 7.2.3 Use of a olor visibility-modulated fringe attern The olor 3D shae measurement system is omosed of a olor LCD rojetor whih is used to rojet the visibility-modulated fringe attern and two olor ameras whih are used to ature the fringe images of the objet. In this system, eah amera only needs to take on olor fringe image for 3D reonstrution. Then eah olor fringe image an be searated into three graysale fringe images for alulation of the hase values. In this way, we an signifiantly redue the image aquisition time and make the system more suitable for measuring objets in motion. In general, the olor imbalane and olor ouling roblems ause signifiant errors in the traditional hase shifting method. Fortunately, the same thing does not haen in the ase of the ombined hase shifting and stereovision method. We will show in the next setion that errors aused by olor imbalane and olor ouling are signifiantly redued in the same way as the errors aused by hase misalulation due to the rojetor s nonlinearity in the B/W system. 7.3 Exeriments System setu The olor system has the same onfiguration as the B/W system shown in Figure 6.2, exet that all the devies used are olor devies and the rojetor works in olor mode. The ameras are single-ccd Bayer ameras with a resolution of ixels. The rojetor is an LCD rojetor with a resolution of ixels. The two ameras are re-alibrated by using the Matlab amera alibration toolbox and are fixed 97

114 in osition relative to eah other. A frame grabber is used to synhronize and ontrol the ameras. The rojetor is neither alibrated nor fixed relative to the ameras. It an be laed at any osition as long as it an roerly illuminate the objet Comensation methods Use of the ombined hase shifting and stereovision method an signifiantly redue the systemati hase errors. However, when the nonlinearity of the rojetor s gamma urve or the olor imbalane is too large, there will be slight errors left on the reonstruted 3D model, as illustrated in Figure 6.8. To further redue the hase errors and obtain otimal results, a basi omensation for the nonlinearity of the rojetor s gamma urve is adoted. An LUT was built to generate a alibrated fringe attern. Sine we only need a rough alibration, this roedure is needed to be done for only one and the result is reorded. To solve the severe olor imbalane roblem, the intensities of three different hannels are multilied by three different oeffiients. Also, when the attern is generated, the green hannel is assigned with a lower intensity than the other two hannels. Even though the imbalane ratios are not quite the same over different graysale levels, exeriment shows that this simle adjustment is suffiient for the ombined hase shifting and stereovision method to retify the olor imbalane issue. As for the olor ouling roblem whih is not as severe as olor imbalane in our ase, no omensation was done at all. From the measurement results, whih will be shown in the next setion, the olor ouling issue will not affet the measurement erformane, when the ombined method is alied. 98

115 (a) (b) Figure 7.4: Measurement results of (a) the olor hase shifting method, and (b) the roosed ombined hase shifting and stereovision olor system Exerimental results Figure 7.4 shows the advantage of the olor ombined hase shifting and stereovision method over the traditional olor hase shifting method. The Zeus statue and a lasti art, whih was ainted in white, were measured to show the effetiveness of 99

116 this olor system. Figure 7.4 (a) shows the results measured by the traditional hase shifting system omosed of one olor rojetor and one olor amera. Even though the same omensation methods were used, signifiant errors still aear on the reonstruted 3D models. Figure 7.4 (b) shows the results obtained by the ombined hase shifting and stereovision system where no obvious systemati errors an be observed. The hase errors aused by the nonlinearity of the rojetor s gamma urve, olor imbalane, and olor ouling are mostly eliminated. To show the system s aability of measuring objets in motion, a laster statue of a girl s rofile and the white ainted lasti art were measured by the B/W system and the olor system both based on the ombined hase shifting and stereovision method, while being quikly moved in front of the systems. As shown in Figure 7.5 (b), the measurement results obtained by the olor system are not obviously influened by the motion, while the results obtained by the B/W system shown in Figure 7.5 (a) have large distortions. The exosure time of the olor ameras was set at aroximately 5ms in this ase, but it ould be adjusted deending on the brightness of the illumination and sensitivity of the ameras. It is ossible to adjust the exosure time freely beause that the ameras no longer need to be synhronized with the rojetor. Comared to the total aquisition time of 13.9 ms required by B/W system, the olor system is more than twie as fast. Besides, this seed advantage will be even more signifiant if the exosure time is further redued with higher brightness of the rojetor or higher sensitivity of the ameras. 100

117 (a) (b) Figure 7.5: Measurement results of objets in motion by ombined hase shifting and stereovision systems. (a) Results obtained by B/W system. (b) Results obtained by olor system Disussion The results obtained by the olor system showed a lower resolution than those from the B/W system. This was beause the olor ameras used in this researh are single-ccd Bayer ameras, whih has 50% ixels for green and 25% eah for red and 101

118 blue hannels. When a olor image from suh a amera is searated into three graysale images, the missing ixels are filled by interolation. Figure 7.6 (a) is the red hannel of a olor fringe image taken by a Bayer amera. The noise level is obviously higher than the image taken by a 3-CCD amera, whih is muh learer and smoother as shown Figure 7.6 (b). Using 3-CCD ameras, the measurement results are exeted to be signifiantly imroved. However, a 3-CCD amera is tyially muh more exensive than a single-ccd amera. The olor system is also sensitive to ambient light and objet surfae textures. During the measurement, all the indoor lights need to be turned off for aurate results. In addition, the objets measured so far all have white and diffuse surfaes. Figure 7.6: Red hannels of two olor images taken by different olor ameras. (a) Image taken by single-ccd amera. (b) Image taken by 3-CCD amera. 102

119 7.4 Summery In this Chater, a olor system omosed of two olor ameras and one olor rojetor was roosed to further imrove the image aquisition seed for the ombined hase shifting and stereovision method. With the use of these olor devies, only one fringe image is required for eah amera to reonstrut a 3D model. As a result, the errors aused by objet motions were signifiantly redued. Exerimental results were resented and omared. Moving objets were measured by different systems to show the resistane of the olor system to objet motions. The olor system has a relatively lower resolution sine the olor ameras used are single-ccd Bayer ameras. The measurement results an be imroved by using more exensive 3-CCD ameras. 103

120 Chater 8 Portable 3D Measurement System The usual fringe rojetion systems have a disadvantage that blind areas not atured by ameras or shading areas not illuminated by rojetion an not be reonstruted. Problems also exist when measuring large objets. When the measured objet is larger than the system s field of view, multile data iees need to be aquired from different ersetives to over the whole objet. To stith these data iees together, registration methods are required In this Chater, a ortable 3D shae measurement system based on the ombined hase shifting and stereovision method is roosed to measure objets with large sizes [96]. Simle registration roedure is also designed for stithing loal views. 8.1 Priniles System setu and measurement strategy The shemati diagram of the system layout is shown in Figure 8.1. The basi system setu is the same as introdued in Setion whih inludes two B/W ameras, one DLP rojetor working in B/W mode, and one omuter. The visibility-modulated fringe attern is also utilized. During the measurement, the rojetor must be loated at the same osition and relatively fixed to the objet. The two re-alibrated ameras as a whole art an be moved to as many ositions as needed to obtain enough loal views that over the whole 104

121 objet. Eah loal view should have overlaing areas with other adjaent views. The 3D models are reonstruted in their own loal oordinate systems. The rojetor is fixed during the measurement roedure, whih means that the fringe attern aearing on the objet is not hanged, so that the hase and fringe visibility values of the ixels in the overlaed areas is onsistent and an be used to find the transformation between the loal views. Finally, all of the loal views an be transformed into one global oordinate system to from a omlete 3D model. Objet Cameras at Position1 Cameras at Position2 Fringe Images Comuter Fringe Pattern Projetor Fringe Images Comleted 3D model Loal views Figure 8.1: Shemati diagram of the ortable system. 105

122 To build a ortable system, the measurement seed needs to be as fast as ossible, beause any small ositional hange of the measurement system aused by hand motions during the sequential image aquisition roess will result in motion errors in the reonstruted 3D model. By using the visibility-modulated fringe attern, whih is fully introdued in Setion 6.3, a loal view an be atured in 13.9 ms whih is fast enough for the seed requirement Data Registration An effiient data registration roedure is designed to stith the loal views. To show this roedure, a laster statue of woman fae is measured in two views, eah ontaining a different art of the statue with some overlaing areas with the other. The two images shown in Figure 8.2 were taken by the same amera but at different ositions. The reonstruted areas for eah view are highlighted and the original 3D oordinates are reresented in their own loal oordinate systems. The short red line in eah image is the enterline deteted as a 30 degree hase jum, whih is used to alulate the absolute hase ma. Sine the rojetor and the objet are not moved during the whole roedure, the atterns rojeted on the objet in these two images are idential. With the absolute hase ma and the fringe visibility ma, the orresonding ixels in the overlaing area ould be found (shown as x markers in Figure 8.2). Using the hase and fringe visibility values to assist ixel mathing, no hysial markers or surfae textures are required. One the ixel mathing is omleted, the 3D oint airs ould be used to alulate the transformation between these two loal views. 106

123 A attern ontaining more referene lines, shown in Figure 8.3, is used in order to obtain a wider measurement range for even larger objets. For any two views, if an idential referene line is found, the absolute hase ma an be alulated based on that line for stereo mathing. Figure 8.2: Finding orresonding ixel airs for two loal views. Figure 8.3: Pattern with multile referene lines. 107

124 8.2 Exerimental Results The reonstruted 3D models of the laster statue are shown in Figure 8.4. Figure 8.4 (a) and (b) are the two loal views. Figure 8.4 () shows the two loal views dislayed together before oordinate transformation. Figure 8.4 (d) is the omlete 3D model after the oordinate transformation. It an be seen learly that the two loal views have been aurately merged together by using this method. (a) (b) () (d) Figure 8.4: Exerimental results of a laster statue. (a) The first loal view. (b) The seond loal view. () Loal views before oordinate translation. (d) Merged loal views after oordinate translation. 108

125 Figure 8.5: Measurement results of a metal art. Figure 8.5 shows the measurement results of a long metal art. Three loal views were taken for this art and the 3D models were transformed into the same global oordinate system. The uer iture shows the oint loud of the three loal views, in whih the overlaing areas are learly seen. The lower iture is the stithed 3D model of the whole art. A laster seahorse attahed to a flat white board was also measured in 9 loal views and the results are shown in Figure 8.6. Figure 8.6 (a) shows the oint louds with different loal views dislayed in different olors. Figure 8.6 (b) is the stithed 3D model. It an be seen that using the roosed registration method, all the iees an be suessfully merged together to form the whole model without the hel of any hysial 109

126 markers or surfae textures. Figure 8.6 () and (d) are the zoom-in view of the entral art and its orresonding osition in the whole model. Finally, Figure 8.7 shows the measurement results of a ar fender. Figure 8.7 (a) shows the whole 3D model with olored loal views. Figure 8.7 (b) is the whole model dislayed in a slightly different angle with the same olor. This ar fender is ainted with diffuse white olor, sine its original olor is blak and light refleting whih an not be easily measured by otial tehniques. (a) (b) () (d) Figure 8.6: Measurement results of a laster seahorse attahed on a flat board. (a) Point louds of 9 loal views. (b) The whole 3D model. () The entral loal view. (d) The entral loal view in the whole model. 110

127 (a) (b) Figure 8.7: Exerimental results of a fender. (a) The whole 3D model with olored loal views. (b) The whole 3D model shown in one olor. 8.3 Summary In this hater, we introdued a ortable 3D shae measurement system based on the ombined hase shifting and stereovision method, whih an measure large objets. During the measurement, the movable art is just the two ameras onneted by a metal frame, whih is small and light enough to be held in hand. The image aquisition time is aroximately 13.9 ms for a single loal view, whih is short enough to avoid motion errors. The oordinate transformations between these loal views ould be determined by identifying orresonding oint airs in the overlaing areas without the need of any hysial markers and surfae textures. Several measurement results were resented to show the feasibility of this ortable 3D shae measurement system. 111