PHOTOGRAMMETRIC ANALYSIS OF ASYNCHRONOUSLY ACQUIRED IMAGE SEQUENCES

PHOTOGRAMMETRIC ANALYSIS OF ASYNCHRONOUSLY ACQUIRED IMAGE SEQUENCES Karsten Raguse 1, Chrstan Hepke 2 1 Volkswagen AG, Research & Development, Dept. EZTV, Letter Box 1788, 38436 Wolfsburg, Germany Emal: karsten.raguse@volkswagen.de 2 Insttute of Photogrammetry and GeoInformaton, Unversty of Hannover, Nenburger Straße 1, 3167 Hannover, Germany Emal: hepke@p.un-hannover.de Abstract: The three-dmensonal photogrammetrc analyss of dynamc processes usng mage sequences represents a growng feld of applcaton for dgtal photogrammetry. An mportant precondton s the use of accurately synchronzed cameras. In most applcatons the synchronzaton of the cameras s realzed by external master clocks. Other approaches use stereo-beam splttng for the synchronous acquston of mage sequences. In ths artcle a new method for the synchronzaton of measurements of asynchronously acqured mage sequences s presented. In contrast to the other mentoned procedures our approach models the asynchronsm wthn the photogrammetrc analyss nstead of usng addtonal hardware components. We model the asynchronsm wth a lnear approach. The constant part s called tme-offset and the lnear part s called temporal drft. These two parts are combned and converted to an nterpolaton factor. Ths factor s ncluded n the functonal model as a temporal correcton term and s regarded as an unknown parameter n an extended bundle adjustment. Due to the temporal nterpolaton, measurements from successve epochs are needed. Because of modelng the asynchronsm terms n the analyss, the accuracy of the three-dmensonal object pont determnaton from mage sequences s sgnfcantly mproved n contrast to procedures whch neglect the asynchronsm. Also, usng the suggested method mage sequences of cameras, whch due to techncal reasons cannot be synchronzed by external hardware, can be processed. We have mplemented the suggested method and have run a number of experments n the context of vehcle mpact testng. The test seres confrm the theoretcal expectatons of the new method. Wth a frame rate of 1 Hz, an object speed of up to 7 m/s and an asynchronsm of.8 ms the accuracy of the object coordnates can be mproved approxmately by the factor 1. 1. Introducton The three-dmensonal photogrammetrc analyss of dynamc processes represents a growng feld of applcaton for dgtal photogrammetry. An mportant precondton s the use of accurately synchronzed cameras. Lke some other components of the optcal data channel the synchronsm of the camera systems represents an mportant nfluence factor for the accuracy

of three-dmensonal pont determnaton from mage sequences [5]. The determnaton of three-dmensonal movements from mage sequences s used n dfferent areas of applcaton, e.g. for the dynamc determnaton of wave surfaces [7], for the analyss of car and passenger movements n the context of vehcle mpact testng [6], for human body moton capture from mult staton vdeo sequences [1] and for the so called Partcle Trackng Velocmetry n lquds [3], [9] or gases [4]. For the synchronzaton of the camera systems two dfferent procedures have manly been used n recent years. In most applcatons the synchronzaton of the cameras s guaranteed by an external contnuous synchronzaton trgger sgnal. Wth ths sgnal all startng ponts of exposure tme of the cameras are controlled. The accuracy of ths synchronzaton method depends on the nternal tme delay of mage acquston n the ndvdual camera systems. The second possblty of producng synchronous mage sequences s the use of a beam splttng devce n front of the lens of the camera. [2] use a stereo beam spltter for the acquston of three-dmensonal object movements n pedestran protecton testng n the car ndustry. In [4] a beam spltter whch smulates four vrtual camera postons s used for Partcle Trackng Velocmetry n gases. The beneft of a beam spltter n front of the lens s that exactly synchronous cameras are smulated, however, wth the dsadvantage that per mage sequence only the half or a fourth of the sensor sze of the camera s avalable. A further dsadvantage of a beam splttng system s the fxed camera setup: The confguraton s specfed by the mrror setup of the beam spltter. Apart from usng hardware components to obtan synchronous mage sequences there are also some approaches to synchronze the measurements of asynchronously acqured mage sequences n the analyss. [8] use the slhouettes of movng objects. The basc concept of ths approach s to determne the tangent envelope of the movng object and then fndng correspondng tangents n the mage sequences. The result s the temporal offset between the two cameras. The method s used wthn a mult-camera shape-from-slhouette system. Another approach [1] explots the correlaton of space-tme nterest pont dstrbuton n dfferent mage sequences of the same scene and acheves synchronzaton wthout any mage feature correspondence. The approaches [8] and [1] deal wth non-convergent camera constellatons. They also do not handle the temporal drft. Only the constant part, the temporal offset s calculated. In ths artcle an alternatve possblty for the photogrammetrc analyss of asynchronously acqured mage sequences s presented. Instead of usng hardware components, the synchronzaton s embedded nto the photogrammetrc analyss of the asynchronously acqured mage sequences. The asynchronsm of the camera systems s regarded as an unknown parameter n an extended bundle adjustment. Ths new method was developed for the photogrammetrc analyss of hgh dynamc processes. The man applcatons are the analyss of vehcle mpact testng and pedestran protecton testng for car development. The scenaro s maged wth up to eght dgtal hgh-speed cameras n a crcular setup around the measurng area. The cameras acqure mage sequences wth a frame rate of 1 Hz and the testng objects move wth a speed of up to 18 m/s [6]. The work flow of such a synchronzaton method and expermental results are descrbed n detal n the followng sectons.

2. New approach for the analyss of mage sequences 2.1. Requrements and bass concept Due to the ntended area of applcaton for ths method, there are some requrements for the analyss of mage sequences: sutable for hgh dynamc applcatons, no use of hardware components for the synchronzaton, same accuracy level as the analyss obtaned when usng hardware components for the synchronzaton, user-defned camera acquston network and use of dfferent types of cameras n one network. To meet all these requrements the synchronzaton of the measurements of asynchronously acqured mage sequences s carred out wthn the photogrammetrc analyss. Thus, the problem of the synchronzaton of the cameras s solved by a mathematcal approxmaton wth correcton functons. Note that the same theoretcal prncple s used for the determnaton of the dstorton parameters wthn a camera calbraton. In ths approach the asynchronsm s consdered n form of an nterpolaton factor and ntroduced as an unknown parameter n the bundle adjustment. 2.2. Temporal components of the optcal data channel Dfferent effects of the optcal data channel are denoted as temporal components. Wth respect to the dfferent cameras these are constant tme dfferences between the tmes of acquston, dfferent tme delays durng sgnal transmsson, dfferent exposure tmes and dfferent frame rates. Independent of the actual reasons all these effects wll be consdered as part of the term asynchronsm. For modelng the asynchronsm a lnear approach s used. The constant part s called tme offset and the lnear part s called temporal drft. These two values are determned for every camera and ndcate the tme dfference to a reference system. The reference system can be an external tme measurng system or one of the cameras of the camera network, whch should be synchronzed. 2.2.1. Tme-offset The effects of dfferent exposure tmes and other constant tme dfferences between the cameras are denoted as tme-offset of an magng system. The tme-offset s a value, whch s ndcated for each magng system relatve to the reference system and s constant over the analyzed tme nterval. Ths offset often consttutes the man part of the effects of the temporal components. 2.2.2. Temporal drft The temporal drft s the part of the temporal components, whch changes wth ncreasng recordng tme and causes a change of the asynchronsm. Ths factor can be traced back to dfferences n the quartz frequences, whch are responsble for clockng the magng systems. It s assumed that each ndvdual quartz frequency s constant over the entre tme nterval. Therefore the effect of the temporal drft can be modeled as a lnear functon, dependent on the tme snce the trgger sgnal for the synchronzaton was started:

1 1 t Drft = (1) f f Ref t Drft Temporal drft of the camera [sec] f Ref Frame rate of the reference system [Hz] f Frame rate of the camera to be synchronzed [Hz] 2.2.3. Asynchronsm The tme offset und the temporal drft can be combned to yeld the asynchronsm: t( t ) = t + ( t t ) f t (2) Offset Drft t t ) Asynchronsm of the camera [sec] t ( t Offset Tme offset of the camera [sec] Tme step of the magng sequence [sec] t Tme step of the last synchronzaton pulse [sec] 2.3. Modelng of the temporal components wthn the photogrammetrc analyss 2.3.1. Basc concept If the mage recordng devces acqure accurately synchronous mage sequences, the analyss of the mage sequences can be done analogue to the analyss of statc photogrammetrc mages: For each epoch the mages can be analyzed separately and the three-dmensonal object coordnates are then calculated. In the presence of asynchronsm, however, ths method leads to wrong results. In the presented approach the asynchronsm s modeled by nterpolatng between the measurements of sgnalzed ponts n dfferent epochs. Therefore, measurements of dfferent epochs are needed for the analyss of one tme step. The requred nterpolaton factor s regarded as an unknown parameter and s ntroduced as a temporal correcton term n the functonal model of the extended bundle adjustment. The consderaton of the temporal components n the photogrammetrc analyss s carred out n mage space. The beneft s that only the measurements of the sgnalzed ponts n mage space are needed. No assumptons about object speed or movng drecton must be made. 2.3.2. Extenson of the functonal model of the bundle adjustment The functonal model of the central perspectve projecton s extended for the ntegraton of the temporal components. The basc structure of the functonal model s stll vald, however. By consderng the temporal components, measurements of dfferent epochs are processed smultaneously. -1 +3 +2 +1-1 +3 +2 +1 Fgure 1: Subsets of two mage sequences wth a pont maged at dfferent epochs -1,, +1, etc. and the correspondng trajectores. (left: camera 1; rght: camera 2)

For the followng explanaton we only consder two mage sequences, keepng n mnd that the method can be extended to any arbtrary number of mage sequences just lke conventonal bundle adjustment. The left mage subset of fgure 1 s regarded as the reference system n our example. If both mage sequences are exactly synchronous, the mage ponts at epochs -1,, +1, etc. n the two subsets are correspondng ponts. The asynchronsm between the two cameras leads to a deformaton of one trajectory wth respect to the other. Therefore, the correspondng ponts n the rght subset are nterpolated wth respect to the asynchronsm between the two cameras. Because of the very small dstances between two ponts of the trajectory a lnear nterpolaton can be employed. For the nterpolaton the asynchronsm s converted nto a geometrcal term n mage space whch can be used n the analyss: 1 tred ( t ) = t( t ) n wth n = nt[ t( t ) f ] (3) f sync t ) = f t ( t ) (4) ( red t red ( t ) Reduced asynchronsm [sec] n Renumberng factor of asynchronsm sync t ) Interpolaton factor of asynchronsm ( Frst, f necessary, the asynchronsm has to be reduced by an nteger multple of the exposure nterval (see formula 3). Through the reducton, the numberng of the correspondng mage ponts s adapted by the renumberng factor n and the mage pont of the reference system corresponds to the mage pont +n n the other system. The use of the nterpolaton factor of the asynchronsm leads to the followng temporal correcton terms for the mage coordnates x and y: x y Tme Tme ( t ( t ) = ( x ) = ( y + sgn( sync( t )) + sgn( sync( t )) x ) sync( t ) y ) sync( t ) Analogous to the correcton terms of the nteror orentaton x Dstorton and y Dstorton the temporal correcton terms x Tme (t ) and y Tme (t ) can be ntroduced nto the collnearty equatons, where X, Y, Z are the object coordnates of the consdered pont and X,, yh are the elements of exteror and nteror orentaton. x = y = f ( X, Y, Z, X f ( X, Y, Z, X, Y, Z, Y, Z, Ω, ϕ, κ, c, xh, yh) + x, Ω, ϕ, κ, c, xh, yh) + y Dstorton Dstorton + x + y The results of the extended analyss of the stuaton depcted n fgure 1 are shown n fgure 2. The red ponts n the rght subset are the mage ponts of the asynchronously acqured second mage sequence. The mage ponts -1*, *, +1* are nterpolated by usng the presented approach to elmnate the effects of asynchronsm between the two used cameras. Tme Tme ( t ) ( t ) (5) (6) +3 +2 +1 +3 * +2 * +2 +1 * * +1-1 * +3-1 -1 Fgure 2: Subsets of two mage sequences wth the corrected mage ponts of the trajectory

2.4. Precondtons of the approach There are some precondtons of the new approach: The frame rate of every camera has to be constant over the analyzed tme nterval. Furthermore the object movement and the object speed have to be constant wthn a short tme nterval due to the employed lnear nterpolaton. If the camera network only conssts of two cameras, t s ndspensable, that the object movement does not occur n the eppolar plane, because otherwse the asynchronsm results n a systematc pont shft n that plane snce the two mage rays stll ntersect. Furthermore t s necessary to measure one mage pont n successve mages. Therefore the measurements of at least three successve tme steps of the tme nterval have to be avalable for the analyss. 3. Expermental Results of the new approach 3.1. Test equpment and test condtons The goal of these frst tests s to demonstrate the sutablty of the proposed new approach. For safety and cost reasons a rotatng three-dmensonal stable test feld was used n the tests. The object ponts on the test feld have a maxmum speed of 7 m/sec. They were observed by two NAC H-DCam II hgh-speed cameras whch acqure mage sequences wth a frame rate of 1 Hz. Each camera has a usable sensor sze of 128 x 512 pxels and a pxel sze of 12 µm. The used focal lengths were about 16 mm, the stereo base s about 28 cm and the dstances between the cameras and the rotatng test feld were about 1.9 m. Out of these test propertes and the assumpton of an mage measurng accuracy of.5 pxel a mean theoretcal standard devatons of σ x =.1 mm, σ y =.5 mm and σ z =.8 mm for the object ponts on the test feld were calculated. For the defnton of the coordnate system see fgure 3 and 4. The vewng drectons of the cameras are tlted about 3 to the Z-axs of the coordnate system, the stereo base s parallel to the X-axs and the base-to-dstance rato amounts to 1:7. The coordnate system s algned n the way that the rotaton axs of the test feld s parallel to the Z-Axs of the coordnate system. The nteror orentaton of the camera had been determned wthn a test feld calbraton and the exteror orentaton was calculated wth the help of a non-rgd test feld before the test. Both orentatons are assumed to be constant over the analyzed tme nterval. Each of the targets on the test feld was then measured wth automatc target detecton algorthms. So for each object pont on the test feld a 2D-trajectory n the mages s avalable. For the followng explanaton of the analyss the focus s set to three specal postons on the 2D-trajectory of object pont C12 whch s representatve for the whole setup (see fgure 3). The three selected postons are marked n the fgure 3 and denoted as top, mddle and bottom. Due to the algnment of the test feld wth respect to the coordnate system, the Z-component of the object pont C12 s not changng over the analyzed tme nterval. The movement of the object pont s only n the X-Y level.

Y 2Dtrajectory Mddle C12 Fgure 3. Image sequence 2D-trajectory of object pont C12 of the test feld. 3.2. Analyss of the test wthout modelng the asynchronsm X Frst, the object space coordnates of pont C12 resultng from a conventonal bundle adjustment, whch neglects the asynchronsm between the two cameras, are analyzed. The analyss s done for each tme step separately and subsequently, three-dmensonal trajectores of the object ponts are computed. The effect of the asynchronsm on the three-dmensonal pont determnaton depends on the movng drecton of the object pont. If the object pont moves n the eppolar plane, the asynchronsm results n a translaton of the object coordnates wthn ths plane (see above). If the object pont moves n another drecton, the asynchronsm results n hgher standard devatons of the object coordnates. The translaton effects are shown n fgure 4. Y Mddle Z Devaton of the Z-coordnate [mm] 25 2 15 1 Mddle 5 1 2 3 4 5 6 7 8 9 1-5 -1-15 -2 Tme of the mage sequence [ms] Fgure 4. Translaton of the object pont due to the effect of the neglected asynchronsm n the analyss; theoretcal stuaton (left), analyss results (rght) At the trajectory postons top and bottom the object pont moves n the eppolar plane. At these postons the asynchronsm leads to a translaton of the object pont n vewng drecton. Dependng on the movng drecton the calculated postons of the object pont are n front of the real poston or behnd the real poston (see postons top and bottom n fgure 4). In ths case the asynchronsm s about.8 ms and the object pont C12 moves wth a speed of 2.9 m/s. Thus, the effect of the asynchronsm parallel to the vewng drecton s about 2.3 mm. At the poston bottom, ths results n a translaton of about 16 mm n vewng

drecton. Due to the tlt of the vewng drecton wth respect to the Z-axs of about 3, the translaton affects the Y- and Z-component of the object coordnates. Thus the correct poston s translated about 14 mm n Z-drecton and 8 mm n Y-drecton (see fgure 4). The dotted lne n the left part n fgure 4 shows the plane n whch the object pont C12 s actually movng. At the trajectory poston mddle the object pont moves n a drecton perpendcular to the eppolar plane. At ths poston the asynchronsm results only n an ncreasng standard devaton of the object pont coordnates. The calculated poston of the object pont, however, s correct. 3.3. Analyss of the test wth modelng the asynchronsm In contrast to the analyss, whch neglects the asynchronsm, the new approach wth the extended functonal model of the bundle adjustment s now used. The results of the analyss are shown n fgure 5. The calculated poston for top, mddle and bottom le all on the plane (see left part of fgure 5). The remanng devaton of the Z-component s caused by the rotaton of the test feld, whch s not perfectly straght. 25 Y Mddle Z Devaton of the Z-coordnate [mm] 2 15 1 Mddle 5 2 4 6 8 1-5 -1-15 -2 Tme of the mage sequence [ms] Fgure 5. Accuracy of the analyss wth modelng the asynchronsm; theoretcal stuaton (left), analyss results (rght) 3.4. Comparson of the results and verfcaton of the new approach For the poston mddle the results of the analyss are presented n table 1. It can be seen that modelng the asynchronsm n the descrbed way resulted n an mprovement of the theoretcal standard devaton of the coordnates of pont C12 by factor 1. Furthermore, the theoretcal expectatons are met. Object pont C12 on the test feld Theoretcal values Analyss wthout modellng the asynchronsm Analyss wth modellng the asynchronsm Asynchronsm.8 ms./..79 ms σ x.16 mm 1.82 mm.17 mm Mddle σ y.45 mm 5.23 mm.49 mm σ z.7 mm 8.17 mm.77 mm Table 1: Comparson of the standard devatons of the coordnates of both types of analyss

The dfferences between the calculated coordnates n both types of analyss are shown n fgure 7. They correspond to the theoretcal values for the translaton wth an asynchronsm of.8 ms and an object speed of about 2.9 m/s. The lnear correlaton between the asynchronsm and the translaton of the object coordnates n vewng drecton does only appear n a test setup wth two cameras as n our experments. If mage sequences of more than two cameras are used for the analyss the asynchronsm results n an ncrease of the standard devaton of the calculated object pont coordnates, as can be shown usng eppolar reasonng, and as addtonal experments have actually demonstrated. In our test setup the modelng of the asynchronsm leads to an ncreasng accuracy of the object coordnates and to the correct determnaton of the object coordnates. Dfference of the object coordnates [mm] 2 15 1 5 1 2 3 4 5 6 7 8 9 1-5 -1-15 -2 Mddle Tme of mage sequence [ms] X Y Z Fgure 7: Dfferences of the object coordnates over the analyzed mage sequence caused by the dfferent types of analyss The verfcaton of the determned asynchronsm s carred out by an external tme measurng system (see fgure 3; below the yellow box labeled C12 ). From the lt LEDs the tme of mage capture can be derved. The mage of the left camera was acqured at tme 43.3 ms and the mage of the rght camera was acqured at tme 44.1 ms. Thus, from ths tme measurng system an asynchronsm of.8 ms s computed between the two cameras. Ths s about 8 % of the exposure nterval of the used cameras. In the analyss an asynchronsm of.79 ms was obtaned. The resoluton of the external tme measurng system s restrcted to.1 ms n the used mode, the accuracy s a few orders of magntude better. Therefore, the estmated value of the analyss and the calculated value for the asynchronsm of the tme measurng system can be regarded as equal. 4. Concluson and Outlook In ths artcle a procedure s presented, whch permts the photogrammetrc analyss of asynchronously acqured mage sequences. The asynchronsm s modeled wth a lnear approach and s then converted to an nterpolaton factor. Wth ths nterpolaton factor temporal correcton terms for the mage coordnates are calculated. The modelng of the asynchronsm as a temporal correcton term n the functonal model of the bundle block adjustment leads to a sgnfcant mprovement of the results. At the current state of work there are some precondtons for the successful applcaton of ths new approach for the photogrammetrc analyss of mage sequences. All cameras must have a

constant frame rate over the analyzed tme nterval. Ths precondton corresponds to the lnear asynchronsm model. Furthermore and for the same reason, the object movement and the object speed have to be constant wthn a short tme nterval. In our experments two mage sequences of a rotatng three-dmensonal test feld are analyzed. For the frst test the asynchronsm s only modeled wth the constant part of the asynchronsm, the tme-offset. The use of ths reduced approach leads to a correct determnaton of the object coordnates and to an mprovement of the object pont accuracy of factor 1 n contrast to the analyss, whch neglects the asynchronsm. The calculated accuracy also corresponds to the theoretcally estmated values. The determned asynchronsm s.79 ms, whch corresponds to the external determned asynchronsm of.8 ms. In followng test seres the applcablty of ths new approach has to be further nvestgated. At the current state of work some of the mentoned precondtons are ndspensable. Further nvestgatons wll show f some of them can be relaxed under specal condtons. In addton the descrbed procedure wll be appled to real-world applcatons such as vehcle mpact tests. Also, experments wth more than two cameras and wth dfferent types of cameras wll be carred out, and the smultaneous determnaton of the nteror and exteror orentaton of the cameras and the asynchronsm wll be nvestgated. References: [1] D Apuzzo, N.: Surface Measurement and Trackng of Human Body Parts from Mult Staton Vdeo Sequences. Dssertaton ETH Zürch, No. 15271, 23. [2] Hasted, H., Luhmann, T., Raguse, K.: Synchronous 3-D Hgh-Speed Camera wth Stereo-Beam Splttng. SENSOR 25, 12 th Internatonal Conference, AMA Servce, pp. 443-448. [3] Maas, H.-G.: Dgtale Photogrammetre n der dredmensonalen Strömungsmesstechnk. Dssertaton ETH Zürch, No. 9665, 1992. [4] Putze, T.: Ensatz ener Hghspeedkamera zur Bestmmung von Geschwndgketsfeldern n Gasströmungen. In Seyfert, E. (Ed.): Publkatonen der Deutschen Gesellschaft für Photogrammetrte, Fernerkundung und Geonformaton, Band 12, p. 325-332, 24. [5] Raguse, K., Wggenhagen, M.: Qualty parameters of the optcal data channel used n mpact tests. In Grün, A., Kahmen, H. (Eds.): Optcal 3-D Measurement Technques VI, Vol. II, Repro Zentrum ETH Zürch, p. 252-258, 23. [6] Raguse, K., Derpmann-Hagenström, P., Köller, P.: Verfzerung von Smulatonsmodellen für Fahrzeugscherhetsversuche. In Seyfert, E. (Ed.): Publkatonen der Deutschen Gesellschaft für Photogrammetrte, Fernerkundung und Geonformaton, Band 12, p. 367-374, 24. [7] Santel, F., Lnder, W., Hepke, C., Image Sequence Analyss of Surf Zones: Methodology and Frst Results. In Grün, A., Kahmen, H. (Eds.): Optcal 3-D Measurement Technques VI, Vol. II, Repro Zentrum ETH Zürch, p. 184-19, 23. [8] Snha, S. N., Pollefeys, M.: Synchronzaton and Calbraton of Camera Networks from Slhouettes. Internatonal Conference on Pattern Recognton ICPR, Vol. I, p. 116-119, 24. [9] Wllneff, J.: A Spato-Temporal Matchng Algorthm for 3D Partcle Trackng Velocmetry. Dssertaton ETH Zürch, No. 15276, 23. [1] Yan, J., Pollefeys, M.: Vdeo Synchronzaton va Space-Tme Interest Pont Dstrbuton. Advanced Concepts for Intellgent Vson Systems ACIVS, 24.