Image-based Motion Stabilization for Maritime Surveillance

Transcription

1 Image-based Moton Stablzaton for Martme Survellance Danel D. Morrs, Bran R. Colonna and Franln D. Snyder General Dynamcs Robotc Systems, 151 Ardmore Blvd, Pttsburgh, PA * ABSTRACT Robust mage-based moton stablzaton s developed to enable vsual survellance n the martme doman. The algorthm developed s nether a dense regstraton method nor a tradtonal feature-based method, but rather t captures the best aspects of each of these approaches. It avods feature tracng and so can handle large ntra-frame motons, and at the same tme t s robust to large lghtng varatons and movng clutter. It s thus well-suted for challenges n the martme doman. Advantage s taen of the martme envronment ncludng use of the horzon and shorelne, and fused data from an nexpensve nertal measurement unt. Results of real-tme operaton on an n-water buoy are presented. Keywords: Optcal stablzaton, feature regstraton, martme, buoy survellance, computer vson, nertal measurement unt, water surface, fuson, horzon, shorelne. 1. INTRODUCTION The frst step n survellance s usually stablzaton for camera moton [3,4,9,11], and that s the focus of ths paper. Whle there has been much wor n ground and aeral survellance, the martme doman has not yet drawn sgnfcant attenton. Yet the martme doman presents a number of unque challenges and opportuntes for moton stablzaton. Floatng sensors must deal wth constant moton of ther own and of the water surface around them. Waves may generate hgh-texture features, but ther moton s non-rgd and ther appearance changes rapdly. Martme lghtng typcally has greater contrast than on land due to drect sunlght reflectons wth a resultng loss of detal for cameras wth lmted dynamc range. Rapd changes n lghtng can be countered by an auto-rs, but ths leads to changes n object appearance and detals. Fnally water spray can lead to droplets and salt deposts on the camera protectve cover, causng dstortons and clutter n the magery. Here algorthms are developed for robust mage stablzaton on a floatng buoy. Whle t s possble to use a gyro-based sensor to do stablzaton, ths requres a hgh-precson devce whch s typcally expensve and buly. Drectly usng mages enables stablzaton down to sub-pxel precson, whch s precsely the level needed for moton-based detecton algorthms. There are two general approaches to mage stablzaton: dense technques and feature-based technques. In dense technques mages patches are drectly warped onto each other or correlated wth each other usng moton models, for example see [8,9]. On the other hand, feature-based technques rely on tracng features between mages and drectly calculatng moton from them, see [2,3,4,1,12]. Advantages of dense methods nclude more pxels beng used and requrng explct feature correspondences s avoded. But these methods are typcally not robust to many of the challenges n martme envronments such as rapdly movng specular reflectons and large changes n gan and contrast due to lghtng changes. Feature-based methods can be qute robust to lghtng varatons as well as to movng clutter n the scene, however they face the challenge of mantanng feature tracs especally when moton s large. They also have dffcultes f the scene does not have stable corner features. Here a hybrd technque for moton stablzaton s presented. It acheves robustness to lghtng varatons and movng clutter n a smlar way as feature technques, but at the same tme, le the dense technques, t does not requre fndng corner features or tracng features between frames. It can thus wor wth very large ntra-frame moton and wth large lghtng varatons and movng clutter. The geometry of the martme doman s leveraged ncludng use of the horzon and shorelne as well as an nertal measurement unt (IMU) f avalable. Stablzaton was mplemented n a real-tme system onboard a buoy. Sample results are llustrated. * Wor was performed n part at Northrop Grumman Corp, 151 Ardmore Blvd, Pttsburgh PA

2 The paper s organzed as follows. Frst our assumptons are stated n Secton 2. Then stablzaton of the vertcal axs s descrbed n Secton 3, followed by stablzaton of headng n Secton 4. Flterng to combne nertal and mage measurements s descrbed n Secton 5. Fnally results are gven and dscussed n the concluson. 2. ASSUMPTIONS Stablzaton s performed usng a 36-degree feld of vew, 5-camera array that has been calbrated such that for each u, v there s a nown unt 3-vector, p ˆ, gvng the drecton of the ray ncdent on that pont, pxel n each camera, ( ) namely: f u v s the calbraton functon. where (, ) ( ) p ˆ = f u, v, (1) Snce the cameras are close together compared to the dstance to detected objects, ther optcal centers are approxmated as beng concdent. By worng n sphercal coordnates, data from all cameras can be treated unformly. It s assumed that over short tme perods the buoy translaton s neglgble compared to the objects t observes. Furthermore t s assumed that the horzon or dstant shorelne s vsble for vertcal stablzaton, and that shorelne features are avalable for headng estmaton. 3. VERTICAL STABILIZATION Image moton can be explaned as a rotaton of the platform, and hence stablzaton s acheved by estmatng the platform rotaton. The approach here s to dvde rotaton estmaton s nto two sequental steps: frst vertcal axs estmaton and then headng estmaton. Vertcal axs estmaton s descrbed n ths secton. The ey property of the martme doman that ads stablzaton s that the vsble horzon or dstant shorelne defnes a horzontal plane. Fgure 1 llustrates a buoy rotated by W B R n world coordnates wth respect to a reference frame on the horzontal plane, H. The thrd row of W R, s B B z, the transpose of the vertcal axs n the buoy reference frame. In Euler coordnates ths s [ sn( p),cos( p)sn( r),cos( p)cos( r) ] T W, see 5 pg. 46, where r and p are the roll and ptch around the world x and y axes respectvely. Thus fndng the vertcal vector, B z W, or equvalently the horzontal plane, n buoy coordnates s suffcent to determne the buoy roll and ptch. W R B z B z W H y W x W Fgure 1 The world coordnate system s defned such that z W s perpendcular to the horzontal plane H. The buoy coordnate system s rotated wth respect to ths by W R. B It s assumed that the horzon (or a suffcently dstant shorelne) s vsble as a contrast change n part of the 36-degree mage. However the mages wll typcally be hghly textured wth waves, clouds and shorelne provdng clutter from whch the horzon must be extracted. The followng robust technque was used to determne the horzontal plane.

3 Fgure 2 Peas and troughs of the vertcal gradents are found n each mage column n the regon around the horzon. These ponts nclude the horzon, dstant shorelne and other clutter. The horzon s found by determnng the plane that explans the most peas n all smultaneous mages. For each column of the smoothed vertcal gradent mages of all the cameras, all the local maxma and mnma n the regon around the predcted horzon and above a small threshold are found. A subset of these wll correspond to the horzon. Each of the maxma and mnma s mapped to a pont on the unt sphere, p ˆ, usng Eq. (1). Now the horzon ponts wll all le on the plane H, and at the same tme t s unlely that there wll be any other plane through the orgn generatng a large number of ponts p ˆ. Thus a robust technque, such as RANSAC 6, s used to fnd the best plane fttng these ponts. Pars of ponts are suffcent to defne the perpendcular to a plane through ther cross product: v = pˆ p ˆ (2) j The vector v wth the most nlers s the ntal estmate for B z W, and a least squares estmate usng the nlers can be obtaned as the egenvector correspondng to the mnmum egenvalue of: T A = p p. (3) Examples of stablzaton are shown n Fgures 8 and 9. nlers 4. HEADING STABILIZATION The next step s to determne the change n headng. Our approach to achevng ths s to temporally algn vertcal features on the vsble shorelne. We want these features to be robust to lghtng varaton and we want to avod tracng ndvdual features. A technque that acheves ths s based on vertcal curves. Vertcal curves trace the vertcal edges of objects and landmars on the shorelne, see Fgure 3. They are obtaned wth sub-pxel accuracy and are nvarant to brghtness and contrast changes. Frst the horzontal gradents of all the mages are found. The maxma and mnma of the gradent for each row are parabolcally ft to sub-pxel accuracy, and roughly vertcal contour-based curves are created by connectng close-by maxma and mnma between adjacent horzontal rows as llustrated n Fgure 3.

4 (a) (b) Fgure 3 (a) Curves found on objects on the shorelne. (b) Close-up showng the peas and troughs of the horzontal gradent (red and blue dots), and the curves traced through them. Curves are bult n mage space, but then transformed onto the unt sphere usng Eq. (1). They are then stablzed wth T R wth zero headng. After ths the elevaton and azmuth angles, ( ϕ, θ ) respect to roll and ptch by rotatng by W B each transformed pont, p ˆ, on each curve can be calculated up to an unnown overall headng: 2 2 ( p p p ) ϕ = arctan +, (4) x y z ( p p ) θ = arctan, (5) y x It s assumed that a porton of shorelne s vsble above the horzon, and ths s used for fndng headng change as s ϕ ; θ, at a set of elevatons ϕ and tme t of the transformed curves are made, follows. A seres of horzontal slces, ( ) t and for each slce the locatons, θ, of all curve ntersectons are recorded wth a ± 1 pxels dependng on the sgn of the curve at that pont, see Fgure 4. Correspondng slces are made through subsequent mages, and the relatve headng found by a crcular convoluton of these slces wth the slces at prevous tmes. Crcular convolutons can be effcently calculated S ϕ ; θ s the FFT of wth the use of fast Fourer transforms and ther nverses: FFTs and IFFTs respectvely. If ( ) s ( ϕ ; θ ) and S ( ϕ ; θ ) t t the complex conjugate, then the sum of the crcular convolutons wth slces at tme t m s gven by: ( θ ) = IFFT { ( ϕ ; θ ) ( θ ) ( ϕ ; θ )} tm tm t slces w S G S (6) where G( θ ) s the FFT of a Gaussan added for smoothng and to reduce senstvty to calbraton mprecson. There s no need for zero paddng as the convoluton s crcular. The locaton of the pea of ( ) tm w θ gves the headng wth respect to tme t. Ths s robust to movng objects and spurous curves, snce curves that do not have a match do not contrbute to the result. Hence a robust and precse headng s obtaned. t, for

5 (a) (b) Fgure 4 (a) Usng the roll and ptch estmated prevously, vertcal curves are transformed nto the ( ϕ, θ ) space. A seres of horzontal slces cut through these at varous elevatons above the horzon. The nterpolated locaton of these curves on one of these slces s shown n (b) (a) -5 5 (b) Fgure 5 (a) Convoluton w ( θ ) from Eq. (6) of the slces n Fgure 4. The lower chart tm s a close-up showng a small change n headng at the maxmum n degrees. 5. FILTERING Rotaton estmates can be mproved by flterng. Ths enables the ncluson of a buoy dynamc model and moton estmates from the IMU. A standard Kalman flter was mplemented to acheve ths wth the followng partcular propertes. The buoy dynamcs were modeled as ndependent damped harmonc oscllators n roll and ptch, and damped angular speed model n headng. Wth state vector, x = [ r, r, p, p, h, h ] T, contanng roll, ptch, headng and ther tme dervatves, the equaton of moton s: x = Fx + u( t). (7) The component of x for roll s [ r ṙ ] T and ts dynamcs are descrbed by:

6 1 F roll = 2 (8) ω γ where ω and γ are the characterstc buoy angular speeds and dampng respectvely. Analogous expressons apply to ptch and headng (wth ω = for the latter). Together they form a bloc-dagonal F, and state transfer functon: Φ = exp F t. (9) ( ) The drvng term, u ( t), s the acton of the waves and s unnown, and so s modeled as system nose, Q. It acts as a contnuous acceleraton term and so to ntegrate t nto our dscrete formulaton the followng equaton was used for roll nose (and smlar equatons for ptch and headng): T Q roll = Froll Qroll + Qroll Froll + 2, (1) σ where where 2 σ s the measure of the acceleraton from wave moton. Ths can be re-wrtten n the form: q = Mq + (11) 2 σ The soluton s gven by: q 2 11 q11 q12 2 q 12,, and roll ω γ 1 q12 q 22 2 q 22 2ω 2γ q = Q = M =. ( ( t) ) 1 q = M exp M I, (12) 2 σ from whch Q roll s obtaned. Q ptch and Q headng are obtaned n a smlar manner and together form the bloc-dagonal Q. The measurement matrx, H, defned by Hx = z where z contans the measured quanttes, s smple to calculate. The measurements nclude r, r, p, p, h, ḣ drectly from the IMU, and r, p, h from the mages, where h s the headng relatve to the prevous mage. The covarance, R, on the measurements was chosen to have much larger terms on the IMU components than the mage components. All these terms are plugged nto standard dscrete Kalman flter equatons and produce a fused mage and nertal stablzaton system. For completeness these equatons are summarzed as follows: x = Φ x ( ) ( + ) t t t 1 P = Φ P Φ + Q ( ) ( + ) T t t t 1 t t 1 T ( ) ( ) K = P H H P H + R ( ) T ( ) t t t t t t t x = x + K z H x ( + ) ( ) ( ) t t t t t t P = P K H P ( + ) ( ) ( ) t t t t t 1 (13) 6. RESULTS Stablzaton through horzon-fndng n fve cameras turned out to be very robust. Examples are shown n Fgures 7, 8 and 9. Usng an nexpensve nertal measurement devce sgnfcantly sped up the computaton by reducng the search space as well as the percentage of outlers to be dealt wth by the RANSAC operaton. Relatve headng estmaton was fast, requrng only 1D FFTs and gave pxel-level precson, see Fgure 5. A comparson of IMU and mage-stablzaton as well as fused results s gven n Fgure 6. Dstant shps can help n short-term stablzaton, although over longer term ther

7 moton relatve to the shorelne can be detected. Stablzaton was easly calculated wth a Pentum III processor wth data from 5 124x768 cameras at 7 frames per second. 1 Roll Deg Ptch Deg Headng Deg 55 IMU Image Flter Tme (sec) Fgure 6 Comparson rotaton estmates from IMU and the mage-based algorthm and a fltered approach that fuses these. The IMU s rated to accurate to a RMS accuracy of 2 whereas the mage-based headng s accurate down to at least pxel resoluton, whch for these cameras s ±.2. In ths example of low acceleraton, the IMU accuracy s much hgher than ts rated value and comparable to the mage technque. For larger motons, however, the IMU accuracy degrades as can be seen n Fgure 9, whereas the mage-based technque mantans hgh accuracy. 7. CONCLUSION An effcent and robust mage-based moton stablzaton technque was developed for the martme doman. The use of features based on local maxma and mnma of the gradent mages gave robustness to rapd lghtng changes. The use of curves rather than corners s more approprate to martme envronments where corner features may be rare. Avodng the need to trac features maes the technque robust to large ntra-frame moton. Usng sphercal coordnates enables measurements from cameras pontng n all drectons to be used, greatly reducng ambgutes n horzon-fndng that occur n sngle-camera solutons. The next step s to use the moton stablzaton to acheve movng object detecton. The curved features developed here can be used for ths. Obtanng the absolute vertcal, as ths does, rather than smply an ncremental change n rotaton, enables precse survellance applcatons that search for objects close to the horzon. A lmtaton of mage-based technques s that they depend on envronmental condtons. Fog or haze can obstruct the vew of the horzon or shorelne. In these cases t may be necessary to rely on nertal measurements whch can easly be done wth our flterng approach. A useful advantage of the mage-based approach s that accuracy can ncreased smply by addng more cameras wth hgher resoluton.

8 (a) (b) -12 (c) Fgure 7 (a) A porton of the feld of vew of the camera array ncludng shorelne and watercraft. (b) Horzon and curves are shown overlad. (c) Curves are transformed nto stablzed, sphercal coordnates. ACKNOWLEDGEMENT Ths wor was performed under the sponsorshp of ONR FNC-AO-IA, PM Marc Stenberg, #N421-3-C-27 and # DAAD REFERENCES: 1. H. Asada, M. Brady, The Curvature Prmal Setch, IEEE PAMI 8(1):2-14, D. Burscha, G. Hager, Vson-based control of moble robots, n Proc. Internatonal Conference on Robotcs and Automaton, pages , A. Cens, A. Fusello, and V. Roberto, "Image stablzaton by features tracng," n Proc. 1th Int. Conf. Image Analyss and Processng, Sep 1999, pp I. Cohen, G. Medon Detectng and tracng movng objects n vdeo survellance, Proc IEEE Conf Computer Vson and Pattern Recognton 1999; II: J.J. Crag, Introducton to Robotcs Mechancs and Control, Second Edton, Addson Wesley Longman, M.A. Fschler, R.C. Bolles. Random sample consensus: A paradgm for model fttng wth applcatons to mage analyss and automated cartography, n Comm. of the ACM, volume 24, pages , C. Harrs, M. Stephens, "A combned corner and edge detector", n Alvey Vson Conf., 1988, pp Iran, M., Rousso, B., and Peleg, S., "Recovery of Ego-Moton Usng Image Stablzaton," CVPR 1994, pp C.D. Kugln, D.C. Hnes. The phase correlaton mage algnment method, IEEE Conference on Cybernetcs and Socety, p , A. Ltvn, J. Konrad, W.C. Karl Probablstc vdeo stablzaton usng Kalman flterng and mosacng Image and Vdeo Communcaton and Processng 23, SPIE Vo. 522, pp

9 11. D.G. Lowe, Object recognton from local scale-nvarant features, Internatonal Conference on Computer Vson, Corfu, Greece (September 1999), pp L. Marcenaro, G. Vernazza, C.S. Regazzon, Image stablzaton algorthms for vdeo-survellance applcatons, n Proc. Int. Conf. Image Processng 21, Vo1 1, pp E. Shlat, M. Werman, Y. Gdalyahu, Rdge s corner detecton and correspondence, IEEE Proc. CVPR, 1997, pp Fgure 8 Horzon stablzaton shown on 3 of 5 mages. Top row shows orgnal mages. The vertcal offsets and radal dstortons are unmportant as they are accounted for n the calbraton, Eq (1). Blue lnes n center row show regon bounded by uncertanty of the IMU. In ths regon the yellow dots show all the vertcal gradent peas. Usng RANSAC a subset of these are determned to be nlers to the horzon and these are plotted n the bottom row wth the blue lnes beng the bounds on the nlers. The fnal estmate of the horzon s shown by the orange lne and s obtaned as a least squares ft to the nlers, see Eq. (3). Fgure 9 Another example of horzon fttng wth very large roll and ptch shown n 3 of the 5 cameras. The two blue curves are bounds of the search regon gven by the IMU and centered around ts estmate whch shows sgnfcant error compared to the fnal mage-estmate. Gradent peas are hghlghted n ths regon. Applcaton of RANSAC fnds nlers to the horzontal plane, here mared as whte.