RELATIVE ORIENTATION ESTIMATION OF VIDEO STREAMS FROM A SINGLE PAN-TILT-ZOOM CAMERA Taeyoon Lee a, *, Taeung Km a, Gunho Sohn b, James Elder a a Department of Geonformatc Engneerng, Inha Unersty, 253 Yonghyun-dong, Nam-gu Incheon, Korea - etaeyoon@nha.edu, tezd@nha.ac.kr b GeoICT Lab, York Unersty, 47 Keele St., Toronto, ON M3J 1P3, Canada gsohn@yorku.ca b Centre for Vson Research, York Unersty, 47 Keele St., Toronto, ON M3J 1P3, Canada elder@yorku.ca Commsson I, WG I/5 KEY WORDS: Sngle Pan-Tlt-Zoom Camera Calbraton, Collnearty Condton, Orentaton, Self-calbraton ABSTRACT: As a surellance camera for the securty of urban area, a sngle pan-tlt-zoom (PTZ) camera has been used. For effecte applcaton of the deo sequence of the sngle PTZ camera, accurate orentaton or calbraton of the deo sequence s needed. Ths paper proposes a mathematcal model based on the collnearty condton equaton for relate orentaton estmaton between frames from deo sequence of a sngle PTZ camera. The basc concept of ths mathematcal model s smlar to the homography concept, whch eplans the relatonshp between two mages. Howeer the proposed model s based on the collnearty equaton and the model can estmate the aryng focal length and relate rotaton angles drectly from correspondng ponts between mage frames. For the relate orentaton estmaton by the model, the rato between focal length and CCD sze of the frst frame s requred as ntal nput. In ths paper, we estmated the rato by two anshng ponts. The correspondng ponts were etracted by Scale Inarant Feature Transform (SIFT) and RANdom SAmple Consensus (RANSAC) automatcally. Photos taken by dgtal camera and a deo sequence of a sngle PTZ camera were used for eperments. The eperment results show the proposed method can estmate the relate orentaton from the deo sequence automatcally. 1. INTRODUCTION In modern socety, cameras hae been used for the securty n many places. Usually they are fed n locaton wth pan, tle and zoom capablty. For the effecte applcaton of a deo stream data of surellance cameras and for etracton of the accurate poston nformaton from the deo sequence, accurate calbraton or orentaton of the camera s an essental process. Some calbraton methods for the sngle PTZ camera hae been proposed. Many of the proposed methods are based on the concept of the mage of the absolute conc (IAC) and the concept of homography between two ews (Agapto et al., 2; Km and Hong, 2; Fung and Dad, 29). These methods (Agapto et al., 2; Fung and Dad, 29) etracted nternal parameters from IAC matr estmated usng some constrants such as the zero-skew, the square-pel and dstorton. They estmated eternal parameters by nternal parameters etracted. Km and Hong (2) proposed the method whch calculated rotaton angles and then aryng focal length sequentally. We thnk relate nternal parameters etracted from preous methods may nclude errors as these methods mnmze errors for IAC parameters, not for nternal parameters of the camera. In ths paper, we propose a mathematcal model and automatc method for relate orentaton estmaton between successe frames from a deo sequence of a sngle PTZ camera. In ths paper, relate orentaton means rotaton angles (pan and tle angles) and aryng focal lengths between successe frames. The proposed model uses the concept whch s smlar to the homography between two ews, but the model s n form of the collnearty condton equaton, so t can drectly estmate relate orentaton parameters from the deo sequence of a sngle PTZ camera. 2. PROPOSED METHOD 2.1 Model based on the Collnearty Equaton Proposed mathematcal model eplans the relatonshp between two mages. Collnearty condton can be epress as E.q. (1), The y y f = X X RY Y Z Z λ (1) where f s the focal length, (, y) s the mage coordnates, (, y ) s the coordnates of the prncple pont, (X, Y, Z ) s the coordnates of proecton centre n the obect coordnate system, (X, Y, Z) s the obect coordnates n the obect coordnate system, R s rotaton matr and λ s a scale factor. r r r E.q. (1) can smply epress as = λ R( X X ). * Correspondng author. Ths s useful to know for communcaton wth the approprate person n cases wth more than one author.
In the case of a statonary camera, we can assume that the proecton centre of the camera s at the orgn of the obect coordnate system, so X r T s (,,). 2.2 Intal Focal Length and CCD Cell Sze For estmaton of the relate orentaton by E.q. (4), a rato (f/ccs) between an ntal focal length (focal length of the frst mage) and CCD Cell Sze (CCS) s needed. For the rato estmaton, the method usng two anshng ponts of a sngle mage was used (Gullou et al., 2). In ths method, the anshng ponts are estmated by least square estmaton and coordnates of lnes on the mage. For estmaton of the anshng ponts, we assumed that the slops ( m ) n N lne equatons (E.q. (5)), estmated by coordnates of lnes, had error and the anshng pont (, y ) were estmated by E.q. (5) and non-lnear least square estmaton. y = m + n ( y n ) m = = 1,, N () (5) Fgure 1. The geometrc relatonshp between a sngle PTZ camera Hence, f there are two mages ( and ) and an obect pont X r s proected on the two mages (Fgure 1), the mage and can be epress as r r = λr X r r (2) = λ R X Elmnatng X r from E.q. (2) we can obtan the relatonshp between the mage and : r λ r T = R R (3) λ The coordnates of the lnes were etracted by both hough transform and selectng manually. 2.3 Automatc Estmaton of the Relate Orentaton After estmaton of the rato (f/ccs) between focal length and PSC n the frst mage, the proposed method begns to estmate the relate orentaton between contnuous two frames of a deo sequence automatcally. In ths paper, the successe two frame mages are etracted by OpenCV 2. lbrary (OpenCV2.). From the two frame mages, accurate correspondng ponts are etracted by SIFT (Lowe, 24) and RANSAC technque (Fschler and Bolles, 1981). Then the relate orentaton s estmated by E.q. (4) and the correspondng ponts. In ths paper, for SIFT processng, we used an open source by Ross Hess (Hess). Fgure 2 shows correspondng ponts whch s etracted by SIFT and RANSAC. In relate relatonshp between the mage and, f the mage s the frst mage and s net mage, the all rotaton angles of r T r the mage are. Hence E.q (3) s = μr, where μ λ / λ s. Ths relatonshp can be dered nto the form of the collnearty equatons by cancellng out the scale factor μ : y y = f = f r11( r ( 13 13 r12( r ( 21 23 22 ( y y ( y y 23 ( y y ( y y 31 33 ( f) ( f ) 32 33 ( f) ( f ) (4) where r are elements of R. The relate orentaton of net mage can be estmated by E.q. (4) and non-lnear least square estmaton technque. Fgure 2. The correspondng ponts by SIFT and RANSAC
3.1 Dataset used 3. EXPERIMENTS For eperments, photos taken by dgtal camera and a deo sequence from a securty camera were used. Table 1 shows the spec of the dgtal camera (Photonotes.org; Canon). Table 1. The spec of dgtal camera used North Amercan name EOS Dgtal Rebel XT Mamum magng output 3888 2592 dmensons Physcal sensor sze 22.2 14.8 mm Pel sze on sensor chp (PSC) 5.7 μm square Lens EF-S 18-55 mm Echangeable Image Fle Format (EXIF) when a photo s taken. In ths eperment, we used the focal length recorded to EXIF as reference. The sze of the photos (Fgure 3) s 3888 2592 pels. Fgure 4 shows some frame mages whch were etracted from the securty camera. The deo sequence was obtaned from York Unersty. We could not know nternal parameters of the securty camera. Therefore, we had to estmate f/ccs by anshng pont analyss. 3.2 Results Table 2 shows the f/ccss whch were estmated by two anshng ponts and the f/ccss whch were calculated by reference alues. These results show the method by two anshng ponts from a sngle mage can estmate the f/ccs, but the f/ccs may hae some errors. Ths paper focused on the relate orentaton estmaton, so we gnored small errors and used the f/ccs estmated by two anshng ponts as ntal alue. Table 2. f/ccs from reference and f/ccs estmated by two anshng ponts Num f/ccs calculated from f/ccs estmated by two reference anshng ponts 1 4386. (f: 25 mm) 438.93 2 3157.9 (f: 18 mm) 356.2 Fgure 3. The photos taken by dgtal camera and the parameters of the relate orentaton measured manually Table 3 shows the parameters of the relate orentaton whch are estmated by the proposed method. In Table 3, ω - 1 n Parameter changed means that the mage s rotated from the frst mage through 1 degrees clockwse on -as n Fgure 1. φ 2 means the mage s rotated through 2 degrees on y-as. Case 1 of Table 3 shows the result whch s estmated from photos of Fgure 3. Table 3. The results of the relate orentaton estmaton by the proposed method and the photos Parameter ω -1 φ 2 Zoomng ω-φ-f changed (Reference changed ) Case 1: f/ccs estmated from the frst mage : 438.93 f/ccs 4349.5 4458.3 6518.5 (6666.7) 6587.7 (6666.7) ω ( ) -11.1 -. -. -11.2 φ ( ).1 19.7 -. 19.8 Case 2: f/ccs estmated from the frst mage : 356.2 f 3586.7 3515.3 5286.3 (4736.8) 5257.3 (4736.8) ω ( ) -9.9 -.. -1.2 φ ( ) -. 17.8. 17.6 Fgure 4. Frame mages etracted from the deo sequence Fgure 3 shows the photos and reference alues. These photos were taken n each case of pan, tle and zoomng. We measured the pan and tle angles manually, so the angles may be ncluded measurement errors, but we thnk the angles can be used as reference angles. Ths camera records the focal length to As shown n Table 3, the f/ccss estmated were dfferent wth reference, but the rato of f/ccs estmated by the proposed method between the mages, when camera s zoomed, was smlar to the rato calculated usng reference. In Case 2, there was 3 dfference between the φ (17 ) estmated and reference φ (2 ). The dfference was caused by naccurate ntal f/ccs (Table 2).
In the eperments, the radal dstorton was not consdered. The radal dstorton s well known as one error source (Agapto et al., 2). Een though the radal dstorton was not remoed from the mages, relate f/ccs and rotaton angles estmated by the proposed method were generally accurate. Table 4 shows the parameters of the relate orentaton estmated by the proposed method from the deo sequence of the securty camera. In the eperment, reference alues for the deo sequence were not known. Ths paper does not show accuracy of relate orentaton of the deo sequence by the proposed method. We guessed relate orentaton estmated may be accurate relately when we consder the results usng the mages of dgtal camera (Table 3). Fgure 5. The relatonshp between the model accuracy and rotaton angle (P) Fgure 6 shows the change of the focal length estmated and of the RMSE by tme (stream of the frames). The change of the focal length was smlar to zoomng of the deo sequence. In Fgure 6, when the focal length estmated changed around 8, RMSE was ncreased. The reason for ths s the rotaton angle (φ). The angle (φ) s ncreased between frames of the graph whch the focal length changed around 8. Table 4. The results of the relate orentaton estmaton by the proposed method and the deo sequence Frame 1-2 Frame 96-97 Frame 17-171 Intal f 7.4 977. 1871.7 f/ccs 71.1 114.2 1871.7 ω ( )..2 -. φ ( ) 2.2 -.2 -. RMSE (p.) 1.6.5.1 In Table 4, Frame 1-2, Frame 96-97, Frame 17-171 mean the two frame number (Fgure 4). Intal f of Frame 1-2 s f/ccs estmated by two anshng ponts. The ntal f of rest frames were estmated from preous frame mages by the proposed method. When we obsered the changes of zoom and rotaton angles between successe frames by eyes, we could check whch the changes were smlar to the relate orentaton estmated by the proposed method (Table 4). In ths paper, the accuracy of the model establshed s epressed by Root Mean Square Error (RMSE). RMSEs were calculated usng correspondng ponts, whch were not used for the model establshment, as check ponts. Through the eperment results, we checked that there was some correlaton between the rotaton angle (φ) estmated and an accuracy of the proposed model (Fgure 5). As shown n Fgure 5, the more rotaton angle (φ) s ncreased, the more RMSE s ncreased. Fgure 6. The change of focal length and RMSE by tme, the frame order of the deo sequence 4. CONCLUSION Ths paper proposed a mathematcal model based on the collnearty condton equaton for relate orentaton estmaton between frames from the deo sequence of a sngle PTZ camera. Ths paper also proposed automatc relate orentaton of the deo sequence based on the proposed model. By the proposed model, we tred to estmate accurate f/ccs and rotaton angles drectly from the relatonshp between two frame mages. The proposed method can estmate the relate orentaton usng only the deo sequence. For ths, ntal f/ccs s estmated by two anshng by non-lnear estmaton. The eperment results show that the accuracy of the ntal f/ccs nfluences the accuracy of the relate orentaton by the proposed method. For automatc processng, OpenCV lbrary, SIFT and RANSAC technques are used. The results, whch are estmated from the photos taken by dgtal camera, show that the proposed method can estmate accurate parameters wthout correcton of the radal dstorton. The results, whch are estmated from the deo sequence of the securty camera, show that the correlaton between the rotaton angle and RMSE ets. Though these eperment results, we checked the proposed method could estmate the relate orentaton from the deo sequence of a sngle PTZ camera. For obtanng the more accurate nformaton, the proposed method needs to nclude estmaton of the accurate ntal f/ccs, correcton of the radal dstorton, consderaton of the correlaton between the rotaton angles and the modellng error or focal length changed. Ths paper presented automatc processng for the relate orentaton by well known technques, but ths method was slow. For effcent applcaton of securty camera, real tme processng may be needed.
ACKOWLEDGEMENTS The work n ths paper was supported by a grant (7KLSGC3) from Cuttng-edge Urban Deelopment-Korean Land Spatalzaton Research Proect funded by Mnstry of Land, Transport and Martme Affars of Korea, by the supportng proect to educate GIS eperts funded by Mnstry of Land, Transport and Martme Affars of Korea, and by the proect Three-dmensonalzng surellance network funded by GEOIDE of Canada. REFERENCES Agapto, L., Hayman, E., and Red, I., 21. Self-calbraton of rotatng and zoomng cameras. Internatonal Journal of Computer Vson, 45(2), pp. 17-127. Fschler, M. A., and Bolles, R. C., 1981. Random sample consensus A paradgm for model-fttng wth applcatons to mage-analyss and automated cartography. Communcatons of the ACM, 24(6), pp. 381-395. Gullou, E., Meneeau, D., Masel, E., and Bouatouch, K., 2. Usng anshng ponts for camera calbraton and coarse 3D reconstructon from a sngle mage. The Vsual Computer, 16, pp. 396-41. Km, H., and Hong, K. S., 2. A practcal self-calbraton method of rotatng and zoomng cameras. Pattern Recognton, 2. Proceedngs. 15 th Internatonal Conference on, 1, pp. 354-357. Lowe, D. G., 24. Dstncte mage features from scalenarant keyponts. Internatonal Journal of Computer Vson, 6(2), pp. 91-11. Canon, http://www.usa.canon.com (accessed 21) Fung, N., and Dad, P., 29. Implementaton of effcent pantlt-zoom camera calbraton. Army Research Laboratory. http://www.dtc.ml/cgbn/gettrdoc?ad=ada497476&locaton=u2&doc=gettr Doc.pdf (accessed 21) Hess, R., http://web.engr.oregonstate.edu/~hess/ (accessed 21) OpenCV2., (accessed 21) http://sourceforge.net/proects/openclbrary/ Photonotes.org, http://photonotes.org (accessed 21)