Image-processing Based Panoramic Camera Employing Single Fisheye Lens

Jounal of the Optical Society of Koea Vol. 4, No. 3, Septembe 200, pp. 245-259 DOI: 0.3807/JOSK.200.4.3.245 Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens Gyeong-il Kweon* Nanophotonics Co., Ltd. Rm 206, Goldenvil, 9-5 Sanjeong-dong, Gwangsan-gu, Gwangju 506-053, Koea Young-ho Choi Depatment of Infomation and Communication Engineeing, Honam Univesity, 59- Seobong-dong, Gwangsan-gu, Gwangju 506-74, Koea (Received May 7, 200 : evised August 23, 200 : accepted August 3, 200) We have developed mathematically pecise image-pocessing algoithms fo extacting panoamic images fom fisheye images. Futhemoe, we have successfully built a DSP-based panoamic camea employing single fisheye lens. Keywods : Image pocessing, Fisheye lens, Panoama, Cylindical pojection, Mecato pojection OCIS codes : (00.2000) Digital image pocessing; (20.4820) Optical systems; (220.4830) Optical systems design I. INTRODUCTION Thee have been many studies of and much development of panoamic imaging systems not only in the taditional aeas such as photogaphing buildings, natue scenes, and heavenly bodies, but also in secuity/suveillance systems using CCD (chage-coupled device) o CMOS (complementay metal-oxide-semiconducto) cameas, in vitual touing of eal estate, hotels and touist esots, and in navigational aids fo mobile obots and unmanned aeial vehicles (UAV) [-22]. Refeence 9 povides examples of steeo panoamic images poduced by Pofesso Paul Bouke. Each of the panoamic images follows a cylindical pojection scheme, and a panoamic image of an imaginay scene poduced by a compute as well as a panoamic image poduced by a otating slit camea ae pesented. Fo panoamic images poduced by a compute o poduced by a taditional method of otating slit camea, the lens distotion is not an impotant issue. Povided the lens mounted on the otating head panoamic camea is a distotion-fee ectilinea wideangle lens, the panoamic image poduced by the otating head panoamic camea is an ideal panoamic image having a cylindical pojection scheme. Howeve, a otating slit camea cannot be used to take a eal-time panoamic movie of a eal wold. Futhemoe, the hoizontal field of view of a commecial otating head panoamic camea is limited to aound 40º. A cheape altenative to the panoamic image acquisition method by the peviously descibed camea with a hoizontally-otating lens, consists of taking an image with an odinay camea with the optical axis hoizontally aligned, and epeating to take pictues afte hoizontally otating the optical axis by a cetain amount. Fou to eight pictues ae taken in this way, and a panoamic image with a cylindical pojection scheme can be obtained by seamlessly joining the pictues consecutively. Such a technique is called stitching. QuickTime VR fom Apple compute inc. is commecial softwae suppoting this stitching technology[9-0]. This method equies a complex, time-consuming, and elaboate opeation of pecisely joining seveal pictues and coecting the lens distotion. As anothe viable method of obtaining panoamic images, people ae actively eseaching catadioptic panoamic imaging systems, which ae imaging systems employing both mios and efactive lenses[4, 6,, 3-5, 8, 20-22]. Catadioptic panoamic lenses take panoamic images in one shot with the optical axes of the panoamic lenses aligned vetical to the gound plane. By popely designing the mio pofile, the necessay image pocessing load can be kept at a *Coesponding autho: kweon@nanophotonics.co.k Colo vesions of one o moe of the figues in this pape ae available online. - 245 -

246 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 minimum[20-22]. Nevetheless, catadioptic panoamic lenses tend to be athe bulky and costly. Anothe method of obtaining a panoamic image is to employ a fisheye lens with a wide field of view (FOV). Fo example, the entie sky and the hoizon can be captued in a single image by pointing a camea equipped with a fisheye lens with 80 FOV towad the zenith[]. Fo this eason, fisheye lenses have often been efeed to as all-sky lenses. In paticula, a high-end fisheye lens by Nikon, namely, 6mm f/5.6 Fisheye-Nikko, has a FOV of 220. Theefoe, a camea equipped with this lens can captue a potion of the view behind the camea as well as in font of the camea. Then, afte pope image pocessing, a panoamic image can be obtained fom the fisheye image. Image pocessing on fisheye image has been an active investigation aea fo the past seveal decades[2-3, 23-26]. Fom anothe point of view, all animals and plants including humans ae bound on the suface of the eath due to the gavitational pull, and most of the events which need attention o cautionay measues, take place nea the hoizon. Theefoe, even though it is necessay to monito evey 360 diection on the hoizon, it is not as impotant to monito high along the vetical diection, fo example, as high as to the zenith o deep down to the nadi. Distotion is unavoidable if we want to descibe the scene of evey 360 diection on a two-dimensional plane. Simila difficulty exists in the catogaphy whee geogaphy on eath, which is a stuctue on the suface of a sphee, needs to be mapped on a plana two-dimensional atlas. Descibed in efeence 27 ae the well-known map pojection schemes among the divese map pojection schemes such as equi-ectangula pojection, Mecato pojection and cylindical pojection schemes, and efeence 28 povides a bief histoy of divese map pojection schemes. Among these, the equi-ectangula pojection scheme is the pojection scheme most familia to us when we descibe the geogaphy on the eath, o when we daw the celestial sphee in ode to make a map of the constellations. Among all the distotions, the distotion that appeas most unnatual to people is the distotion whee vetical lines appea as cuved lines. Fo example, people expect a peson in a panoamic image to be standing in an upight position. Othewise, the peson may appea as having been maimed o as falling down, neithe of which is a pleasant sight. Theefoe, even if othe kinds of distotions ae pesent, it is impotant to make sue that this kind of distotion is absent. The goal of this investigation is to povide methods of extacting mathematically pecise panoamic images and to build a panoamic camea poviding such images. II. PROJECTION SCHEMES OF FISHEYE LENS AND IDEAL PANORAMIC LENS Figue is a schematic diagam illustating the pojection Z θ R i Y N FIG.. A schematic diagam illustating the eal pojection scheme of a geneal otationally symmetic lens. scheme of a geneal wide-angle lens such as a fisheye lens[26]. The optical axis coincides with the Z-axis of the coodinate system, and the incidence angle θ of an incident ay R i is measued as a zenith angle. All the ays foming image points on the image senso plane S ae consideed to pass though the nodal point N of the lens. The intesection between the optical axis and the image senso plane S is designated as oigin O in Fig.. The efacted ay R coesponds to the incident ay R i, and foms an image point P on the image senso plane S. The adial distance fom the oigin O to the image point P is the image height. The geneal pojection scheme of a lens can be defined as = (θ), whee the image height is a monotonically inceasing function of the incidence angle θ. Such a eal pojection scheme of a lens can be expeimentally measued using an actual lens, o can be calculated fom the lens pesciption using dedicated lens design softwae such as Code V o Zemax. Figue 2(a) is an imaginay inteio scene poduced by Pofesso Paul Bouke by using a compute, and it has been assumed that the imaginay lens used to captue the scene is a fisheye lens with 80 FOV having an ideal equidistance pojection scheme. This image is a squae image, of which both the lateal and the longitudinal dimensions ae 250 pixels. Theefoe, the coodinate of the optical axis is (25.5, 25.5), and the image height fo an incident ay with a zenith angle of 90 is given as '(π/2) = 250 25.5 = 24.5. Hee, ' is not a physical distance, but an image height measued in pixel distance. On the othe hand, Fig. 2(b) is a fisheye image of an inteio scene having 90 FOV. The eal pojection scheme of the fisheye lens used to captue this image is descibed in detail in efeence 29. The pojection scheme of a fisheye lens is diffeent fom the ideal pojection scheme of a panoamic lens. R P O S

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 247 (a) (a) (b) FIG. 2. An example of images (a) poduced by a compute assuming that a fisheye lens with an equidistance pojection scheme has been used to take the pictue of an imaginay scene (b) obtained using a fisheye lens having 90º FOV. Figue 3(a) shows the wold coodinate system descibing the object points in the outside wold in the spheical pola coodinate system and Fig. 3(b) descibe the same in the longitude-latitude system. The nodal point of the lens is taken as the oigin of the wold coodinate system, the vetical axis is taken as the Y-axis, and the optical axis is taken as the Z-axis. As has been illustated in the figue, an object point Q is shown at a distance R fom the oigin N. All the object points ae assumed as lying on a spheical object suface with a adius R. An incident ay oiginating fom this object point Q will have a zenith angle θ and an azimuth angle in the wold coodinate system. Refeing to Fig. 3(b), the same incident ay will have a hoizontal incidence angle ψ and a vetical incidence angle δ. Since thee is no possibility of confusion, we will also designate the object suface as a celestial sphee, the hoizontal incidence angle ψ as a longitude and the vetical (b) FIG. 3. A schematic diagam of the wold coodinate (a) in the spheical pola coodinate system and (b) in the longitudelatitude system. incidence angle δ as a latitude. On the othe hand, Fig. 4 is a schematic diagam of a plana map mapping the spheical suface on a two-dimension plane. A point Q on the object suface having a longitude ψ and a latitude δ has a coesponding point P on the plana map. The ectangula coodinate of this coesponding point is given as (x, y). Futhemoe, the efeence point on the equato having a longitude 0 and a latitude 0 has a coesponding point O on the plana map, and this coesponding point O is the oigin of the ectangula

248 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 FIG. 4. A conceptual dawing of a plana map mapping the suface of a sphee onto a two dimensional plane. coodinate system. Fo the mapping to be a useful one, the lateal coodinate x must be a function only of the hoizontal incidence angle(i.e., x = x(ψ)) and the longitudinal coodinate y must be a function only of the vetical incidence angle(i.e., y = y( )). Futhemoe, fo the mapping in Fig. 4 to be panoamic, the same inteval in the longitude (i.e., the same angula distance along the equato) must coesponds to the same lateal inteval on the plana map. In othe wods, the lateal coodinate x on the plana map is popotional to the longitude. x = cψ () Hee, c is a popotionality constant. The exact functional dependence of the longitudinal coodinate y on the latitude depends on the paticula pojection scheme. If the plana map follows an equi-ectangula pojection scheme, then the longitudinal coodinate y is also popotional to the latitude, and has the same popotionality constant as the lateal coodinate. y = cδ (2) The span of the longitude is 360º anging fom -80 to +80, and the span of the latitude is 80 anging fom -90 to +90. Theefoe, a map dawn accoding to the equi-ectangula pojection scheme must have a width W: height H atio of 360:80 = 2:. Such an equi-ectangula pojection scheme appeas as a natual pojection scheme fo mapping the Eath s suface consideing the fact that the Eath s suface is close to a spheical suface. Nevetheless, it is disadvantageous in that the size of a geogaphical aea is geatly distoted. Fo example, two vey close points nea the Noth Pole can appea as if they ae on the opposite sides of the Eath in a map dawn accoding to the equi-ectangula pojection scheme. Fig. 5 is a conceptual dawing of a cylindical pojection scheme o a panoamic pespective. In a cylindical pojection FIG. 5. A conceptual dawing illustating a cylindical panoama. scheme, an imaginay obseve is located at the cente N of a celestial sphee with a adius R, and it is desied to make a map of the celestial sphee centeed on the obseve, the map coveing most of the egion excluding the zenith and the nadi. In othe wods, the span of the longitude can be as lage as 360 anging fom -80 to +80, but the span of the latitude is naow anging fom -Δ to +Δ, whee Δ must be smalle than 90. In this pojection scheme, a hypothetical cylindical plane is assumed which contacts the celestial sphee at the equato. Then, fo a point Q(ψ, δ) on the celestial sphee having a given longitude ψ and a latitude δ, a line segment connecting the cente N of the celestial sphee and the object point Q is extended until it meets the cylindical plane. This intesection point is designated as P(ψ, δ). In this manne, the coesponding point P on the cylindical plane can be obtained fo evey object point Q on the celestial sphee within the given latitude ange. Then, a map having a cylindical pojection scheme is obtained by cutting the cylindical plane and laying flat on a plana suface. Theefoe, the lateal coodinate of the point P on the flattened-out cylindical plane is given by Eq., and the longitudinal coodinate y is given by Eq. 3. y = c tan δ (3) Such a cylindical pojection scheme is the natual

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 249 pojection scheme fo a panoamic camea that poduces a panoamic image by otating in the hoizontal plane. Especially, if the lens mounted on the otating panoamic camea is a distotion-fee ectilinea lens, then the esulting panoamic image exactly follows a cylindical pojection scheme. In pinciple, such a cylindical pojection scheme is the most accuate panoamic pojection scheme. Howeve, the panoamic image appeas unnatual when the latitudinal ange is lage, and thus it is not widely used in pactice. Anothe widely used pojection scheme is the Mecato pojection scheme. In a map dawn accoding to the Mecato pojection scheme, the longitudinal coodinate is given as a complex function given in Eq. 4. π δ y = c ln tan + (4) 4 2 The Mecato pojection scheme does not have any obvious geometical meaning. The Mecato pojection scheme is elated to sinh function while the cylindical pojection scheme is elated to tan function. III. CHARACTERISTICS OF CYLINDRICAL PANORAMIC IMAGE The pojection scheme of a fisheye lens is diffeent fom the ideal pojection scheme of a panoamic lens. Figue 6 illustates the pojection scheme of a hypothetical panoamic lens poviding a cylindical panoama. The panoamic lens is assumed as attached on a vetical wall. The wall coincides with the X-Y plane of the wold coodinate system, and the Y-axis uns fom the gound plane(i.e., X-Z plane) to the zenith. The oigin of the coodinate is located at the nodal point N of the lens, and the optical axis of the lens coincides with the Z-axis. The wold coodinate system is a coodinate system fo descibing the envionment that is captued by the lens. In a igoous sense, the diection of the optical axis is the diection of the negative Z-axis of the wold coodinate system. This is because, by the notational convention of imaging optics, the diection fom the object(o, an object point) to the image plane(o, an image point) is the positive diection. Despite this fact, we will descibe the optical axis as coinciding with the Z-axis of the wold coodinate system fo the sake of simplicity in agument. The image senso plane S is a plane having a ectangula shape and pependicula to the optical axis, wheeof the lateal dimension is B, and the longitudinal dimension is V. Hee, we assume a fist ectangula coodinate system, wheein the nodal point N of the lens is taken as the oigin, and the optical axis is taken as the negative z-axis. In othe wods, the diection of the z-axis is the exact opposite diection of the Z-axis. The intesection FIG. 6. A conceptual dawing illustating a panoamic camea having a cylindical panoamic pojection scheme. point between the z-axis and the image senso plane S is O. The x-axis of the fist ectangula coodinate system passes though the intesection point O and is paallel to the lateal side of the image senso plane, and the y-axis passes though the intesection point O and is paallel to the longitudinal side of the image senso plane. The X-axis of the wold coodinate system is paallel to the x-axis of the fist ectangula coodinate system, and points in the same diection. On the othe hand, the Y-axis of the wold coodinate system is paallel to the y-axis of the fist ectangula coodinate system, but the diection of the Y-axis is the exact opposite of the diection of the y-axis. Theefoe, in Fig. 6, the positive diection of the x-axis of the fist ectangula coodinate system uns fom the left to the ight, and the positive diection of the y-axis uns fom the top to the bottom. This complies with the convention in digital image pocessing. The intesection point O between the z-axis of the fist ectangula coodinate system and the senso plane S will be efeed to as the fist intesection point. The fist intesection point is not geneally located at the cente of the senso plane, and it can even be located outside the senso plane. Such a case can happen when the cente of the image senso is moved away fom the cente position of the lens - i.e., the optical axis - on pupose in ode to obtain an asymmetic vetical o hoizontal field of view. The lateal coodinate x of an abitay point P - heeinafte efeed to as the fist point - on the senso plane has a minimum value x and a maximum value x 2(i.e., x x x 2). By definition, the diffeence between the maximum lateal coodinate and the minimum lateal coodinate is the lateal dimension of the senso plane (i.e., x 2 - x = B). In the same manne, the longitudinal coodinate y of the fist point P has a minimum value y and a maximum value y 2(i.e., y y y 2). By definition, the diffeence between the maximum longitudinal coodi-

250 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 nate and the minimum longitudinal coodinate is the longitudinal dimension of the senso plane (i.e., y 2 - y = V). Schematically shown in Fig. 6, a hypothetical panoamic lens poviding a cylindical panoamic image assumes a hemi-cylindical object plane I with a adius Γ and having the Y-axis as the otational symmety axis, and the image of an abitay object point Q on the object plane appeas as an image point P on the senso plane S. The image of an object on the hemi-cylindical object plane I is captued on the senso plane with its vetical popotions peseved, and the lateal coodinate x of the image point is popotional to the hoizontal ac length of the coesponding object point on the object plane, and the image points on the image senso plane by all the object points on the object plane collectively fom a eal image. When such a condition is satisfied, the obtained image follows a ectilinea pojection scheme in the vetical diection, and follows an equidistance pojection scheme in the hoizontal diection. An abitay otationally symmetic lens including a fisheye lens, howeve, does not follow such a pojection scheme. Theefoe, to ealize such a pojection scheme, an image pocessing stage is inevitable. Fig. 7(a) is a conceptual dawing of an uncoected image plane pio to the image pocessing stage, which coesponds to the image senso plane S. If the lateal dimension of the image senso plane S is B and the longitudinal dimension is V, then the lateal dimension of the uncoected image plane is gb and the longitudinal dimension is gv, whee g is a popotionality constant [26]. The fist intesection point O is the intesection point between the optical axis and the image senso plane S. Theefoe, a ay enteed along the optical axis foms an image point on the fist intesection point O. By definition, the hoizontal incidence angle ψ and the vetical incidence angle δ of a ay enteed along the optical axis ae both zeo. Theefoe, the point O' on the uncoected image plane coesponding to the fist intesection point O in the image senso plane - heeinafte efeed to as the second intesection point - coesponds to the image point by an incident ay having a hoizontal incidence angle of 0º as well as a vetical incidence angle of 0º. A second ectangula coodinate systems is assumed wheein x'-axis is taken as the axis that passes though the second intesection point O' and is paallel to the lateal side of the uncoected image plane, and y'-axis is taken as the axis that passes though the second intesection point and is paallel to the longitudinal side of the uncoected image plane. In Fig. 7(a), the positive diection of the x'-axis uns fom the left to the ight, and the positive diection of the y'-axis uns fom the top to the bottom. Then, the lateal coodinate x' of an abitay image point P' on the uncoected image plane has a minimum value x' = gx and a maximum value x' 2 = gx 2(i.e., gx x' gx 2). In the same manne, the longitudinal coodinate y' of the image point has a minimum value y' = gy and a (a) (b) FIG. 7. Conceptual dawings of (a) an uncoected image plane (b) a pocessed image plane that is displayed on a monito. maximum value y' 2 = gy 2 (i.e., gy y' gy 2 ). Figue 7(b) is a conceptual dawing of a pocessed image plane showing ideal panoamic images. The pocessed image plane has a ectangula shape, wheeof the lateal side measues as W and the longitudinal side measues as H. Futhemoe, a thid ectangula coodinate system is assumed wheein x''-axis is paallel to the lateal side of the pocessed image plane, and y''-axis is paallel to the longitudinal side of the pocessed image plane. The z''-axis of the thid ectangula coodinate system is paallel to the z-axis of the fist ectangula coodinate system and the z'-axis of the second ectangula coodinate system. The intesection point O'' between the z''-axis and the pocessed image plane can take an abitay position, and it can even be located outside the pocessed image plane. In Fig. 7(b), the positive diection of the x''-axis uns fom the left to the ight, and the positive diection of the y''-axis uns fom the top to the bottom. Table summaizes the vaious coodinate systems defined in this section.

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 25 TABLE. Coespondences between diffeent planes defined in this aticle Suface Object plane Image senso plane Uncoected image plane Pocessed image plane Lateal dimension of the plane L B gb W Longitudinal dimension of the plane T V gv H Coodinate system Location of the coodinate oigin Symbol of the intesection point with the optical axis wold coodinate system nodal point of the lens the fist ectangula coodinate system nodal point of the lens the second ectangula coodinate system nodal point of the lens the thid ectangula coodinate system nodal point of the lens O O' O'' Coodinate axis (X, Y, Z) (x, y, z) (x', y', z') (x'', y'', z'') Alias of the object point o the image points object point the fist point the second point the thid point Symbol of the object point o the image point Q P P' P'' Two-dimensional coodinate of the image point (x, y) (x', y') (x'', y'') (a) (b) FIG. 8. Conceptual dawing of (a) a hoizontal coss-section of the object plane (b) a vetical coss-section of the object plane. Figue 8(a) shows the coss-section of the object plane I in Fig. 6 in the X-Z plane. The hoizontal FOV of the imaging system is not necessaily 80º, and it can be smalle o lage than that. Fo this eason, illustated in Fig. 8(a) is a case whee the FOV is lage than 80º. The hoizontal incidence angle of an abitay incident ay R i impinging on the imaging system, which is the angle subtended by the incident ay and the Y-Z plane, is ψ. In othe wods, it is the incidence angle in the hoizontal diection with espect to the Z-axis(i.e., the optical axis) in the X-Z plane(i.e., the gound plane). The minimum value of the hoizontal incidence angle is ψ, the maximum incidence angle is ψ 2(i.e., ψ ψ ψ 2), and the hoizontal FOV is Δψ = ψ 2 - ψ. In geneal, if the hoizontal FOV is 80º, then a desiable ange of the hoizontal incidence angle will be given by ψ 2 = -ψ = 90º. Since the adius of the object plane is Γ, the ac length of the object plane is given by Eq. 5.

252 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 ( ψ ψ ) = ΓΔ ψ L = Γ 2 (5) Hee, it has been assumed that the unit of the field of view Δψ is adians. This ac length L must be popotional to the lateal dimension W of the pocessed image plane. Theefoe, if this popotionality constant is c, then the following Eq. 6 is satisfied. L = cw (6) On the othe hand, Fig. 8(b) shows the coss-section of the object plane I in Fig. 6 in the Y-Z plane. The adius of the object plane I is Γ, and the height of the object plane is T. The vetical incidence angle of a ay enteing into the lens, which is the angle with espect to the Z-axis(i.e., the optical axis) in the Y-Z plane(i.e., a plane containing a vetical line), is δ. In othe wods, the vetical incidence angle the incident ay makes with the X-Z plane is δ. The minimum value of this vetical incidence angle is δ, and the maximum value is δ 2 (i.e., δ δ δ 2 ). When the vetical FOV is Δδ = δ 2 - δ, it is simple if the ange of the vetical incidence angle is given as δ 2 = - δ = Δδ/2, but accoding to the needs, it may be desiable if the two values ae diffeent. Fo example, if it is installed on the oof of a vehicle, then it is desiable to mainly monito the aea above the hoizon, but if it is installed on an aiplane, it is desiable to mainly monito the aea below the hoizon. Hee, the height T of the object plane seen fom the oigin N of the coodinate system is given by Eq. 7. ( tan δ δ ) T = Γ 2 tan (7) Futhemoe, the height T of the object plane must satisfy the same popotionality elation with the height H of the pocessed image plane. T = ch (8) Equation 9 can be obtained fom Eqs. 5 and 6, wheein A is a constant. Γ W A = (9) c Δψ On the othe hand, Eq. 0 can be obtained fom Eqs. 7 and 8. H A = (0) tan δ 2 tan δ Theefoe, fom Eqs. 9 and 0, it can be seen that the following equation must be satisfied. W H Δψ = tan δ 2 tan δ () In most of the cases, it will be desiable if the ange of the hoizontal incidence angle and the ange of the vetical incidence angle ae symmetical. When designing a lens o evaluating the chaacteistics of a lens, the hoizontal FOV Δψ and the vetical FOV Δδ ae impotant paametes. Fom Eq., it can be seen that the symmetical vetical FOV must be given as in Eq. 2 as a function of the symmetical hoizontal FOV. Δ δ = 2 tan H 2W Δ ψ (2) Moe geneally, when the pocedue fom Eq. 5 though Eq. 0 is epeated on an inteval containing the thid intesection point O'', then Eq. 3 can be obtained. A H y y 2 2 = = = = = = = = tanδ 2 tanδ tanδ tanδ 2 tanδ Δψ ψ ψ ψ (3) 2 Theefoe, when setting-up the desiable size of the pocessed image plane and the FOV, it must be ensued that Eq. 3 is satisfied. IV. IMAGE-PROCESSING ALGORITHM FOR OBTAINING PANORAMIC IMAGES FROM FISHEYE IMAGES If the pocessed image plane in Fig. 7(b) satisfies the cylindical panoamic pojection scheme, then the hoizontal incidence angle of an incident ay coesponding to the lateal coodinate x'' of a thid point P'' on the pocessed image plane is given by Eq. 4. Δψ x ψ = x = (4) W A Likewise, the vetical incidence angle of an incident ay coesponding to the thid point having a longitudinal coodinate y'' is given as Eq. 5. y δ = tan (5) A Theefoe, the signal value of a thid point on the pocessed image plane having an ideal cylindical panoamic pojection scheme must be given as the signal value of an y W x x x

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 253 image point on the image senso plane fomed by an incident ay oiginating fom an object point on the object plane having a hoizontal incidence angle(i.e., the longitude) given by Eq. 4 and a vetical incidence angle(i.e., the latitude) given by Eq. 5. The location of the object point Q on the object plane having given hoizontal and vetical incidence angles can be obtained by the following method. Refeing to Fig. 3(b), a vecto fom the oigin N of the wold coodinate system to an object point Q on the object plane having given hoizontal and vetical incidence angles can be witten as R. The diection of this vecto is the exact opposite of the popagation diection of the incident ay, and this vecto in the wold coodinate system can be witten as Eq. 6. R = XXˆ + YYˆ + ZZˆ (6) In Eq. 6, = (,0,0) is the unit vecto along the X-axis, and likewise, = (0,,0) and = (0,0,) ae the unit vectos along the Y-axis and the Z-axis, espectively. On the othe hand, the vecto R can be given in the spheical pola coodinate system as a function of the zenith angle θ and the azimuth angle as given in Eq. 7. Y = R sin δ (24) Z = R cos δ cos ψ (25) Using Eqs. 23 though 25, the hoizontal and the vetical incidence angles can be obtained fom the ectangula coodinate (X, Y, Z) of the object point as in Eqs. 26 and 27. X ψ = tan (26) Z Y δ = tan (27) 2 2 X + Z Since the coodinates given in the spheical pola coodinate system and in the cylindical pola coodinate system must agee, the following elations given in Eqs. 28 though 30 must hold. sin θ cos φ = cos δ sin ψ (28) R = R ˆR ( θ, φ ) (7) sin θ sin φ = sin δ (29) Hee, R is the magnitude of the vecto R and Rˆ is the diection vecto. Then, the following elation holds between the ectangula coodinate and the spheical pola coodinate. X = Xˆ R = R sin θ cos φ Y = Yˆ R = R sin θ sin φ (8) (9) cos θ = cos δ cos ψ (30) Eq. 3 can be obtained by dividing Eq. 29 by Eq. 28. tan δ tan φ = (3) sin ψ Z = Zˆ R = R cos θ (20) Theefoe, the azimuth angle is given by Eq. 32. 2 2 2 2 (2) R R = X + Y + Z = R In Eqs. 8 though 2, dot( ) epesents a scala poduct. On the othe hand, the diection vecto can be given by Eq. 22 as a function of two incidence angles descibing the pojection scheme, namely the hoizontal incidence angle ψ and the vetical incidence angle. Heeinafte, this coodinate system will be efeed to as a cylindical pola coodinate system. R = R ˆR ( ψ, δ ) (22) Using these two incidence angles, the ectangula coodinate can be given as follows. X = R cos δ sin ψ (23) tan δ φ = tan (32) sin ψ Fom Eq. 30, the zenith angle θ is given by Eq. 33. ( cos δ cosψ ) θ = cos (33) Theefoe, an incident ay having a hoizontal incidence angle ψ and a vetical incidence angle is an incident ay in the spheical pola coodinate system having a zenith angle θ given by Eq. 33 and an azimuth angle given by Eq. 32. In ode to pocess an image, the position on the image senso plane coesponding to an incident ay having such a zenith angle θ and an azimuth angle must be detemined. It has been assumed that the pojection scheme of a

254 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 lens is given as a geneal function of the zenith angle θ of the incident ay as given in Eq. 34. ( θ ) = (34) This function is a monotonically inceasing function of the zenith angle θ of the incident ay. Figue 9 is a conceptual dawing illustating the convesion elation between the ectangula coodinate and the pola coodinate of the second point P' on the uncoected image plane coesponding to the fist point on the senso plane. Refeing to Fig. 9, the two-dimensional ectangula coodinate (x', y') of the second point on the uncoected image plane can be obtained fom the two-dimensional pola coodinate (', ' ) as in Eqs. 35 and 36. g ( θ ) cos φ (35) x = g ( θ ) sin φ (36) y = Using Eqs. 4 though 36, a panoamic image having an ideal pojection scheme can be extacted fom an image acquied using a fisheye lens exhibiting a distotion abeation. Fist, depending on the use's need, a desiable size (W, H) of the panoamic image and the location of the thid intesection point O" ae detemined. The thid intesection point can be located even outside the pocessed image plane. In othe wods, the ange of the lateal coodinate (x'' x'' x'' 2 ) on the pocessed image plane as well as the ange of the longitudinal coodinate (y'' y'' y'' 2 ) can take abitay eal numbes. Also, the hoizontal FOV Δψ of this panoamic image(i.e., the pocessed image plane) FIG. 9. Anothe schematic diagam of an uncoected image plane. is detemined. Then, the hoizontal incidence angle ψ and the vetical incidence angle δ of an incident ay coesponding to the thid point in the panoamic image having a ectangula coodinate (x'', y'') can be obtained using Eqs. 4 and 5. Then, the zenith angle θ and the azimuth angle of an incident ay having given hoizontal and the vetical incidence angles ae calculated using Eqs. 32 and 33. Next, the eal image height coesponding to the zenith angle θ of the incident ay is obtained using Eq. 34. Utilizing the eal image height, the magnification atio g, and the azimuth angle of the incident ay, the ectangula coodinate (x', y') of the image point on the uncoected image plane is obtained using Eqs. 35 and 36. In this pocedue, the coodinate of the second intesection point on the uncoected image plane, o equivalently the location of the fist intesection point on the senso plane has to be accuately detemined. Such a location of the intesection point can be easily found using vaious methods including image pocessing method. Since such technique is well known to the people in this field, it will not be descibed in this document. Finally, the video signal (i.e., RGB signal) fom the image point by the fisheye lens having this ectangula coodinate is given as the video signal fo the image point on the panoamic image having the ectangula coodinate (x'', y''). A panoamic image having an ideal pojection scheme can be obtained by image pocessing fo all the image points on the pocessed image plane by the above-descibed method. V. QUANTIZATION EFFECTS A complication aises due to the fact that all the image sensos and display devices ae digitized devices. Pocessed image plane has pixels in the fom of a two-dimensional aay having J max columns in the lateal diection and I max ows in the longitudinal diection. Although, in geneal, each pixel has a squae shape with both the lateal dimension and the longitudinal dimension measuing as p, the lateal and the longitudinal dimensions of a pixel ae consideed as in the image pocessing field. To designate a paticula pixel P'', the ow numbe I and the column numbe J ae used. Theefoe, the signal stoed on this pixel can be designated as S(I, J). A pixel has a finite aea. To coect the distotion of a digitized image, the physical coodinate of an abitay pixel P'' is taken as the cente position of the pixel. Thee is an image point - i.e., the fist point - on the image senso plane coesponding to a pixel P'' on the pocessed image plane. The hoizontal incidence angle of an incident ay in the wold coodinate system foming an image at this fist point can be witten as ψ I,J ψ(i, J). Also, the vetical incidence angle can be witten as δ I,J δ(i, J). Incidentally, the location of this fist point does not geneally coincide with the exact location of any one pixel.

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 255 Hee, if the pocessed image plane coesponds to a panoamic image, then as given by Eq. 37, the hoizontal incidence angle must be a sole function of the lateal pixel coodinate J. ψ I, J = ψ J ψ ( J ) (37) Likewise, the vetical incidence angle must be a sole function of the longitudinal pixel coodinate I. δ = δ δ ( I ) I, J I (38) Futhemoe, if an equidistance pojection scheme is satisfied in the lateal diection, and a ectilinea pojection scheme is satisfied in the longitudinal diection, then the ange of the hoizontal incidence angle and the ange of the vetical incidence angle must satisfy the elation given in Eq. 39. J ψ max J max ψ I = tan δ max Im ax tan δ (39) Compaing with the image pocessing method descibed peviously, the image pocessing method fo a digitized image goes though the following pocedue. Fist, the eal pojection scheme of the wide-angle lens that is meant to be used in the image pocessing is obtained eithe by expeiment o based on the accuate lens pesciption. ( θ ) = (40) This function is a monotonically inceasing function of the zenith angle θ. Next, the location of the optical axis on the uncoected image plane, in othe wods, the location of the second intesection point O' coesponding to the fist intesection point O on the image senso plane is obtained. The pixel coodinate of this second intesection point is assumed as (K o, L o ). In addition to this, the magnification atio g of the pixel distance ' on the uncoected image plane ove the eal image height on the image senso plane is obtained. This magnification atio g is given by Eq. 4. g = (4) Once such a seies of pepaatoy stages have been completed, then a camea mounted with a fisheye lens is installed with its optical axis aligned paallel to the gound plane, and a aw image(i.e., an uncoected image plane) is acquied. Next, the desiable size of the pocessed image plane and the location (I o, J o ) of the thid intesection point is detemined, and then the hoizontal incidence angle ψ J and the vetical incidence angle δ I given by Eqs. 42 and 43 ae computed fo all the pixels (I, J) on the pocessed image plane. ψ J J max ψ ψ J = ( J J o ) (42) max ψ J max ψ δ I = tan ( I I o ) (43) J max Fom these hoizontal and vetical incidence angles, the zenith angle θ I,J and the azimuth angle I,J of the incident ay in the fist ectangula coodinate system ae obtained using Eqs. 44 and 45. θ I, J = cos ( cos δ I cos ψ J ) (44) tan δ I φ = I J tan, (45) sin ψ J Next, the image height I,J on the image senso plane is obtained using Eqs. 40 and 44. ( ) I, J θ I, J = (46) Next, using the location (K o, L o ) of the second intesection point on the uncoected image plane and the magnification atio g, the location of the second point on the uncoected image plane is obtained using Eqs. 47 and 48. (47) x I, J = Lo + gi, J cos φ I, J (48) y I, J = K o + gi, J sin φ I, J The location of the second point does not exactly coincide with the location of any one pixel. Theefoe, (x' I,J, y' I,J ) can be consideed as the coodinate of a vitual pixel on the uncoected image plane coesponding to the thid point on the pocessed image plane, and has a eal numbe value in geneal. Since the second point does not coincide with any one pixel, an intepolation method must be used fo image pocessing. Figue 0(a) is a panoamic image following a cylindical pojection scheme that has been extacted fom the image in Fig. 2(a), whee the lateal and the longitudinal dimensions ae all 250 pixels, and the thid intesection point is located at the cente of the pocessed image plane. Futhemoe, the hoizontal FOV of the pocessed image plane is 80 (i.e., π). As can be seen fom Fig. 0(a), all the vetical lines in the thee walls, namely the font, the

256 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 (a) (a) (b) FIG. 0. Panoamic images following cylindical pojection schemes extacted (a) fom the fisheye image given in Fig. 2(a) (b) fom the fisheye image given in Fig. 2(b). left, and the ight walls in Fig. 2(a) appea as staight lines in Fig. 0(a). The fact that all the vetical lines in the wold coodinate system appea as staight lines in the pocessed image plane is the chaacteistic of a panoamic image. On the othe hand, Fig. 0(b) is the panoamic image that has been extacted fom the image in Fig. 2(b) by following a cylindical pojection scheme. In a panoamic image following a cylindical pojection scheme, at least a potion on the top and the bottom egion in an oiginal fisheye image cannot appea in the panoamic image. This is due to the existence of the tangent function in the pojection scheme in the vetical diection. On the othe hand, Fig. (a) is a panoamic image following an equi-ectangula pojection scheme extacted fom the fisheye image given in Fig. 2(a), and Fig. (b) is a panoamic image following a Mecato pojection scheme. Especially in the panoamic image in Fig. (a) with an equi-ectangula pojection scheme, the hoizontal and the vetical FOVs ae both 80º. Theefoe, all the infomation that exists in the oiginal image also exists in the panoamic image, but no moe. (b) FIG.. Panoamic images extacted fom the fisheye image given in Fig. 2(a) following (a) equi-ectangula pojection scheme (b) Mecato pojection scheme. VI. PANORAMIC IMAGE PROCESSING MODULE Figue 2 is a schematic diagam of a panoamic image pocessing module. The image pocessing module has an input fame buffe stoing one fame of image acquied fom the camea mounted with a fisheye lens. The input fame buffe stoes a digital image acquied fom the camea in the fom of a two dimensional aay. This digital image is the uncoected image plane. The output fame buffe stoes an output signal in the fom of a two dimensional aay, which coesponds to the pocessed image plane that can be displayed on a monito. A cental pocessing unit(cpu) geneates a pocessed image plane fom the uncoected image plane existing in the input fame buffe and stoes in the output fame buffe. The mapping elation between the output fame buffe and the input fame buffe is stoed in a non-volatile memoy such as a NOR Flash in the fom of a lookup table(lut). In othe wods, a long

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 257 FIG. 4. The panoamic camea mounted on a side-window of a passenge ca. FIG. 2. A schematic diagam illustating an image-pocessing based panoamic camea. FIG. 5. Sample image obtained fom the expeimental set-up shown in Fig. 4. FIG. 3. The developed panoamic camea module. list of pixel addesses fo the input fame buffe coesponding to paticula pixels in the output fame buffe is geneated and stoed. Cental pocessing unit efes to this list stoed in the nonvolatile memoy in ode to pocess the image. If necessay, the opeation of the image pocessing module can be made to eact to input signals fom vaious sensos. Figue 3 shows a developed panoamic image pocessing module. The fisheye lens descibed in efeence 29 is used, and a pogessive scan CMOS image senso(model: MT9D3) fom Micon is used as the image senso. This senso is a /3.2-inch senso having 600 200 pixels. Since this is a SOC(System-on-a chip) senso, camea head boad is easily FIG. 6. The panoamic camea installed nea the ea bumpe of a passenge ca. built. A DSP chip (model: TMS320DM6437) has been chosen as the CPU. The CMOS image senso has been pogammed to opeate in peview mode, whee the 2Mega pixels image has been binned to output 800 600 images. Fom this uncoected image plane, pocessed image plane having 720 480 pixels(i.e., D-gade) has been geneated. Due to

258 Jounal of the Optical Society of Koea, Vol. 4, No. 3, Septembe 200 Fig. 6 shows the panoamic camea installed nea the ea bumpe of a passenge ca. Fig. 7 shows a seies of still images captued while paking the ca. VII. CONCLUSION (a) In conclusion, we have developed mathematically pecise image-pocessing algoithms fo extacting panoamic images fom fisheye images. Futhemoe, we have successfully built a DSP-based panoamic camea employing a single fisheye lens. Imaging systems using this method can be used not only in secuity/suveillance applications fo indoo and outdoo envionments, but also in divese aeas such as video phones fo apatment entance doos, ea view cameas fo vehicles, visual sensos fo unmanned aeial vehicles and obots, and boadcasting cameas. ACKNOWLEDGMENT (b) This wok was suppoted by gant No. RTI04-03-03 fom the Regional Technology Innovation Pogam of the Ministy of Knowledge Economy (MKE). We ae deeply gateful to Koean Intellectual Popety Office (KIPO) and Koea Invention Pomotion Association(KIPA) fo honoing us with the Gand Pize(Pesidential awad) in 2009 Koea Invention and Patent Exhibition(KINPEX). We ae also thankful to Pof. Bouke fo his geneous pemission to use synthetic fisheye image authoed by him. REFERENCES (c) FIG. 7. Seies of sample images obtained fom the expeimental set-up shown in Fig. 6. the size mismatch between the fisheye lens and the CMOS image senso, the captued image has a hoizontal FOV of 75º. The DSP chip has been veified to be able to pocess 30 fames/second fo D-gade panoamic images. Howeve, the fame ate has been limited by the CMOS image senso, which slows down unless the scene is bightly illuminated. Figue 4 shows the developed panoamic camea mounted on a side-window of a passenge ca, and Fig. 5 is a sample image captued while diving though a busy steet.. R. Hill, A lens fo whole sky photogaphs, Q. J. R. Meteo. Soc. 50, 227-235 (924). 2. M. Onoe and Y. Kuno, Digital pocessing of images taken by fish-eye lens, IEEE Poceedings, 05-08 (982). 3. N. L. Max, Compute gaphics distotion fo IMAX and OMNIMAX pojection, Poc. NICOGRAPH 83, 37-59 (983). 4. P. Geguss, A new device fo panoamic infaed photogaphy, Poc. Soc. Photo-Opt. Instum. Eng. 380, 93-99 (983). 5. N. Geene, Envionment mapping and othe applications of wold pojections, IEEE Compute Gaphics and Applications 6, 2-29 (986). 6. P. Geguss, Panoamic secuity, Poc. SPIE 509, 55-66 (99). 7. K. Yamazawa, Y. Yagi, and M. Yachida, Obstacle detection with omnidiectional image senso hypeomni vision, in Poc. The 995 IEEE Intenational Confeence on Robotics and Automation (Nagoya, Japan, 995), pp. 062-067. 8. S. E. Chen and G. S. P. Mille, Cylindical to plana image mapping using scanline coheence, U.S. Patent 5396583 (995). 9. T. R. Halfhill, See you aound, Byte, 85-90 (May 995).

Image-pocessing Based Panoamic Camea Employing Single Fisheye Lens - Gyeong-il Kweon et al. 259 0. S. E. Chen, Quicktime VR - an image-based appoach to vitual envionment navigation, ACM SIGGRAPH, 29-38 (995).. I. Powell, Design study of an infaed panoamic optical system, Appl. Opt. 35, 690-694 (996). 2. V. S. Nalwa, Panoamic pojection appaatus, U.S. Patent 5539483 (996). 3. J. S. Chahl and M. V. Sinivasan, Reflective sufaces fo panoamic imaging, Appl. Opt. 36, 8275-8285 (997). 4. S. Bake and S. K. Naya, A theoy of single-viewpoint catadioptic image fomation, Int. J. Compute Vision 35, 75-96 (999). 5. H. Ishiguo, Development of low-cost compact omnidiectional vision sensos, in Panoamic Vision: Sensos, Theoy, and Applications, R. Benosman and S. B. Kang, eds (Spinge, New Yok, USA, 200), pp. 23-38. 6. P. C. Teo, Method and system fo ceating panoamas, U.S. Patent 624643 (200). 7. L. D. Paulson, Viewing the wold though inteactive panoamic images, IEEE Compute 37, 28 (2004). 8. G. Kweon, K. Kim, G. Kim, and H. Kim, Folded catadioptic panoamic lens with an equidistance pojection scheme, Appl. Opt. 44, 2759-2767 (2005). 9. P. D. Bouke, Synthetic steeoscopic panoamic images, Lectue Notes in Compute Gaphics 4270, 47-55 (2006). 20. G. Kweon, K. Kim, Y. Choi, G. Kim, and S. Yang, Catadioptic panoamic lens with a ectilinea pojection scheme, J. Koean Phys. Soc. 48, 554-563 (2006). 2. G. Kweon and M. Laikin, Design of a mega-pixel gade catadioptic panoamic lens with the ectilinea pojection scheme, J. Opt. Soc. Koea 0, 67-75 (2006). 22. G. Kweon, Y. Choi, G. Kim, and S. Yang, Extaction of pespectively nomal images fom video sequences obtained using a catadioptic panoamic lens with the ectilinea pojection scheme, in Poc. Technical Poceedings of the 0th Wold Multi-Confeence on Systemics, Cybenetics, and Infomatics (Olando, FL, USA, Jun. 2006), pp. 67-75. 23. N. Alvetos, E. L. Hall, and R. L. Andeson, Omnidiectional viewing: the fish-eye lens poblem, in Poc. IEEE Southeastcon (Olando, FL, USA, Ap. 983), pp. 74-79. 24. M. Ehtashami, S. J. Oh, and E. L. Hall, Omnidiectional position location fo mobile obots, Poc. SPIE 52, 62-73 (984). 25. G. Kweon, Methods of obtaining panoamic images using otationally symmetic wide-angle lenses and devices, Koean Patent 0-08820 (2009). 26. G. Kweon, Panoamic image composed of multiple ectilinea images geneated fom a single fisheye image, J. Opt. Soc. Koea 4, 09-20 (200). 27. E. W. Weisstein, Cylindical Pojection, http://mathwold. wolfam.com/cylindicalpojection.html. 28. W. D. G. Cox, An intoduction to the theoy of pespective - pat, The Bitish Jounal of Photogaphy 4, 628-634 (969). 29. G. Kweon, Y. Choi, and M. Laikin, Fisheye lens fo image pocessing applications, J. Opt. Soc. Koea 2, 79-87 (2008).