Super-Resolution on Moving Objects using a Polygon-Based Object Description

Size: px

Start display at page:

Download "Super-Resolution on Moving Objects using a Polygon-Based Object Description"

Melina Robbins
5 years ago
Views:

1 Super-Resolution on Moving Objects using a Polgon-Based Object Description A. van Eekeren K. Schutte O.R. Oudegeest L.J. van Vliet TNO Defence, Securit and Safet, P.O. Box 96864, 2509 JG, The Hague, The Netherlands Quantitative Iaging Group Delft Universit of Technolog Lorentzweg 1, 2628 CJ, Delft, The Netherlands ada.vaneekeren@tno.nl klaer.schutte@tno.nl L.J.vanVliet@tudelft.nl Kewords: detection, oving object, polgon description, super-resolution, TOD Abstract Moving objects are often the ost interesting parts in iage s. When iages fro a caera are undersapled and the oving object is depicted sall on the iage plane, processing of the iage afterwards a help to iprove the visibilit / recognition of the object. This paper addresses this subject and presents an approach which perfors Super- Resolution () specificall on sall oving objects using a polgon-based object description. This approach is not bound to a finite pixel grid and has the advantage that less unknowns have to be estiated in the optiization process in coparison to a standard pixel-based algorith. This paper describes the setup of the proposed polgon-based algorith and shows its superior perforance in coparison to a pixel-based algorith. 1 Introduction Although the concept of Super-Resolution () alread exists for ore than 20 ears [1], not uch attention is given to a specific case: reconstruction on oving objects. In [2] we presented an algorith which siultaneousl perfors on the foreground and background of a scene. Both are processed separatel with a standard pixel-based algorith. This algorith works fine if a oving object (foreground) is relativel large (the nuber of boundar pixels is sall in coparison to the total nuber of object pixels) and contains enough inner structure to perit precise registration. However, if an object is sall (the nuber of boundar pixels coprises ore than fift percent of the total nuber of object pixels) and the object itself contains hardl an internal structure, a standard pixel-based approach will fail. On one hand there is not enough gradient inforation to perfor a proper registration of the object and on the other hand a standard pixel-based approach akes an error across the object boundar (assuing a cluttered background). This boundar region consists of so-called ixed pixels, which contain partl inforation fro the object and partl fro the background. To tackle the aforeentioned probles we propose to perfor on sall oving objects using a polgon-based object description. Assuing rigid objects with hoogeneous intensit that ove (constant speed and acceleration are assued) along a known trajector through the real world, a proper registration is done b fitting a odel trajector through the object s center of ass in each frae. The boundar of a oving object is odeled with a polgon description. With this odel a relation is obtained between the position of the edges of the polgon and the estiated intensities of the ixed pixels. B iniizing the odel error between the easured intensities and the estiated intensities in a least squared error sense, which is siilar to conventional pixel-based algoriths, a sub-pixel accurate polgon description is obtained. The paper is organized as follows. In Section 2 the setup of the proposed polgon-based algorith is presented. Section 3 describes the TOD-based perforance easureents of the polgon-based algorith copared to a conventional pixel-based algorith and it shows the results. In the last section conclusions are drawn.

2 2 Algorith Our algorith can be split up into three parts: 1) obtaining a background odel and oving object detection, 2) odel-based trajector fitting for object registration and 3) obtaining a sub-pixel accurate polgon description b solving an inverse proble. The latter is the ain part of the algorith and will be explained first. 2.1 Polgon description Given that a proper detection (object ask ) and registration (object shift vector ) of a oving object is obtained, a polgon description (ordered set of vertices) of the boundar of a oving object can be estiated. A flow diagra of the polgon description part is depicted in Figure 1. object ask r Object shift vector registered odeled a-priori knowledge Cost function Caera odel Polgon initialization p0 ~ iterative iniization p f p i polgon description Figure 1: Flow diagra of the polgon description part of the algorith Polgon initialization The optiization schee depicted in Figure 1 ust be initialized with an initial polgon description of the object. This is done b first appling a pixel-based algorith to the object ask and threshold that result. This so-called HR object ask is used to deterine the vertices of the initial polgon description p 0. In this paper the centre of gravit and the size of the HR object ask are used to construct p 0 as an equilateral triangle. For larger and ore coplex shapes, the boundar pixels of the HR object ask can be selected as vertices of p 0. To siplif this polgon description (and liit the nuber of unknowns) a Douglas-Peucker algorith can be used [3] Caera odel The Caera odel aps a polgon description to an estiated (odeled) Low-Resolution () iage. The estiated intensit of each pixel ỹ is calculated given a polgon description p and assuing a unifor rectangular pixel support (see eq. 1). ỹ (p) = I bg A bg + I fg A fg (1) Here I bg is the local background intensit, which is assued to be known fro a pixel-based ethod. The foreground (object) intensit I fg is assued to be constant and known. A bg is the partial area of the support of pixel that overlaps the background given the polgon description p. Hence, the partial area covering the foreground is defined as: A fg = 1 A bg. The width (w) of the pixel support is also assued constant Cost function The cost function that is iniized for p i is defined as: ( ỹ (p i )) 2 + µ... (2) where the first part (suation) iniizes the odel error for all pixels and the last part, which is left ept, can be used to regularize the process b incorporating a-priori knowledge (e.g. a odel of the expected object and/or a penalt for self-intersecting polgons). are the easured intensities of the registered object pixels and ỹ are the corresponding estiated intensities (eq. 1). The actual iniization of the cost function is done in an iterative wa with a large-scale algorith. This algorith is a subspace trust region ethod and is based on the interior-reflective Newton ethod. To show qualitativel the perforance of the polgon description part of the algorith, we fitted a polgon with 5 vertices to a siulated iage (10 fraes) containing a sub-pixel shifted binar object. The iage is noise free and is siulated with the sae caera odel as described in Section The resulting polgon (red) is superiposed on the first frae (see Figure 2). 2.2 Background odeling and Moving object detection Our detection of oving objects is based on the assuption that a oving object deviates fro a static background. Ideall, a static background is estiated using a robust pixel-based algorith [4, 5]. Such a robust algoriths provided a proper background estiate even in the presence of sall oving objects. In the experients as described in Section 3.2 a edian of all iage fraes is used as robust background estiate; in this case it is allowed as in the experient no otion in the background scene is present.

3 Hardie [6, 7]. The perforance is tested on siulated data containing a sall oving object on a cluttered background. 3.1 Triangle Orientation Discriination Figure 2: Polgon with 5 vertices fitted to undersapled data (10 fraes) of a binar object. The can points represent the ground-truth vertices used to siulate the data. The green points are the vertices used as initialization (b hand) and the red points are the final vertices found with the proposed polgonbased algorith. As a-priori knowledge is used that the polgon a not be self-intersecting and the angle between adjacent edges a not be saller than 20 degrees. The HR background odel, after shifting and downsapling with a caera odel, can be copared to each frae of the iage. After thresholding these difference iages and reoval of spurious detections with orphological operations, an object ask for each frae results. 2.3 Registration With the object asks, obtained fro the detection part, a global position of the object in each frae is known. For perforing however, a ver precise sub-pixel registration is needed. When sufficient internal structure is present, gradient-based registration can be perfored. In the setting of sall oving objects this is usuall not the case and therefore another approach is needed. Assuing that the contrast between object and background is sufficient, in each frae the Center Of Gravit (COG) of the object is calculated. When the otion odel of the oving object is known, such a odel is fit in a robust wa to the calculated COGs. We assue a linear otion odel in the real world, which sees realistic given the setting of sall oving objects: the objects are far awa fro the observer (viewer) and will have a sall displaceent in the iage plane due the high frae rate of toda s iage sensors. When enough (> 20) fraes are available, a robust fit to the COGs results in a sufficient precise and accurate sub-pixel registration of the oving object. 3 Experients To test the perforance of our polgon-based algorith for oving objects, it is copared in a quantitative wa with the perforance of a wellperforing pixel-based algorith developed b The TOD ethod is an evaluation ethod designed for sste perforance of a broad range of iaging sstes [8], but it can also be used to test the perforance of iage enhanceent techniques [9]. It is based on the observer task to discriinate four different oriented equilateral triangles (see Figure 3). up right down Figure 3: The four different stiuli used in the TOD ethod. The observer task is a four-alternative forcedchoice, so the observer has to indicate which of the four orientations is perceived, even if he is not sure. The probabilit of a correct observer response increases with the triangle size. For a nuber of observations at different contrast (Signal-to-Noise Ratio) a 75% correct triangle size is deterined. These 75% correct triangle sizes correspond with the perforance of the iaging sste under easureent. In our experient an autoatic observer is used that discriinates the orientation of the triangles after processing. Each triangle is copared with a database containing ideal binar triangles (see Figure 3) of different size. The orientation of the triangle that axiizes the siilarit easure is taken as the result. 3.2 Coparison polgon versus pixelbased approach The setup of the experient is depicted in Figure 4. triangle object ask Object shift r vector Polgon initialization p 0 Pixel-based a-priori knowledge Polgonbased p f left TOD TOD perforance easure pixelbased perforance easure polgon description Figure 4: Flow diagra of the coparison between a conventional pixel-based algorith and the proposed polgon-based algorith on oving objects. The perforance easure is obtained with the TOD ethod. Here, the triangle is a siulated, under-sapled, iage containing a oving triangle with hoogeneous intensit. The

data is obtained fro Hper-Resolution (HYR) data, which has a 17 higher resolution, through a siulated caera odel. This caera odel siulates onl the caera s sensor with a 100% fill-factor.

Note that both the pixel-based algorith as well as the polgonbased algorith are ipleented in such a wa that no odel errors are introduced.

4 data is obtained fro Hper-Resolution (HYR) data, which has a 17 higher resolution, through a siulated caera odel. This caera odel siulates onl the caera s sensor with a 100% fill-factor. This is accoplished b blurring the HYR data with a unifor rectangular filter (width = 17 pixels) followed b down-sapling with factor 17. Note that both the pixel-based algorith as well as the polgonbased algorith are ipleented in such a wa that no odel errors are introduced. To be able to appl the TOD ethod, different s with various triangle sizes and SNR s are siulated. An exaple of a HYR iage and its corresponding iage is depicted in Figure 5. On both iages the sub-pixel accurate polgon description as found b our algorith is superiposed in red. Here, α j is the inner angle of polgon p i at vertex j and α 0 is the preferred angle. σ α expresses the steepness of the joint probabilit densit function. The value of σ α should be seen in relation to the preferred angle α 0. Differences in angle saller than 5-10% should be allowed. In our experients α 0 = π/3 (equilateral triangle constraint) and σ α = π/60 are used. Furtherore a flat prior is used for the angle difference. The second ode onl assues that a triangle (three vertices) is present, which ust have a iniu angle (α in ) between its edges of 45 degrees. In this case the positions of all three vertices have to be estiated (six d.o.f.). The functional that is iniized for this ode is defined as: ( ỹ (p i )) 2 + P µ 2 s( (α j α in )/s 0 ) (4) j with (a) (b) Figure 5: (a) Part of original HYR iage containing a oving triangle of 32 pixels. (b) Corresponding iage (triangle size is here reduced to 31/17 = 1.88 pixels). On both iages the sub-pixel accurate polgon description as found b our algorith is superiposed in red. As is depicted in the upper part of Figure 4, the detected oving triangle data together with the easured object positions () is processed with a pixelbased algorith (Hardie). Afterwards the orientation of the triangle is deterined with the TOD ethod. In the lower part of Figure 4 the data is processed with the proposed polgon-based algorith, which allows us to use a-priori knowledge about the object. The polgon-based ethod is tested in two different odes: one using a lot of a-priori knowledge and one using onl a iniu aount of a-priori knowledge. In the first ode the following a-priori knowledge is assued: the polgon ust atch one of the four equilateral triangles as depicted in Figure 3. Therefore onl the position and size of the triangle have to be estiated (three degrees of freedo (d.o.f.)). The functional that is iniized for this ode is depicted in equation 3. ( ỹ (p i )) 2 + µ 1 (1 e P j (αj α0)2 /2σ 2 α ) (3) 1 s(t) =. (5) 1 + e t s(t) is the sigoid function, α in denotes a soft threshold on the vertex angle α j and s 0 controls the steepness of the penalt function. Please note the inus-sign in the arguent of s, which liits the penalt to angels saller than the soft threshold. In our experients α in = π/4 and s 0 = π/100 are used. In this ode also a flat prior is used for the angle difference. Furtherore, for both odes the foreground intensit and the width of the pixel support (w = 1 pixel) are assued to be known. Also the initialization step is the sae: according to the centre of ass and the size of the HR object ask (see Section 2.1.1) four different oriented equilateral triangles are used as initial polgon description p 0. This is done to prevent that the optiization process gets stuck in a local iniu. The polgon description (p f ) that results in the sallest value of functional C is used for the TOD easureent. The pose (up, right, down, left) of the equilateral triangle that has a iniu area error with p f is taken as the TOD easure. 3.3 Results The results obtained fro the coparison experient are depicted in Figure 6. Note that a saller triangle 75% threshold indicates a better perforance. The Signal-to-Noise Ratio (SNR) is defined here as 20 log 10 ( Ībg I fg /σ n ), with Ībg the ean intensit of the local background, I fg the foreground intensit and σ n the standard deviation of the noise.

5 Triangle 75% threshold ( pixels) Hardie polgon, 3 vert. a-priori ode 2 polgon, 3 vert. a-priori ode SNR (db) Figure 6: TOD perforance coparison between a polgon and a pixel-based approach on siulated data (effective unifor rectangular blur of 1 pixel, 40 fraes). Black line: Hardie, zoo 4, w = 1, λ = Red line: polgon description with 3 vertices using little a-priori knowledge, Blue line: polgon description with 3 vertices using uch a-priori knowledge. 4 Conclusions Fro the obtained results the following conclusions can be drawn: For high SNR s the polgon-based algorith with a-priori knowledge perfors significantl better than a conventional wellperforing pixel-based algorith according to the TOD easure considering a sall oving object on a cluttered background. When using less a-priori knowledge the perforance of our proposed algorith decreases to a level, coparable to that of the conventional pixel-based approach. [3] D. Douglas and T. Peucker. Algoriths for the reduction of the nuber of points required to represent a digitized line or its caricature. The Canadian Cartographer, 2(10): , [4] S. Farsiu, M.D. Robinson, M. Elad, and P. Milanfar. Fast and robust ulti-frae super resolution. IEEE Trans. Iage Processing, 13(10): , [5] T.Q. Pha, L.J. van Vliet, and K. Schutte. Robust fusion of irregularl sapled data using adaptive noralized convolution. EURASIP Journal on Applied Signal Processing, pages 1 12, [6] R.C. Hardie, K.J. Barnard, J.G. Bognar, E. Arstrong, and E.A. Watson. High-resolution iage reconstruction fro a of rotated and translated fraes and its application to an infrared iaging sste. Optical Engineering, 37(1): , [7] A. van Eekeren, K. Schutte, O.R. Oudegeest, and L.J. van Vliet. Perforance evaluation of superresolution reconstruction ethods on real-world data. Eurasip JASP, Special Issue on Super- Resolution Enhanceent of Digital Video, 2007 (under review). [8] P. Bijl and J.M. Valeton. Triangle orientation discriination: the alternative to iniu resolvable teperature difference and iniu resolvable contrast. Optical Engineering, 37(7): , [9] P. Bijl, K. Schutte, and M. Hogervorst. Applicabilit of tod, tdp, rt and drt for dnaic iage enhanceent techniques. In Infrared Iaging Sstes XVII, volue 6207, pages SPIE, For low SNR s the perforance is coparable to the pixel-based algorith. The tendenc of the perforance curves in Figure 6 shows us that the pixel-based ethod sees to be perforing better when SNR < 15 db. References [1] R.Y. Tsai and T.S. Huang. Multifrae iage restoration and registration. In Advances in Coputer Vision and Iage Proscessing, volue 1, pages JAI Press, [2] A. van Eekeren, K. Schutte, J. Dijk, D.J.J. de Lange, and L.J. van Vliet. Super-resolution on oving objects and background. In Proc. 13th International Conference on Iage Processing, volue 1, pages IEEE, 2006.

Super-Resolution on Moving Objects and Background

Super-Resolution on Moving Objects and Background A. van Eekeren K. Schutte J. Dijk D.J.J. de Lange L.J. van Vliet TNO Defence, Security and Safety, P.O. Box 96864, 2509 JG, The Hague, The Netherlands