Bias of Higher Order Predictive Interpolation for Sub-pixel Registration

Transcription

1 Bias of Higher Order Predictive Interpolation for Sb-pixel Registration Donald G Bailey Institte of Information Sciences and Technology Massey University Palmerston North, New Zealand D.G.Bailey@massey.ac.nz Andrew Gilman Institte of Information Sciences and Technology Massey University Palmerston North, New Zealand agilman@iee.org Abstract Sb-pixel registration has application in many image processing tasks. Predictive interpolation solves the problem of choosing a particlar interpolation fnction and needing to search for the best offset by determining the optimm interpolation fnction for a given pair of images, and estimating the offset from the interpolation weights. By analysing sinsoids and step edges, it is shown that even order predictive interpolation filters inherently have more bias than odd order filters. It is also demonstrated that increasing the filter order significantly improves registration bias. The form of the bias is verified by measring the accracy of registration on sample images. Keywords sb-pixel, sper-resoltion, registration, motion estimation, interpolation, linear prediction, imaging model I. INTRODUCTION Image registration is an important step in many image processing applications. While pixel accrate registration is adeqate for many applications, many techniqes can benefit from registration to sb-pixel accracy. These inclde: sperresoltion []; motion compensation for video coding []; sensor fsion [3]; stereo imaging; image stitching [4]; motion detection sing optical flow [5]; and image stabilisation. Given two images (or sbimages in some applications), f and g, which differ only by a translation (rotation and scaling are not considered in this paper), registration involves estimating the offset between the pixel locations of the two images. The general approach is to designate one of the images as the reference, and measre the offset of the other image relative to this reference. Sb-pixel registration reqires estimating the offset to an accracy of a small fraction of a pixel. Often this takes place in two steps: first the images are registered to the nearest pixel, and then the offset within the pixel is estimated. In this paper, it is assmed that the images have been pre-registered to the nearest pixel (sing any method described in [6] or [7]). The focs of this paper is the accracy with which the sb-pixel offset may be estimated, and in particlar analyses the systematic bias associated with offset estimation. For simplicity, the analysis here will be restricted to -D (althogh it easily generalises to two and higher dimensions). Let f(x) represent the pixel vales for the reference image. The target image g(x) is offset from f by sch that f( x+ ) = f = g( x). () The sb-pixel registration problem then is given f and g, to estimate, where < < (or -.5 < <.5 for even orders). The performance otside this range is also of interest becase of the ncertainty associated with the pixel-level registration. Most analyses of image processing operations, inclding registration, assme that the images are band-limited. In particlar, the Nyqist sampling criterion reqires that the highest sinsoid freqency component within an image is less than twice the sampling freqency to prevent aliasing. In practice, many images of real world objects contain some degree of aliasing by virte of the fact that objects have sharp edges or bondaries. Natral objects in particlar have detail at a wide range of scales so will contain energy over a broad bandwidth. The only bandwidth limiting elements within image captre systems are the optical transfer fnction of the lens, and area sampling performed by the sensor. Therefore some degree of aliasing is inevitable. In fact, applications sch as sperresoltion reqire that the inpt images be aliased in order to obtain more information from the image ensemble than is available within any single image [8]. Conseqently, any sbpixel image registration techniqe shold be tolerant of, or insensitive to aliasing. The rest of this paper is strctred as follows: Section II defines predictive interpolation and derives the optimal interpolator. This is then sed to estimate the sb-pixel offset between the target and reference images. The bias associated with the estimated offset is analysed in section III for sinsoid and step image models. Section IV compares the bias observed from real data with that from the theoretical analysis. The implications are discssed in Section V. II. PREDICTIVE INTERPOLATION While there are many different registration methods (see for example [,6,7,9]), this paper focses on a relatively new techniqe: predictive interpolation. This was first introdced in [] and analysed in more detail in [] and []. Conventional interpolation based methods for sb-pixel registration se an interpolator to create a continos srface /7/$5. 7 IEEE ICICS 7

2 from the reference image samples. The continos srface is then offset (and effectively resampled) and compared with the target image. The sb-pixel offset is determined by searching for the offset that gives the best fit with the target image. = arg min g f. () Interpolation can be considered as a linear filter. The resampled, offset reference image may be formed by sing a sampled interpolation fnction, reslting in a discrete filter: f = + h f + h f + hf+ h f + = hf i i where W is the region of spport for the interpolation kernel and the filter weights h i, depend on the desired offset,. Different interpolation kernels are derived based on making different assmptions abot the image. As a reslt, they may have different regions of spport, and will give different sets of weights derived from those assmptions. Solving () is non-trivial becase the difference between the target and reference image is a non-linear relationship of the offset,, and doesn t necessarily have well defined gradients. Conseqently, the offset is sally fond throgh iterative optimisation techniqes. Predictive interpolation avoids this search by trning the problem arond. It does not choose a particlar interpolation fnction, bt instead ses the image data itself to determine the interpolation kernel. It ses the pixel vales in the reference image to predict those in the target image, effectively determining the best (optimal in a least sqares sense []) interpolation kernel (h i, ) that relates the target to the reference: g i i (3) h f. (4) Since (4) is linear, the optimal filter coefficients may be easily determined by least sqares minimisation. As an interpolator, the prediction weights are sbject to the constraint hi =. (5) If this was not the case, then the intensity within niform regions wold change. Eliminating h from (4) gives: g f hi( fi f). (6), i Least sqares minimisation can then be sed over the whole image to give the vales of the coefficients that minimise the prediction error from f to g. If we define gˆ = g f (7) fˆi = f i f then () transforms to ˆ ˆ i = arg min i i, (8) hi,, i pixels h g h f which has well defined gradients, and being qadratic in h i, has a single global minimm that can be determined analytically. Taking the partial derivative with respect to each coefficient and solving for when these derivatives are eqal to gives a system of linear eqations. The order of the interpolator is given by the nmber of independent coefficients fitted (which is less than the size of W). For example, for a 3 rd order predictive interpolator: fˆ fˆ f ˆ fˆ f ˆ ˆ h f gˆ ˆˆ ˆ ff ˆˆ ˆˆ f ff h = fg ˆ ˆ ˆ ˆ ˆ f h ˆ f f f f fg ˆ or more generally (9) Fh = g. () This is then solved to give the optimal interpolation weights. The problem then is how to determine the offset from the interpolation coefficients. In previos work [], we have matched the weights to those that wold be obtained sing conventional low order polynomial interpolators. This was valid for a first order filter (linear interpolation in -D) where the eqations are the same. However, for higher orders the optimal interpolation fnction does not necessarily correspond with any standard interpolation method, so matching coefficients is harder to jstify. If we consider integer offsets, i.e. g =f i, then in the absence of noise the optimm interpolation procedre will derive the corresponding filter coefficient, h i, to be, and all of the remaining coefficients to be. In the general case, however, the target image is part way between pixels, so a mixtre of reference pixels is sed to predict the target. If we assme that the offset is a linear combination of the filter coefficients ˆ wh i i = () then from the sper-position principle, we get the weights as wi = i. () Note that this is exactly the same as the reslt obtained by matching the weights prodced for a polynomial interpolator (for example a spline with finite spport) and solving the reslting eqations for the offset,. It can be shown that () holds regardless of the particlar interpolation kernel sed. Since h makes no contribtion to the offset, it is convenient to eliminate h (rather than one of the other weights) in (6). Combining () and () we get û = wf g, (3) where w is a row vector made of the sample locations within the region of spport, eliminating the origin (i=). Note that the F - terms depend only on the reference image, so the matrix inverse only needs to be calclated once regardless of the nmber of target images that are registered. The rows of this inverse are then weighted (by w) giving a single row vector that depends only on the reference image. This leads to an efficient implementation when there are a nmber of target images that

3 need to be registered relative to one another (for example in sper-resoltion). III. BIAS ANALYSIS Any estimate of the offset between the two images will be sbject to ncertainties reslting from the characteristics of the images. There will also be ncertainties reslting from nderlying differences between the two images (noise for example). It has been demonstrated [] that when performing predictive interpolation with a sfficiently large image area the bias in the estimate dominates over the variance, making the bias of particlar interest. The bias in the estimator can be defined as any systematic deviation of the measred offset from the actal offset: Bias = E ˆ. (4) To obtain an analytic expression of the bias, it is necessary to have a mathematical model of the featres within the images. In this paper, we will consider two different types of featres: sinsoids directly measre the freqency dependence of the bias and allow the effects of aliasing to be considered explicitly; and step edges provide a more realistic model of typical images where a significant proportion of the high freqency content is contained within the edges between regions. Since the relationship between the samples and the featres of the image is nknown, the expected vale is calclated by assming that all possible relationships are eqally likely within the image. The smmation of (8) over all pixels within an image is therefore replaced by an integral over all possible relationships between the sampling grid and the featre positions: [ ] h arg min gˆ h fˆ. (5) = i i i hi,, i sampling offsets When performing least sqares minimisation, the smmations giving the elements in F and g in (9) and () are therefore replaced by integrals. A. Bias for sinsoids Most imaging algorithms implicitly assme that the inpt image is band-limited, with a sample freqency at least twice the maximm freqency present. Since sch signals can be reconstrcted perfectly, it is interesting to analyse any freqency dependence of the bias. Consider registering a pair of sinsoidal signals f( x) = sinωx (6) gx ( ) = sin ω( x+ ). Unless the sinsoid freqency, ω, is harmonically related to the sample freqency, the smmation will range over all possible angles. Therefore the expected vale of û may be obtained by averaging over all possible phase angles. This may be accomplished by integrating over a single period (for x from π/ω to π/ω). For a first order predictive interpolator, this gives [] cos ω( ) cosω+ cosω E [ ˆ] =, (7) ( cos ω) and for a second order interpolator: sinω E [ ˆ] =. (8) sinω For sinsoidal signals, the matrix F is singlar for 3 rd and higher orders. A sampled sine wave is actally only a second order fnction, in that sccessive samples may be generated recrsively sing a second order filter: hn [ ] = (cos ω) hn [ ] hn [ ]. (9) As a reslt, there are only two linearly independent eqations in (), and the optimal interpolation filter is ndefined for 3 rd or higher orders. Bias, E[]-, pixels Bias at ω=π/ st order nd order Offset,, pixels Figre. Bias in estimating the offset of sinsoids as a fnction of the offset. The bias (the difference between the expected offset and actal offset) is shown in Fig. as a fnction of offset at a sampling freqency of half the Nyqist rate. Several observations may be drawn from Fig. : In the absence of noise, there is no bias for offsets eqal to an integer nmber of pixels (within the region of spport of the interpolation kernel). This reslts from the se of the sper-position principle in (). The filter performs reasonably well at interpolating, bt extrapolates poorly, as indicated by the growing bias otside the region of spport. The bias from extrapolation is towards the region of spport. Otside the region of spport the offset is nderestimated, reslting in a positive bias for negative offsets and negative bias for positive offsets. The previos observation implies that there will be an odd nmber of offsets that give zero bias. It is this extra bias zero that gives the first order filter a significantly lower bias within the region of spport. For the first order interpolator, the bias is towards the nearest pixel, and away from the centre. Similarly, for the second order filter, the bias is towards the end samples and away from the central sample.

4 The bias pattern remains mch the same shape for different freqency signals (below the Nyqist freqency), althogh the amplitde of the error is larger at higher freqencies. As the offset is not known in advance (otherwise registration is nnecessary), the RMS error over all possible offsets in the range to provides an estimate of the average error. Fig. compares the freqency response of the bias for the two filters. Both filters perform well if the sinsoid freqency is mch lower than the sample freqency. The first order filter is effective past the Nyqist freqency and gives meaningfl estimates of the offset even for aliased signals. The performance of second order filter deteriorates mch earlier, with the estimated offset becoming meaningless even for naliased sinsoids close to the Nyqist freqency. The reason for this instability is that as the Nyqist freqency is approached, the sample at f(x+) becomes the same as f(x-), and the matrix F becomes ill-conditioned. RMS bias, pixels nd order st order Sinsoid freqency, relative to sample freqency Figre. Freqency dependence of bias for sinsoids. While the sinsoidal model yields sefl insights into the behavior of predictive interpolation, real images do not consist of a single pre sinsoid. B. Bias for step edges A more realistic model is that of a step edge since many images can be approximated by piecewise constant regions (with step edges in between). In forming the image, a step edge is blrred to a single pixel wide ramp by area sampling. This means that the edge pixels take on an intermediate vale depending on the exact position of the edge relative to the sampling grid. Withot loss of generality, consider a single step edge of height. Since the location of the edge is nknown relative to the samples, the expected vale for the bias may be determined by integrating (5) over all possible edge positions within the region of spport of the interpolator. This effectively considers, and averages over, all of the different configrations of the pixel vales as a reslt of an edge being somewhere within the region of spport. The advantage of explicitly enforcing the constraint of (5) is that there is no contribtion from edges otside the region of spport (becase of the sbtraction in (6)). After solving (3), the bias is a series of piecewise cbic crves (Fig. 3). For the first and second order filters, the general shape is the same as that obtained for a sinsoidal model. Again, the even orders have poorer performance than the odd orders. This is becase of the bias zero that odd order filters have at an offset of.5. Increasing the order by not only significantly improves the bias, bt it also extends the sefl region of relatively low bias by extending the region of spport. Otside the region of spport, the bias grows consistently in a similar manner regardless of the filter order. Bias, E[]-, pixels Bias of step edges st nd Offset,, pixels Figre 3. Bias characteristics of a blrred step edge for different order interpolation filters (first to fifth order). IV. VALIDATION WITH REAL IMAGES To assess the registration accracy, it is necessary to work with images where the offset is known in advance. Captring a seqence of images with precisely known offsets is difficlt, if not impossible. Therefore, a high-resoltion image was sed as the image sorce, and a simple imaging model was sed to simlate the captre of the sample images. A 7x7 sorce image was filtered sing a x horizontal box average filter to simlate D area integration, shifted by known offsets, then downsampled by a factor of horizontally to give low resoltion (7x85) images for registration. This provides low resoltion images offset in steps of.5 pixels. Taking pairs of images with a given offset, the bias may be determined by averaging the individal measred offsets. Fig. 5 shows the measred bias patterns obtained from the test image Beach (Fig. 4). The pattern if the bias is very similar to those obtained analytically from the step edge model. This implies that the piecewise constant model with area sampling provides a reasonable representation of the dominant image characteristics. The bias from the test image is slightly higher than that estimated from the model. 3rd 4th 5th

5 Bias, E[]-, pixels Figre 4. Sample low-resoltion image, Beach, dominated by low freqencies, with some sharp edges reslting in a limited degree of aliasing Bias of "Beach" st nd Offset,, pixels Figre 5. Bias characteristics measred from the Beach test image for different order interpolation filters (first to fifth order). V. DISCUSSION AND CONCLUSIONS From the sinsoidal test image, there is less bias at lower freqencies. Althogh the relationship between the offset and the image content is not linear, this does imply that low pass filtering the images prior to registration shold redce the bias. Preliminary reslts verify this, bt frther research is reqired to determine the best filter to se. Higher order filters not only significantly redce the bias, bt becase of their wider region of spport, also give good estimates of the offset even if the initial pre-registration is not accrate to the nearest pixel. This opens the possibility of extending predictive interpolation to performing the preregistration sing a pyramidal scheme performing an initial registration at a low resoltion, and sing predictive interpolation to refine the estimate at sccessive levels of the pyramid. This improved performance of higher order filters comes at the expense of an increased comptational cost. The most time consming step is performing the smmations to form F and g. The nmber of elements calclated grows with the sqare of 3rd 4th 5th the filter order. This is offset by the fact that only a single pass is reqired throgh each of the reference and target images. The reslts have been validated with a typical scene, giving reslts that closely match those obtained theoretically from a simple piecewise constant with area sampling image model. This implies that sch a model closely approximates the important image characteristics from the point of view of registration. The method needs to be tested with a wider range of images, particlarly to investigate the effects of significant aliasing on registration accracy. Preliminary experiments indicate that when significant aliasing is present, the bias increases and the bias pattern changes sbtly. The analysis here needs to be extended from D to D registration. While in principle the predictive interpolation filters are straight forward to implement, in general the optimal filters are not separable, making the analysis more complex. For example the eqivalent of a D 3 rd order filter has 5 degrees of freedom (independent filter coefficients) in D. Previos work toched on the effects of noise on first order registration []. This needs to be extended to higher order prediction filters. To conclde, higher-order predictive interpolation filters give significantly lower registration bias than the simple first order filter. Sb-pixel registration bias can be redced to a fraction of a percent of a pixel sing this method. REFERENCES [] D.G. Bailey and T.H. Lill, "Image Registration Methods for Resoltion Improvement", in Image and Vision Compting NZ, Christchrch, New Zealand, 9-96 (3-3 Agst, 999). [] G. Dane and T.Q. Ngyen, "The effect of global motion parameter accracies on the efficiency of video coding", in IEEE International Conference on Image Processing, 5: (4). [3] R. Sharma and M. Pavel, "Mltisensor image registration", in Society for Information Display, XXVIII: (997). [4] C.Y. Chen and R. Klette, "Image Stitching - Comparisons and New Techniqes", in 8th International Conference on Compter Analysis of Images and Patterns, Lectre Notes in Compter Science, 689: 65-6 (999). [5] J.L. Barron, D.J. Fleet, S.S. Beachemin, and T.A. Brkitt, "Performance of optical flow techniqes", in 99 IEEE Compter Society Conference on Compter Vision and Pattern Recognition (CVPR '9), 36-4 (99). [6] L.G. Brown, "A Srvey of Image Registration Techniqes", ACM Compting Srveys, 4:(4) (99). [7] B. Zitova and J. Flsser, "Image Registration Methods: a Srvey", Image and Vision Compting, : 977- (3). [8] D.G. Bailey, "Sper-Resoltion of Bar-Codes", Jornal of Electronic Imaging, :() 3- (). [9] Q. Tian and M.N. Hhns, "Algorithms for sb-pixel registration", Compter Vision, Graphics and Image Processing, 35:() -33 (986). [] D.G. Bailey, "Predictive Interpolation for Registration", in Image and Vision Compting Conference NZ, Hamilton, New Zealand, 4-45 (7-9 November, ). [] D.G. Bailey, A. Gilman, and R. Browne, "Bias Characteristics of Bilinear Interpolation Based Registration", in IEEE Region Conference (IEEE Tencon 5), Melborne, Astralia, (-4 November, 5). [] A. Gilman and D.G. Bailey, "Near optimal non-niform interpolation for image sper-resoltion from mltiple images", in Image and Vision Compting New Zealand (IVCNZ'6), Great Barrier Island, NZ, 3-36 (7-9 November, 6).