Spatial Transform Technique

Transcription

1 The overlooked advantage of the two-pass spatial transform technique is that complete, continuous, and time-constrained intensity interpolation can be conveniently achieved in one dimension. A Nonaliasing, Real-Time Spatial Transform Technique Karl M. Fant Honeywell, Inc. The advantage of a two-pass spatial transform technique over a one-pass technique has been that it offers a more effective and efficient match with current implementation technology.'2 But only part of this advantage has been recognized. Previous thinking has focused on the mathematical notion of point, or coordinate, transforms mapping points of the output image to points in the input image. Resampling and interpolating input intensities about the mapped point to generate the output pixel intensity is a secondary concern. Distinct advantages can be gained simply by reversing this order of significance, that is, by considering first the intensity interpolation technique and The development of the technique described in this article was fully funded by Honeywell, Inc., and is being patented by the company. January /86/-7$ 986 IEEE 7

2 second the way to impose spatial significance on the interpolation technique. The overlooked advantage of the two-pass technique is that complete, continuous, and time-constrained intensity interpolation can be conveniently achieved in one dimension. I will present the basic one-dimensional interpolation algorithm and then show how two-dimensional spatial significance can be imposed on the interpolation algorithm to achieve a two-dimensional transform mapping. The technique was developed intuitively and experimentally and will be presented in an intuitive style. The formal mathematical characterization has proved difficult. However, hardware has been built that performs two-dimensional image transforms in real time. One-dimensional resampling interpolation The one-dimensional resampling interpolation algorithm maps a limited line of discrete input pixel intensity values into a limited line of discrete output pixel intensity values. The mapping is determined by a sizing factor of the output line in relation to the input line and by a position factor in relation to the output line. It can be characterized by the following formulation, where S is a scaling factor, a size factor of the output data in relation to the input data, and /S is the inverse of the size factor. Note that S can vary pixel by pixel. Each consecutive output pixel = S (I/ S consecutive input pixels). The inverse size factor indicates how much of an input pixel contributes to each successive output pixel, or OUTVAL. The size factor can be a constant for an entire line of pixels, or it can change for each new output pixel. The interpolation algorithm is illustrated in Figure. INSFAC is /S, or the inverse size factor; it indicates how much of an input pixel contributes to each output pixel. Figure. Resampling Interpolation algorithm. 72 SIZFAC, the size factor, is a direct multiplicative scale factor from input to output. INSEG indicates how much of the current input pixel is available to contribute to the next output pixel. OUTSEG indicates how much of an input pixel is required to complete the next output pixel. Pixel is the intensity value of the current input pixel. INSEG and OUTSEG can be viewed as fractionalvalued pointers, or position markers, in the stream of contiguous input pixels. The process begins by comparing the values of INSEG and OUTSEG to determine which is smaller. If OUTSEG is smaller, there is sufficient current input pixel available to complete an output pixel. If INSEG is smaller, there is not sufficient current input pixel left to complete the output pixel; the current input pixel will be used up, and a new input pixel must be fetched. There are only these two conditions: Either the current input pixel will be used up without completing an output pixel, or an output pixel will be completed without using up the current input pixel. Equality of INSEG and OUTSEG can be assigned to either condition. If OUTSEG is smaller than INSEG, an output pixel will be completed. The current input pixel value is multiplied by OUTSEG and added to the accumulator. INSEG is decremented by the value of OUTSEG to indicate that the OUTSEG portion of the input pixel has been used; then OUTSEG is reinitialized to INSFAC for the next output pixel. The contents of the accumulator are scaled by SIZFAC, yielding the value of the current output pixel. The accumulator is zeroed, and the process returns to compare the new values of INSEG and OUTSEG. If INSEG is smaller than OUTSEG, an input pixel will be used up. The current input pixel value is multiplied by INSEG and added to the accumulator. OUTSEG is decremented by the value of INSEG to indicate that the INSEG portion of the output pixel has been satisfied; then INSEG is reinitialized to, and the next input pixel is fetched. The process then returns to compare the new values of INSEG and OUTSEG. If INSEG and OUTSEG are equal, the input pixel will be used up and an output pixel will be completed, but only one event at a time can occur. In this case either event can be chosen to occur first, and the other will occur next. If the choice is to complete the output pixel, the pixel intensity is scaled by OUTSEG and added to the accumulator. OUTSEG is subtracted from INSEG, which goes to zero, and OUTSEG is reinitialized to INSFAC. At the next cycle of comparison OUTSEG is a nonzero value and INSEG is zero. INSEG is smaller than OUTSEG, so this cycle uses up the current input pixel. Because INSEG is zero, the current pixel value is multiplied by zero and added to the accumulator. Then zero is subtracted from OUTSEG, effectively performing a null computation, or IEEE CG&A

3 I pixel is multiplied by OUTSEG and added to the accumulator. OUTSEG is subtracted from INSEG and is then reinitialized to.33. The contents of the accumulator are scaled by SIZFAC, producing the current output pixel value 5. The accumulator is zeroed, and the next output pixel is addressed. This process continues until all input pixels are used up or until all output pixels are completed. Figure 3 is an example of an expanding size factor; Figure 4 is an example with zero cycles. The transform is complete in that all the information from the input pixels contributed to the output pixels. The Figure 2. Example of interpolation. zero cycle. The next input pixel is fetched, and INSEG is reinitialized to. No matter which condition is chosen for equality, the interpolation algorithm proceeds correctly. Figure 2 is an example of the interpolation algorithm. A size factor of.75 will shrink the input line by 3/4. The inverse size factor.33 indicates that each output pixel should be composed of.33 input pixels. The top row represents the values of input pixels; the bottom row represents the calculated values of output pixels. The lines between the rows indicate the spatial mapping from input to output. Initial values are SIZFAC =.75 INSFAC =.33 INSEG = OUTSEG =.33 Accumulator = For the first cycle INSEG is smaller than OUTSEG. The input pixel intensity is multiplied by INSEG and added to the accumulator. INSEG is subtracted from OUTSEG, the next input pixel is addressed, and INSEG is reinitialized to. For the second cycle OUTSEG is smaller. The input pixel is multiplied by OUTSEG and added to the accumulator. OUTSEG is subtracted from INSEG and is then reinitialized to.33. The contents of the accumulator are scaled by SIZFAC, producing the current output pixel value 2. The accumulator is zeroed, and the next output pixel is addressed. For the third cycle INSEG is smaller. The input pixel intensity is multiplied by INSEG and added to the accumulator. INSEG is subtracted from OUTSEG, the next input pixel is addressed, and INSEG is reinitialized to. For the fourth cycle OUTSEG is smaller. The input January 986 Figure 3. Example of expansion. resampling of the line of input pixels is contiguous, without gaps, and without overlaps. The transform is continuous in that the interplay between INSEG and OUTSEG and the scaling of input pixels can be performed with arbitrary precision. Precision issues The precision sensitivity of the interpolation algorithm is in the spatial domain rather than in the intensity domain. The algorithm will always produce a smooth output line free of intensity aliasing artifacts, but there may be spatial position inaccuracies in the output line. These inaccuracies can result in spatial jitter between consecutive spatial transforms on the right edge of the output line (assuming 73

4 Sl. li/s= Output Cycle Cycle 2 Output Cycle 3 Cycle 4 Output Cycle 5 INSEG OUTSEG ACCUM OUTVAL Cycle 6 2 Output Cycle precision was reached. The same thing occurs during expansion, where SIZFAC is large. The intensity mapping is quite correct until INSFAC underflows its fraction bits, becomes zero, and the output line again disappears. Subpixel positioning Output lines can be subpixel positioned, or phased in the output line, by scaling the output pixel intensity value proportionally to the fractional part of the output pixel to be represented. A fractional value scale can be superimposed on the pixel number scale as shown in Figure 5. The fractional position value.5 is halfway through pixel. To position a line beginning at.5, the first output pixel should be half its normal intensity value. This can be accomplished simply by initializing OUTSEG to.5 of INSFAC instead of of INSFAC. This makes the first, Figure 4. Example with zero cycles. that the algorithm proceeds from left to right). This arises from the continued mutual subtraction of INSEG and OUTSEG. Notice in Figure 2 that the final value of INSEG is when it should properly be. This is due to truncation from carrying only two decimal digits of precision. The significance of this is that the output line will be slightly longer than it should be. After several hundred input pixels have been traversed, this error grows, and the last output pixel may be misplaced by one or more whole pixels beyond where it should properly be. Such a spatial misregistration is not important for many applications, but if it is important to the application, sufficient fractional precision must be carried with SIZFAC, INSFAC, INSEG, and OUTSEG to satisfy the application. For instance, to achieve I/256 of a pixel accuracy over 52 input pixels, about 24 fraction bits are required. The algorithm is very insensitive to pixel intensity mapping. The effects of precision on intensity mapping can only be seen at the extremes of shrinking and expanding, where factors overflow or underflow their precision range, suddenly go to zero, and the image disappears. If, for instance, INSFAC has eight integer bits, it can specify that 255 input pixels be added to the accumulator, which will then be scaled by a SIZFAC of /255 to shrink a line by /255. A correct average of 255 input pixel intensities is generated for the output pixel. If, however, a SIZFAC of /256 is attempted, INSFAC must represent 256 and set a ninth integer bit, which it doesn't have, and it is suddenly zero. OUTSEG is always smaller and equal to zero, so that all output pixels are zero, and the output line disappears. There was not a gradual degradation, but there was integrity of intensity mapping until the limit of Figure 5. Real-valued scale over pixel grid. or leftmost, pixel of the output line a partial-intensity pixel and shifts the mapping of the entire line by half an output pixel. When the algorithm runs out of input pixels, the value left in the accumulator is very likely not a full output pixel's worth. This value is scaled by SIZFAC and output as the last output pixel, which will also be a partial-intensity pixel. So partial-intensity output pixels are generated at the beginning and end of the output line to represent continuous subpixel positioning of the output line. The example in Figure 6 positions an output line with a size factor of.75 to begin at output location 2.4. The first output pixel location is set to 2, and the initial value of OUTSEG is (-.4) * INSFAC to indicate.6 of full intensity for the first output pixel. Notice the partial intensity for the last output pixel. Output clipping Clipping is the process of ignoring output pixels that don't fall within the output line. These pixels can be ignored prior to being calculated. By projecting the left edge of the output line into the input pixel line, it can be determined where in the input pixel line the first output pixel should begin. This point is IEEE CG&A I

5 S =.75 I/S=.33 Output location = S =.75 /S =.33 Output location = INSEG OUTSEG ACCUM OUTVAL Output Cycle Cycle Cycle Output Cycle Cycle Output Cycle Cycle Output Cycle Cycle Last input cycle No more input pixels 9 Figure 6. Example of subpixel phasing. INSEG Cycle.94 Output Cycle 2.6 Cycle 3.6 Output Cycle 4.28 Cycle 5.28 Cycle 6 Output Cycle 7.95 Figure 7. Example of clipping. OUTSEG No more output pixels S= 3. /S=.33 ACCUM OUTVAL unlikely to be on an input pixel boundary, which means that interpolation should begin with a partial input pixel. This can be simply accommodated by initializing the first value of INSEG to a value less than. In this manner all beginning border pixels in the output line can be properly mapped. Figure 7 illustrates clipping where the output line is to begin at -2.3 with a size factor of.75. If the beginning of the first input pixel maps to -2.3 in the output line, then we have to move 2.3 output pixels' worth of input pixels into the input line. This input offset (INOFF) is INOFF = 2.3 *.33 = 3.6 To begin at input pixel location 3.6, just set the input pixel to 3 and INSEG to INSEG = -6 =.94 End-of-line clipping is automatically accommodated by simply terminating when the last whole output pixel is generated. The last output pixel at the end of the line will automatically be a properly mapped output pixel. Expansion smoothing Expansion with the algorithm as described results in a blocky output image. As illustrated in Figure 8, several output pixels of identical value are generated from a single January 986 INSEG OUTSEG ACCUM OUTVAL Output Cycle Output Cycle Output Cycle Cycle Output Cycle Output Cycle Output Cycle Figure 8. Example of expansion blockiness. input pixel. These areas of identical value lend a blockiness to the output line. This can be accommodated with a simple expedient. Instead of inputting raw pixel values to the interpolation algorithm, we can take pixel values from a one-pixel-wide averaging window traversing the raw 75

6 input pixels based on the value of INSEG. INSEG can be viewed as a pointer to the spatial position of utilization of input pixels. The averaging window can be viewed as delivering one pixel's worth of input value beginning at INSEG. This relationship is input pixel = INSEG * current pixel + (- INSEG) * next pixel The operation of the averaging window and its effect on the output is illustrated in Figure 9. During shrinking the averaging window is ineffective because INSEG is most of the time, effectively disabling the averaging. Therefore the window can be operated continuously for all transforms, whether they are expanding or shrinking Output Cycle Cycle 2 Output Cycle 3 Cycle 4 99 Raw 2 Averaged Output Cycle I Output Cycle Output Cycle 3.34 o. Cycle 4 Output Cycle Output Cycle Output Cycle Ill Figure 9. Example of expansion with averaging window. Nonconstant-size-factor mapping So far only situations that perform an entire mapping with a constant size factor have been discussed. If the size factor, and hence INS FAC, are varied for each new output pixel, any arbitrary mapping of contiguous pixel values can be achieved. Figure is a one-dimensional example of a mapping with a size factor that halves for each output pixel. Limited-time performance For contiguous mappings of consecutive input pixels into consecutive output pixels, a maximum time limit can be established. The nature of the interpolation algorithm is 76 Cycle 7 Cycle 8 2 I INSEG OUTSEG ACCUM SIZFAC OUTVAL Figure. Example with varying size factor. INSEG OUTSEG ACCUM OUTVAL.33 4 Cycle 5 Output Cycle 6 /S=.33 S= 3. /S=.5,, 2., 4. S=2.,,.5 such that every cycle is either completing an output pixel or using up an input pixel. This includes zero cycles (see Figure 4). If there are 52 input pixels and a range of 52 possible output pixels, and if pretransform clipping is performed to ensure that only output pixels that fall within the output line are generated by the algorithm, then the algorithm will not use up more than 52 input pixels nor generate more than 52 output pixels. Any arbitrary contiguous mapping can be guaranteed to occur in 24 cycles. A worst case, interestingly enough, is a size factor of where, for instance, 52 output pixels are generated directly from the corresponding input pixel and 52 zero cycles are generated to get the next input pixel. A size factor of slightly less than uses 24 cycles with no zero cycles. This ability to deterministically bound the computation, plus the simplicity of the computation, makes the algorithm ideal for real-time implementation. A two-dimensional transform Two-dimensional images can be spatially transformed by combining a series of one-dimensional interpolations in two passes over the image. The first pass, which we will consider to be the vertical pass, maps all input pixels into their correct vertical orientation and creates an intermediate image. The second, or horizontal, pass maps all the pixels from the intermediate image into their correct horizontal orientation to create the final output image. A technique of mapping to four target corner points to achieve size, translation, and rotation of an image can be used. IEEE CG&A

7 Four-corner-to-four-corner mapping procedure This procedure operates on an input image residing in a bounded image plane and maps the input image into a bounded output image plane. Line and column extents XBEG, XEND, YBEG, and YEND specify the bounds and the four corner points of the input image to be mapped. Four target corner points specify the corner points of the mapping in the output image plane. These target corner points can be computed by applying standard coordinate transforms to the input corner points. Both input and output images use a fractional-valued coordinate system overlaying the discrete grid of pixels, as shown in Figure. All computations are in relation to this reference system. The origin is at the upper-left corner; X values increase to the right and Y values increase downward. Figure 2 illustrates the four-corner transform. The input image corner points are labeled, 2, 3, 4, and the target corner points corresponding to the input points are similarly labeled. The figure shows the mapping from the input image to the intermediate image and the intermediate image to the output image. The first, or vertical, pass maps all intensities into their correct vertical orientation column by column. Corner I must be mapped to Y and corner 4 must be mapped to Y4; therefore, the correct vertical orientation for the first, or leftmost, column of the input image is between Y and Y4. The correct vertical orientation for the last, or rightmost, column is between Y2 and Y3. The correct vertical orientation for all intervening columns is along uniformly interpolated positions between the first and last column. The real output location (ROTLOC) and size factor (SIZFAC) for the first column are ROTLOC = Y, SIZFAC = (Y4-Y)/( YEND-YBEG + ) The size factor for a rectangular output transform is a constant for all columns, but the output location varies. This variation is linear and can be accommodated by adding an increment (LOCDLT) to ROTLOC for each successive column. This increment is LOCDLT = (Y2-Yj)/(XEND-XBEG) x o,, 2, 3, 4,,, 2, 3, 4, 2. 3.,2,2 2,2 3,2 4,2,3, 3 2, 3 3, 3 4, 3 4. Figure. Real-valued coordinate system over image. January 986 Figure 2. Four-corner-to-4our-corner mapping technique. 77

8 The value of ROTLOC will be incremented by LOCDLT for each successive column one less time than the number of columns, and the output location (ROTLOC) for the last column will equal Y2. For nonrectangular output mappings, an increment can be similarly applied to the size factor. The values to initialize the interpolation algorithm for each column are SIZFAC = SIZFAC INSFAC = I/ SIZFAC INSEG = OUTSEG = INSFAC * ( - fraction (ROTLOC)) OUTLOC = Integer (ROTLOC) INLOC = YBEG If ROTLOC is less than, clipping must be performed.by projecting the left edge of the zero output pixel into the input line. The input line offset (INOFF) is intermediate image maps to the 2-3 edge of the output image. Beginning at the top, the output location for the first row is the intersection of the -4 edge with Y = Y2. ROTLOC= SIZFAC = SIZFAC INSFAC = /SIZFAC INSEG = - fraction (INOFF) OUTSEG = INSFAC OUTLOC = INLOC = YBEG + Integer (INOFF) A true-perspective mapping and various arbitrary mappings have also been developed within the technique. After the interpolation process has operated on each column of the input image, an intermediate image is produced, as shown in Figure 2. The next step is to transform the intermediate image row by row into the output image. The extents of the intermediate image are from XBEG to XEND and from Integer (Y2) to Integer (Y4). This bounding rectangle is the input to the second pass. The triangular areas in the rectangle outside the intermediate image must be set to zero. The second pass basically maps the 4- edge of the intermediate image into the 4- edge projected through Y2 of the output image with a size factor such that the 2-3 edge of the The intermediate image line must be stretched a little to fit between ROTLOC and X2. The size factor is X2 - ROTLOC SIZFAC = XEND - XBEG + I The last ROTLOC must equal X4 after Integer (Y4) Integer (Y2) increments, so the location delta is LOCDLT=- INOFF = -ROTLOC * INSFAC The values to initialize the interpolation algorithm for clipped columns are Y4) X4)_ ((Y2 ±(X ( Y - Y4) X4 - ROTLOC Integer ( Y4) Integer ( Y2) - The values to initialize the interpolation process for each row are SIZFAC = SIZFAC INSFAC = I/ SIZFAC INSEG = OUTSEG = INSFAC * ( - fraction (ROTLOC)) OUTLOC = Integer (ROTLOC) INLOC = YBEG These are identical to the values for the first pass. Clipping is also performed exactly as in the first pass. Mapping of the first row, which consists mostly of zeros, begins at the initial ROTLOC with the proper size factor such that the data at corner 2 of the intermediate image maps into corner 2 of the output image. As for each successive row, ROTLOC moves along the 4- edge of the output image the 2-3 edge of the intermediate image is mapped to the 2-3 edge of the output image. The internal data of the image follows with these edge mappings through both passes and is itself correctly and proportionally mapped. Summary The four target corner points are determined by applying the traditional coordinate transform equations. The remainder of the transform is accomplished by providing IEEE CG&A

9 output location and size factor values to the interpolation algorithm for each column of the first pass and each row of the second pass. Because the interpolation from the input image is complete, contiguous, and continuous, there are no resampling artifacts in the interior of the output image. Because each row and each column is mapped with subpixel position phasing, there is no edge aliasing of the output image. The output image is generated directly in the form that antialiasing attempts to achieve under conventional approaches to spatial image transform. Furthermore, the transformation is achieved with relatively simple computations. Figures 3, 4, and 5 illustrate a 3-degree rotation. Figure 6 is a 2-degree rotation and shrink by /2. Figure 7 shows the upper-left corner of the 2-degree rotation magnified by pixel replication to show the partial-intensity edge pixels which avoid edge aliasing. A true-perspective mapping and various arbitrary mappings have also been developed within the technique. Figures 8 and 9 show the intermediate and final images of a true-perspective mapping. Figure 2 shows a mapping of the image driven by a sine wave. Figure 3. Original image. (The images in Figures 3 Figure 5. Thirty-degree rotation-final image. through 2 are reproduced with the permission of Jean Grapp.) Figure 4. Thirty-degree rotation-intermediate image. Figure 6. Two-degree rotation and shrink by /2. January

10 Figure 7. Partial-intensity edges of the 2-degree rotation. Figure 9. True-perspective mapping-final image. Figure 8. True-perspective mapping-intermediate image. Figure 2. Sine-wave mapping in both directions. Conclusion 2. M. Shantz, "Two Pass Warp Algorithm for Hardware Implementation," Proc. SPIE, Vol. 36, "Processing and Display of Three Dimensional Data," 982, pp.'6-64. The one-dimensional resampling interpolation technique is computationally simple, deterministically bounded, and provides a high-fidelity mapping from discrete input pixels to discrete output pixels. It can be combined in two passes over a two-dimensional image to effect two-dimensional spatial transforms. Real-time systems using the technique can be implemented more economically and perform higher fidelity transforms than systems using the standard technique of mapping points first and then interpolating intensities. M References. E. Catmul and A.R. Smith, "3D Transformations of Images in Scanline Order," Computer Graphics (Proc. SIGGRAPH 8), Vol. 4, No. 3, July 98, pp Karl Fant is a principal research scientist at Honeywell Systems and Research Center. He has a wide range of experience in algorithm design and system design for real-time image processing systems. His experience includes both image analysis and image generation systems. Fant received a BS in computer science from the University of Minnesota in 974. He is a member of ACM and IEEE. The author's address is Honeywell, Inc., Systems and Research Center, MN7-236, 26 Ridgway Parkway, Minneapolis, MN IEEE CG&A