PYRAMID FILTERS BASED ON BILINEAR INTERPOLATION

Transcription

1 PYRAMID FILTERS BASED ON BILINEAR INTERPOLATION Martin Kraus Computer Grapics and Visualization Group, Tecnisce Universität Müncen, Germany Magnus Strengert Visualization and Interactive Systems Group, Universität Stuttgart, Germany Keywords: Abstract: signal processing, image processing, multi resolution, pyramid algoritm, grapics ardware. Te implementation of several pyramid metods on programmable grapics processing units (GPUs) in recent years led to additional researc interest in pyramid algoritms for real-time computer grapics. Of particular interest are efficient analysis and syntesis filters based on ardware-supported bilinear texture interpolation because tey can be used as building blocks for many GPU-based pyramid metods. In tis work, several new and extremely efficient GPU-implementations of pyramid filters are presented for te first time. Te discussion employs a new notation, wic was developed for te consistent and precise specification of tese filters and also allowed us to systematically explore appropriate filter designs. Te presented filters cover analysis and syntesis filters, (quasi-)interpolation and approximation, as well as discontinuous, continuous, and smoot filters. Tus, a tool of filters and teir efficient implementations for a great variety of GPU-based pyramid metods is presented. 1 INTRODUCTION Many tecniques in real-time image processing employ te pyramid algoritm by Burt (Burt, 1981), and GPU-based image processing is no exception to tis rule (Williams, 1983; Krüger and Westermann, 2003; Lefebvre et al., 2005; Strengert et al., 2006). Pyramid metods are of particular interest because tey usually feature a linear time complexity and require only a limited number of switces of te render target. Altoug modern GPUs offer an enormous rasterization performance, te actual rasterization budget for eac pixel often consists of te equivalent to only a few dozens of texture reads. Tis is already a serious limitation for many image processing tecniques, wic often employ large two-dimensional convolution filters. Tus, even GPU-based implementations of complex image processing tecniques are often restricted to small image sizes and/or low frame rates. Terefore, several very efficient tecniques ave been developed to implement specific (often small) convolution filters on GPUs (Sigg and Hadwiger, 2005; Green, 2005). Unfortunately, many of tese tecniques are restricted to very specific filters and particular applications; tus, te efficient implementation of many oter filters is still a callenging task in GPUbased image processing. In tis work, we generalize a recently publised tecnique (Strengert et al., 2006) for a 2 2 analysis filter and a 2 2 B-spline syntesis filter. By formalizing te underlying concept wit te elp of a new notation, we are able to systematically explore appropriate filter designs and present several filters tat can be more efficiently implemented on GPUs using bilinear texture interpolation tan previous publications suggested. In order to make tese filters and teir implementations more easily accessible to readers wo are looking for particular filters, we ave also included some previously publised filter implementations. Moreover, tese filters sould elp to illustrate our new notation and te systematic filter construction suggested in tis work. We classify filters as basic filters presented in Section 3; i.e., convolution filters witout reduce or expand operation; analysis filters presented in Section 4; i.e., convolution filters combined wit a reduce operation; and syntesis filters presented in Section 5, wic are combined wit an

2 image g top of pyramid: g 2 filter 2 2 g 1 analysis g 0 g 0 syntesis g g 2 2 g (a) [ ] 1,1 1,2 2,1 2,2 g 1,1 g 1,2 0 g 2,1 g 2, ,1 1,2 0 2,1 2,2 (b) g 1,1 g 1,2 0 g 2,1 g 2,2 0 1,1 g 1,1 1,2 g 1,1 + 1,1 g 1,2 1,2 g 1,2 2,1 g 1,1 + 1,1 g 2,1 2,2 g 1,1 + 2,1 g 1,2 + 1,2 g 2,1 + 1,1 g 2,2 2,2 g 1,2 + 1,2 g 2,2 2,1 g 2,1 2,2 g 2,1 + 2,1 g 2,2 2,2 g 2,2 (c) Figure 1: (a) An image pyramid consisting of 3 levels: g (0), g (1), and g (2). (b) Illustration of te convolution of a 3 3 image g wit a 2 2 filter 2 2. (c) Te formal equation of te convolution illustrated in (b) including te equivalent 3 3 filter padded wit zeros. Note te mirrored indices of 2 2 and te sift of non-zero components in te resulting matrix. expand operation. Analysis and syntesis filters are furter classified according to teir featured smootness (in te limit of infinitely many reduce or expand operations). Section 6 concludes tis work wit our plans for future work. 2 PYRAMID METHODS ON GPUS Pyramid metods usually convolve image data of one pyramid level wit a small analysis filter and reduce te resulting image data to compute coarser pyramid levels. Tis process is called analysis wile te opposite process, called syntesis, expands image data and convolves it wit small syntesis filters to compute finer pyramid levels; see Figure 1a. Te reduce and expand operation are also called downsampling and upsampling, respectively. Pyramid metods more specifically spoken, pyramid images ave been most successful in computer grapics in te form of mipmap textures as proposed by Williams (Williams, 1983). More recently, te possibility to read pixel data of a rasterized image by texture interpolation witout crucial overead led to new real-time image processing tecniques on GPUs (Green, 2005). For pyramid metods in GPU-based image processing, we suggested to employ ardware-supported bilinear texture interpolation for te reduce operation combined wit a 2 2 convolution filter (Strengert et al., 2006). However, te only analysis filter presented in tat work is a simple 2 2 filter. For te syntesis, we proposed to employ bilinear texture interpolation for te combination of te expand operation and a 2 2 syntesis filter corresponding to te biquadratic B-spline subdivision sceme (Catmull and Clark, 1978), wic is better known as te Doo-Sabin subdivision sceme for regular quadrilaterals. 3 BASIC FILTERS Te filters discussed in tis section are simple convolution filters witout reduce or expand operation. Tus, strictly speaking, tey are not pyramid filters. However, tey act as building blocks for te more complex filters discussed in Sections 4 and 5. A discrete convolution of a filter mask and an image g resulting in an image f is denoted by f g. If te filter mask is represented by an n i n j matrix and te image g is of dimensions n r n c, te matrix component f r,c for te row index r and te column

3 index c is ined by: f r,c n i n j i1 j1 i, j g r i ni /2,c j n j /2. Standard matrix notation is employed; i.e., row and column indices are given in tis order and indices start wit 1. Moreover, matrix products of column vectors times row vectors (i.e., outer products of vectors) are employed for separable filters. Components of te image g wit indices outside te ranges from 1 to n r and 1 to n c, respectively, are eiter set to 0 if te image g represents a filter mask, or determined by clamping te indices to te valid ranges. Te dimensions of f depend on te particular application; in tis work we usually determine te dimensions of f by g s dimensions; exceptions are mentioned explicitly. For even filter dimensions n i and n j, it is often useful to tink of te pixel positions of image f being sifted by alf a pixel along te diagonal relatively to image g. In order to make tis sift explicit, one can use zero padding of te convolution mask; e.g., for a 2 2 convolution mask 2 2 tere is an equivalent 3 3 convolution mask wit zeros in te first row and column: 2 2 [ ] 1,1 1,2 2,1 2,2 0 1,1 1,2 0 2,1 2,2 For an illustration of tese initions and te described sift of components of f, see Figures 1b and 1c. As mentioned, g and terefore f may also represent filter masks. In tis case, g and denote two convolution masks (applied from rigt to left), wic can be combined in one convolution mask f. In fact, te main motivation for our formalism is to decompose complex filter masks (e.g., f ) into multiple smaller filters (e.g., and g, but usually more tan two), wic are small enoug (i.e., 2 2) to be implemented by bilinear texture interpolations. In tis way, complex filters can be implemented by a sequence of bilinear texture interpolations Box Filter Te most important building block for te filters discussed in tis work is te 2 2 filter, also known as uniform, average, or mean filter: 2 2 [ ]. Since tis filter multiplies four neigboring pixels wit equal weigts, it is easily implementable wit. one ardware-accelerated bilinear texture image interpolation. Te sampling position for tis texture interpolation (usually specified by texture coordinates) is determined by te position of te sared corner of te four pixels (in te little squares model of pixels) or te barycenter of te four pixels (if te pixels temselves represent sampling points in a uniform grid). Due to our inition of te convolution, te sampling point for te resulting matrix component wit indices r and c is located at te upper, left corner of te pixel specified by r and c in te original source matrix. (We assume a coordinate system tat corresponds to traditional matrix notation wit te (positive) r axis pointing downwards and te (positive) c axis pointing to te rigt.) Tis sift by alf a diagonal of one pixel is more explicit in te equivalent zero-padded 3 3 filter mask, wic will be called : 2 2 Te arrow in te symbol indicates te position of non-zero elements in te 3 3 matrix as well as te sift of non-zero elements in te resulting matrix illustrated in Figure 1c. Note, owever, tat te sampling position for te bilinear texture interpolation is sifted in te opposite direction relatively to te original pixel position. Obviously, tere are furter (non-equivalent) zeropadded 3 3 filters, wic sift pixel positions in oter directions; e.g., te opposite direction for te filter : Tis filter can also be implemented by a bilinear texture interpolation if te sampling point is set to te opposite (lower, rigt) pixel corner. Tese two filters are te only building blocks for all filters presented in tis section and Section 4; i.e., all tese filters can be implemented by a decomposition into a sequence of convolutions wit and, and te application of te corresponding sequence of bilinear texture interpolations. Te most basic example is te 3 3 Bartlett filter discussed next Bartlett Filter Bartlett filters are also called triangular or (in particular in one dimension) triangle filters. Tey are separable filters; terefore, tey may be decomposed into.

4 matrix products of column times row vectors. In tis work, te 3 3 Bartlett filter is of particular interest: 3 3 Bartlett [ ]. Te separation into two one-dimensional filters can also be expressed by a sequence of two convolutions wit tese filters (wit appropriate filter dimensions and index mirroring). However, te representation as a convolution of filters leads to a more efficient implementation: 3 3 Bartlett I.e., a sequence of two convolutions wit and is equivalent to a single convolution wit te 3 3 Bartlett filter Bartlett 3 3. Terefore, te 3 3 Bartlett filter may be implemented by a sequence of two bilinear texture interpolations corresponding to te two 2 2 filters. Note tat te second texture interpolation as to access te result of te first convolution; tus, a ardware-accelerated implementation will usually ave to switc te render target in order to access te previously rasterized image. Note also tat it is crucial to alternate between and ; oterwise, te discussed sifts would not cancel and te resulting image would be sifted by one full pixel position. Since te 3 3 Bartlett filter Bartlett 3 3 is particularly useful, we will also use it for building up more complex filters. However, it is always understood tat a convolution wit Bartlett 3 3 is equivalent to a sequence of one convolution wit and one convolution wit. 3.3 Gaussian Filters Repeated convolutions of Bartlett filters are equivalent to approximations of Gaussian filters if all nonzero matrix components of te resulting filters are considered; for example: 5 5 Gauss 7 7 Gauss Bartlett 3 3 Bartlett, Bartlett 3 3 Bartlett 3 3 Bartlett. Te basic reason for te approximation of Gaussian filters is te central limit teorem; in fact, any reasonable, positive filter will converge to a Gaussian filter in te limit of infinitely many convolutions. In te case of te filter and terefore also for te Bartlett filter te continuous convolutions are actually iger-order B-splines. Since eac convolution wit a Bartlett filter can be implemented by two bilinear texture interpolations, n 1 texture interpolations are necessary to implement an approximation to te convolution wit an n n Gaussian filter. For comparison, a separable n n filter wit fixed filter weigts for te two onedimensional filters would require 2n nearest-neigbor texture reads. 4 ANALYSIS FILTERS Analysis filters are convolution filters tat are combined wit a reduce operation, wic reduces te number of pixels by a factor of 2 in eac dimension. Terefore, tis operation is also called downsampling. In our notation it is indicated by a downward pointing arrow: f g, wic ines te components of f as: f r,c n i n j i1 j1 i, j g 2r i ni /2,2c j n j /2. Te issue of sifts by alf a pixel diagonal is different from te problem discussed in Section 3. Since te number of components is divided by 2, it is preferable to work wit even dimensions of images and filter masks. Moreover, it is often preferable to use sym-

5 metric filter mask (in te sense of i ni i+1) in order to avoid asymmetric weigting of even and odd pixels. For convenience we introduce a particular notation for te combination of an analysis filter wit a reduction by a factor of 2 m in eac dimension: f m g, wic results in tese components of f : f r,c n i n j i1 j1 i, j g 2 m r i n i /2,2 m c j n j /2. Our notation combines a reduce operation wit a convolution in a single operation. Tus, if te convolution can be implemented wit a bilinear texture interpolation, te combination wit te reduce operation is also implementable wit a bilinear texture interpolation. To tis end, it is only necessary to use a sparser grid of texture sampling points. 4.1 Discontinuous Filter In order to illustrate our notation for analysis filters, it is first applied to te 2 2 filter, wic is te standard analysis filter for mipmap generation: 2 2 g g. In te case of analysis filters, te use of 2 2 migt be preferable, altoug it is equivalent to. One bilinear texture interpolation is sufficient to implement tis analysis filter in pyramid metods (Strengert et al., 2006). Te 2 2 filter is classified as discontinuous analysis filter because te equivalent filter for a reduction of te dimensions by a factor of 2 m is always a discontinuous filter even in te limit of m : Consider te squared filter, wic is ined by a sequence of two reduce operations and convolutions: ( 2 2 g ). ( ) g 2 2 Tus, te actual (separable) filter is: ( ) [ ] [ ] For a general factor 2 m, te equivalent filter ( ) 2 2 m 1 [ ] 1 [ ] m }{{}} {{} 2 m } {{}} {{} 2 m 2 m (2 m 1) (2 m 1) is still a discontinuous filter. Te onedimensional filter, wic tis separable filter is constructed from, is illustrated in Figure 2a for various values of m. Te same tecnique is employed to discuss and classify te smootness of all analysis filters presented in tis section (a) m 1 m 2 m 3 m (b) m 1 m 2 m 3 Figure 2: (a) Powers of te 1D filter mask corresponding to 2 2. (b) Same as (a) for C 0 -Continuous Filter Wile te 2 2 filter is a discontinuous analysis filter, te 4 4 filter results in a C 0 -continuous analysis filter in te limit of infinitely many analysis steps. It is ined as: g ( g )., As indicated by our notation, tis filter can be constructed by a reduce operation tat includes a convolution wit and a simple convolution wit. Terefore, te implementation requires only two bilinear texture interpolations. However, tere exists one additional implementation difficulty: te first reduce operation will in general compute non-zero components for te 0-t row and column. Tese intermediate results ave to be stored and used in te second convolution, oterwise te components of te first row and column of te total result will be corrupted. For te discussion of te smootness of tis analysis filter, we consider te equivalent (separable) squared filter first: ( ) [ ] [ ] Te first tree powers are illustrated in Figure 2b. In contrast to te 2 2 filter, te 4 4 filter as overlapping domains and terefore becomes C 0 - continuous in te limit m ; in fact, te 1D filter becomes piecewise-linear wit a linear ascending part, a constant part, and a linear descending part.

6 (a) (b) Figure 3: Pyramid image blurring using (a) te 2 2 analysis filter and (b) te 4 4 analysis filter. (Te employed syntesis filter is discussed in Section 5.3.) Our notation also suggests te construction of an alternative combination of two filters wit one reduce operation: 3 3 Bartlett g ( g ). In te limit of m tis filter is also C 0 -continuous (in fact, it is a triangle function); owever, it includes an additional undesirable sift and wat is worse te analysis of even and odd pixels becomes asymmetric, wic is likely to result in flickering artifacts in animations. Terefore, te 4 4 filter appears to be te preferable C 0 -continuous analysis filter; e.g., for pyramidal image blurring (Strengert et al., 2006) as illustrated in Figure C 1 -Continuous Filter By combining one reduce operation and tree convolutions wit filters a C 1 -continuous filter can be constructed, wic we call quadratic 4 4 since it corresponds to a biquadratic B-spline in te limit of infinitely many analysis steps. Te filter is ined as quadratic , quadratic 4 4 g ( 3 3 Bartlett g). Te decomposition sows tat tree bilinear texture interpolations are sufficient for an implementation of tis analysis filter. Te squared filter already indicates a sape similar to te quadratic B-spline: ( ) 2 quadratic [ ] [ ] In te limit of infinitely ig powers it actually converges to te C 1 -continuous biquadratic B-spline function. Anoter combination of tree filters and one reduce operation is: 3 3 Bartlett ( g ) ; owever, tis analysis filter is only C 0 -continuous according to our classification and requires one more texture interpolation tan 4 4. Yet anoter combination is: ( Bartlett 3 3 g ), wic results in a C 1 -continuous analysis filter but includes an undesirable sift and an asymmetric weigting of pixels. Tus, quadratic 4 4 is usually te preferable C 1 -continuous analysis filter. Te construction of iger-order B-spline analysis filters consists of additional convolutions wit 2 2 filters before te reduce operation is applied, e.g.: ( ( cubic 5 5 g Bartlett 3 3 )). g 5 SYNTHESIS FILTERS Analogously to analysis filters, syntesis filters are convolution filters tat are combined wit an expand operation, wic increases te number of pixels by a factor of 2 in eac dimension. Tis operation is also called upsampling and is indicated by a upwards pointing arrow in our notation: f g, were te components of f are: f r,c n i i1 n j j1 r mod 2,c mod 2 i, j g (r+1)/2 i ni /2, (c+1)/2 j n j /2. Te syntesis filters presented in tis section are strongly related to popular subdivision scemes, wic are very well known in computer grapics; tus, we will considerably sorten te discussion by refering te reader to te corresponding concepts for subdivision scemes as discussed, for example, by Zorin et al. (Zorin et al., 2000). Similarly to te case of analysis filters, te combination of an expand operation and convolution filters in our notation is closer to an efficient implementation using bilinear texture interpolation tan traditional notations. One crucial difference to analysis filters is te coice of sampling positions in te corresponding bilinear texture interpolation. For most interpolating syntesis filters, te sampling points are

7 te centers of pixels, teir corners, and te s of teir edges. Tis corresponds to face-split subdivision scemes, wic are also known as primal scemes. Te most important alternative sampling positions are te positions of te Doo-Sabin subdivision sceme for regular quadrilaterals (Strengert et al., 2006). Tis alternative corresponds to vertex-split subdivision scemes (also known as dual scemes) and offers te advantage of symmetric computations for all sampling positions wile face-split scemes distinguis between old and new positions. In bot cases, tere are four different kinds of sampling positions, wic correspond to four convolution filters 1,1, 1,0, 0,1, and 0,0 wit te superscripts determined by te new row index modulo 2 and te new column index modulo 2. Tis notation is illustrated wit te elp of te well-known syntesis filters for nearest-neigbor interpolation, bilinear interpolation, and te biquadratic B-spline filter. On te oter and, te suggested implementation of te bicubic B-spline syntesis filter and te construction of iger-order syntesis filters is a new result. 5.1 Discontinuous Filter A discontinuous syntesis filter is already provided by nearest-neigbor texture interpolation; tus, it is not of particular interest for our work. However, we ave included it ere for completeness and to demonstrate our notation for tis very basic syntesis filter: nearest 1,1 nearest 1,0 nearest 0,1 nearest 0,0 5.2 C 0 -Continuous Filter Since bilinear texture interpolation already provides a C 0 -continuous filter even witout te overead of a pyramid metod, te equivalent syntesis filter is not very useful in itself; owever, it may be used as a building block for more complex syntesis filters. It is called because of te corresponding subdivision sceme; its inition is: 1,1 1,0 0, (, ) 1,0,,. 0,0 nearest. 0,0 Note tat 0,0 corresponds to te convolution filter for sampling positions at pixel centers wile 1,1 corresponds to te convolution filter for sampling positions at teir corners, and te remaining two filters correspond to te centers of edges of pixels. Also note tat te first row and te first column of te new image data will be sampled at te upper, left corners of old pixels and te s of edges between tese positions. Terefore, te presented convolution filters will access old pixels of te 0-t row and 0-t column. In practice, tese lookups could return te corresponding pixel data of te first row and first column, respectively. An alternative is to discard te first row and first column of te resulting image. 5.3 C 1 -Continuous Filter Te syntesis filter discussed ere corresponds to te biquadratic B-spline subdivision sceme (Catmull and Clark, 1978), also known as te Doo-Sabin subdivision sceme for regular quadrilaterals or te twodimensional generalization of te Caikin sceme. It is an approximating vertex-split subdivision sceme; tus, all pixels are processed in symmetric ways: 1,1 Doo-Sabin 1 1,0 Doo-Sabin 1 0,1 Doo-Sabin ( 0,0 Doo-Sabin , Doo-Sabin) 1, Te implementation of tis syntesis filter requires only one bilinear texture interpolation (Strengert et al., 2006). Note tat te convolution for all boundary pixels of te resulting image accesses pixels outside of te original image. Our notation also suggests an equivalent but less efficient implementation variant wit two texture interpolations instead of one:,,. Doo-Sabin g ( g ). Anoter syntesis filter may be constructed by reordering te convolution filters: ( ). g

8 Since te resulting syntesis filter is only C 0 - continuous and requires one additional texture interpolation, tere is no apparent advantage compared to te C 1 -continuous Doo-Sabin filter. 5.4 C 2 -Continuous Filter Te C 2 -continuous syntesis filter corresponding to te bicubic B-spline subdivision sceme can be constructed easily in our notation wit te elp of one additional convolution wit a 2 2 filter: cubic g ( Doo-Sabin g ). Permutations of te components of tis construction result in less symmetric and/or less smoot syntesis filters. On te oter and, te construction of iger-order B-spline filters by additional convolutions wit 2 2 filters sould now be obvious. Note, owever, tat and sould appear in alternating order as discussed in Section (Quasi-)Interpolating Filters Unfortunately, te construction of a C 1 -continuous interpolation syntesis filter is considerably more difficult tan te approximation syntesis filter corresponding to B-splines. If quasi-interpolation is sufficient, te image data of te coarsest level can be convolved wit a filter before te syntesis is performed. Te computation of appropriate convolution filters is discussed by Litke et al. (Litke et al., 2001). For example, a convolution filter for quasi-interpolation wit bicubic B-splines can be constructed tat is implementable wit tree texture interpolations: Bartlett. Our recommendation for an actually interpolating, C 1 -continuous syntesis filter corresponds to te tensor-product generalization of te four-point subdivision sceme. Since it is separable by construction, it can be implemented by a syntesis operation for te rows followed by a syntesis operation for te columns. Tis appears to result in te most efficient implementation using 4.5 bilinear texture interpolations per pixel of te resulting image. new, precise and consistent notation for convolutions wit tese filters enabled us to construct appropriate filters in a systematic way and elped us to discuss many important implementation details. In particular, we ave proposed an efficient implementation of biquadratic (and iger-order) B-spline analysis filters and of bicubic (and iger-order) B-spline syntesis filters. Apart from applications of tese filters, our plans for future work on pyramid filters include more efficient interpolating syntesis filters, tree-dimensional filters, and efficient implementations of derivative filters and nonlinear filters. REFERENCES Burt, P. J. (1981). Fast Filter Transforms for Image Processing. Computer Grapics and Image Processing, : Catmull, E. and Clark, J. (1978). Recursively Generated B-Spline Surfaces on Arbitrary Topological Meses. Computer Aided Design, 10(6): Green, S. (2005). Image Processing Tricks in OpenGL. Presentation at GDC Krüger, J. and Westermann, R. (2003). Linear Algebra Operators for GPU Implementation of Numerical Algoritms. ACM Transactions on Grapics, 22(3): Lefebvre, S., Hornus, S., and Neyret, F. (2005). Octree Textures on te GPU. In Parr, M., editor, GPU Gems 2, pages Addison Wesley. Litke, N., Levin, A., and Scroeder, P. (2001). Fitting Subdivision Surfaces. In Proceedings IEEE Visualization 2001, pages Sigg, C. and Hadwiger, M. (2005). Fast Tird-Order Texture Filtering. In Parr, M., editor, GPU Gems 2, pages Addison Wesley. Strengert, M., Kraus, M., and Ertl, T. (2006). Pyramid Metods in GPU-Based Image Processing. In Proceedings Vision, Modeling, and Visualization 2006, pages Williams, L. (1983). Pyramidal Parametrics. In Proceedings ACM SIGGRAPH 83, pages Zorin, D., Scröder, P., DeRose, T., Kobbelt, L., Levin, A., and Sweldens, W. (2000). Subdivision for Modeling and Animation. SIGGRAPH 2000 Course Notes. 6 CONCLUSION In tis work, a set of discrete pyramid filters tat are suitable for an efficient implementation based on bilinear texture interpolation as been presented. A