Correlation-based color mosaic interpolation using a connectionist approach

Transcription

1 Correlation-base color mosaic interpolation using a connectionist approach Gary L. Embler * Agilent Technologies, Inc. 75 Bowers Avenue, MS 87H Santa Clara, California 9554 USA ABSTRACT This paper presents a specialize etension of a general correlation-base interpolation paraigm, for interpolating image sample color values obtaine through a color filter mosaic. This etension features a kernel etermine from a priori assume image characteristics in the form of pre-efine (as oppose to learne) local sample neighborhoo patterns. The interpolation proceure locally convolves the colorfilte image samples with the kernel to obtain the interpolate color values. The kernel establishes a mapping from the color-filte input values to the recove color output values using weighte, ore, an threshole sums of sample values from the local sample neighborhoo. This mapping attempts to eploit local image sample interepenencies in orer to preserve etail, while minimizing artifacts. The proceure is simulate for the Bayer RGB color filter mosaic using a quasi-linear connectionist architecture that is real-time-harware-implementable. A perceptual comparison of images obtaine from this interpolation with images obtaine from bilinear interpolation shows a visible uction in interpolation artifacts. Keywors: Demosaicing, color interpolation, image interpolation, image reconstruction, superresolution, Bayer color filter array, connectionist architecture, neural network.. INTRODUCTION Image sample interpolation is well establishe as an ill-pose or uner-constraine problem with no unique solution. However, images of natural scenes typically have common characteristics that ai in obtaining a esirable interpolation solution. Color mosaic interpolation is a special case of image sample interpolation, where the interpolation nominally applies to the same-color samples in the mosaic. Eisting color mosaic interpolation approaches use static reconstruction theory-base methos (e.g., bilinear interpolation), or ynamic aaptive methos (e.g., irection-aaptive bilinear interpolation).,, 4, 5, 6, 7, 8 These methos make various assumptions about image characteristics, such as the assumption that neighboring image samples will show some egree of smoothness. * gembler@inspirics.com; phone ; fa ; this paper an supplemental materials (image files an MATLAB coe) are at Copyright Society of Photo-Optical Instrumentation Engineers. This paper will be publishe in Proceeings of SPIE Vol an is mae available as an electronic preprint with permission of SPIE. One print or electronic copy may be mae for personal use only. Systematic or multiple reprouction, istribution to multiple locations via electronic or other means, uplication of any material in this paper for a fee or for commercial purposes, or moification of the content of the paper are prohibite.

2 Images of natural scenes are known to have local luminance interepenencies. This trait may be eploite for interpolating image samples to obtain higher image resolutions. 9,, Go et al. present an etension to this paraigm for interpolating image color sample values obtaine from a color filter mosaic. This paper proposes another approach to this color mosaic interpolation metho, erive from assumptions about epecte image characteristics. These assumptions serve to facilitate real-time harware implementation of the interpolation proceure, while still offering better image quality than image quality obtainable from bilinear interpolation.. METHODOLOGY FOR COLOR MOSAIC INTERPOLATION The propose methoology makes the following assumptions regaring image characteristics: Different local sample neighborhoos within images can share similar patterns. For the similar local patterns mentione above, the patterns are similar across ifferent scales. These assumptions allow a priori information from local sample patterns to be use in obtaining interpolate color values. Accoring to these assumptions, patterns of high-resolution neighborhoos (sample neighborhoos with all color values present) are similar to patterns of the corresponing lowresolution neighborhoos (color mosaic sample neighborhoos; i.e., the inputs to the interpolation). In orer to use this methoology, the local sample patterns an their mapping to the corresponing interpolate color values must be efine. This information coul either be learne from available ata (e.g., sample images) as in Ref., or erive from assumptions about epecte image characteristics. This paper is concerne with the later approach. The following assumptions are mae: A small, o-number imensione, square spatial sample neighborhoo (e.g., samples) cente on the color output value location, is sufficient for the local sample neighborhoo size an configuration. The set of local neighborhoo sample patterns may be erive from combinations of minimum an maimum sample values in the local neighborhoo. These patterns represent various luminance graient configurations in the local neighborhoo. The local neighborhoo bilinear-interpolate color value may be use as a starting color output value. The starting color output value (i.e., the bilinear-interpolate value) may be ajuste base on the similarity of its local neighborhoo sample values to the set of sample pattern values. The ajustment amount is proportional to the luminance graient (as represente by the patterns), an the ajustment irection (increase or ecrease) is mae in the irection towars the center sample value. At the maimum ajustment amount (corresponing to the maimum luminance graient), the color output value is mae equal to the center sample value. This proceure attempts to take into account an assume correlation between the local neighborhoo sample values an the epecte center color values. This proceure also attempts to improve perceptual quality by riving the overall center color towars a more neutral tone, thus minimizing the perception of chromatic aliasing artifacts ( color fringes ) near large luminance graients. The local sample patterns are etermine from combinations of minimum an maimum local sample values. For each ifferent pattern of minimum an maimum local sample values, a esi color output value (minimum or maimum) is assigne. This assignment is mae base on the rules above; i.e., equal to the center sample value for patterns with large luminance graients. For the Bayer RGB color mosaic, patterns arising from the following two sources are use to etermine approimately the luminance graients configuration an amount: Different combinations of the green samples minimum an maimum values. Different combinations of the minimum an maimum values of samples the same color as the center color being interpolate.

3 For interpolating green, only patterns arising from the green samples minimum an maimum values are use, ue to the high perceptual correlation of green with image luminance. 4 To etermine approimately each pattern s egree an irection of ajustment to the bilinear-interpolate color values, the pre-assigne output value for each pattern is compa with the bilinear-interpolate value for the same pattern. The approimate egree an irection of ajustment is the ifference between the preassigne value an the bilinear-interpolate value. This ifference etermines the approimate weighting factors for the mapping of the patterns sample values to the color output values, through the ajustment of the bilinear-interpolate starting values. For an N N -element mosaic, where N is an o integer, the proceures of this methoology are of the general form ~ > ~ neg if cent an neg > cent, ˆ = f ( [ n, ]) = + ~ if < an + ~ n pos cent pos < cent, or, () cent where ˆ is the interpolate color value for the local neighborhoo center, [n,n ], wheren,n =,,..., N, is the local neighborhoo sample value D array, is the local neighborhoo bilinear-interpolate center value, neg an pos are, respectively, the local-neighborhoobase negative an positive ajustments to the bilinear-interpolate center value, an cent = N the local neighborhoo center sample value. Forms for iniviual, green, an e interpolate color values of a Bayer color filter mosaic are [ ], N is ~ > ~ neg if cent an neg > cent, ˆ = + ~ if < an + ~ pos cent pos < cent, or, () cent ~ > ~ neg if cent an neg > cent, ˆ = + ~ if < an + ~ pos cent pos < cent, or, () cent ~ > ~ neg if cent an neg > cent, ˆ = + ~ if < an + ~ pos cent pos < cent, or, (4) ˆ, ˆ, ˆ cent are the final interpolate, green, an e values,,, are bilinear- neg, pos are, where interpolate, green, an e values, an neg, pos, neg, pos, green, an e negative an positive ajustments to the bilinear-interpolate values. The negative an positive ajustment values neg an pos are

4 [ wnegpat fnegpat () ] ~ = ma, (5) neg [ wpospat fpospat () ] ~ = ma, (6) pos where =,,...,D is the pattern imension ine, wnegpat an wpospat are ajustment weights for negative an positive ajustments, an fnegpat () an fpospat () are negative an positive ajustment functions of local neighborhoo sample patterns. Forms for iniviual, green, an e negative an positive ajustment values of a Bayer color filter mosaic are [ wnegpat fnegpat () ] ~ = ma, (7) neg [ wpospat fpospat () ] ~ = ma, (8) pos [ wnegpat fnegpat () ] ~ = ma, (9) neg [ wpospat fpospat () ] ~ = ma, () pos [ wnegpat fnegpat () ] ~ = ma, () neg [ wpospat fpospat () ] ~ = ma, () pos where wnegpat, wpospat, wnegpat, wpospat, wnegpat, wpospat, fnegpat (), fpospat (), fnegpat (), fpospat (), fnegpat (), fpospat () are ajustment weights an functions of local neighborhoo sample patterns for, green, an e interpolation. The local neighborhoo sample pattern functions are [ fneg () + fneg ( ) + fneg ( cent) ] b if ma[] ma > b, or f negpat () =,() [ fpos () + fpos ( ) + fpos ( cent) ] b if ma[] ma > b, or f pospat () =, (4) where b is an activation threshol bias term, fneg () an fpos () are functions of the local neighborhoo samples the same color as the color being interpolate (ecept green samples), fneg an fpos ( ) ( ) are functions of the local neighborhoo green samples (ecept a center sample), an fneg ( cent ) an fpos cent ( ) are functions of the center sample (, green, or e). Note that the bias (), fpospat ()> only if term b performs a variable activation thresholing function, with fnegpat ma[]> b.

5 The local neighborhoo sample pattern functions for, green, an e interpolation are f negpat f pospat () () [ fneg ( ) + fneg ( ) + fneg ( cent) ] b if ma[] ma > b = [ fpos ( ) + fpos ( ) + fpos ( cent) ] b if ma[] ma > b = [ fneg ( ) + fneg ( cent) ] b if ma[], or,(5), or,(6) ma > b, or f negpat () =, (7) [ fpos ( ) + fpos ( cent) ] b if ma[] ma > b, or f pospat () =, (8) f negpat f pospat () () [ fneg ( ) + fneg ( ) + fneg ( cent) ] b if ma[] ma > b = [ fpos ( ) + fpos ( ) + fpos ( cent) ] b if ma[] ma > b =, or,(9), or,() where fneg ( ), fpos ( ), fneg ( ), fpos ( ), fneg ( ), fpos positive functions of the local neighborhoo, green, an e samples, ecept the center sample. The negative an positive local neighborhoo sample functions are fneg fpos ( ) are negative an ( ) = w neg [ n,n,p, ][ n, n ], () n,n, p ( ) = w pos [ n,n, p, ][ n, n ], () n,n,p where p =,,..., P is the pattern ine, an wneg[ ], wpos[ ] {~,} are sample weights, with ~ efine as the one s complement of the sample value in base-, unsigne-binary representation. The sample D location inices n, n are restricte to values from,,..., N where the requi color sample is present in the mosaic. The local neighborhoo sample functions for, green, e, an center samples are fneg ( )= w neg n,n, p, n,n, p fpos ( ) = w pos n,n, p, n,n, p [ ] [ n,n ], () [ ] [ n,n ], (4)

6 fneg ( )= w neg n,n, p, n,n,p fpos ( )= w pos n,n, p, n, n, p fneg ( )= w neg n,n, p, n,n, p fpos ( )= w pos n, n, p, n,n, p ( cent ) neg [ ] [ n,n ], (5) [ ] [ n,n ], (6) [ ] [ n,n ], (7) [ ] [ n,n ], (8) N N N N [,, ] [, ] N N N N [,, ] [, ] N N N N [,, ] [, ] for center sample, f neg = w for green center sample, or,(9) ( cent ) w w neg neg pos N N N N [,, ] [, ] N N N N [,, ] [, ] N N N N [,, ] [, ] for e center sample for center sample f pos = w for green center sample, or.() w w pos pos for e center sample The sample D location inices n, n are restricte to values from,,..., N where the corresponing color sample value is present in the mosaic. Sets of local sample patterns for a -local-neighborhoo Bayer mosaic are shown in Figures through 8. Each figure shows patterns for one of the eight possible input-output configurations of this mosaic. This number of configurations arises from the four ifferent Bayer color sample types: Re samples (R), Green samples in rows with samples (GR), Green samples in rows with e samples (GB), Blue samples (B),

7 .. cent = = X X X X X X X X X X X X X X X X XXXX X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X XXXX X X X X X X X X X X X X X X X X.. cent = = Input [,] [,] [,] = = = [,] [, ] = cent [,] = = = [,] [,] [,] = = = Pattern Locations ˆ Output Ajustment ˆ ˆ. 75 = ˆ ˆ. 5 = ˆ ˆ. 5 ˆ = ˆ = ˆ ˆ +. 5 = ˆ ˆ +. 5 = ˆ ˆ = Notes. Pattern values quantize to, ˆ, { }.. Fig. Bayer RGB R-Center G-Output patterns. cent = = X - X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X - X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. cent = = Input Pattern Locations Output Ajustment Notes [,] [,] [,] = = = [,] [, ] = cent [,] = = = [,] [,] [,] = = = ˆ ˆ ˆ ˆ ˆ. 75 ˆ ˆ ˆ ˆ ˆ = = ˆ =.5 ˆ =.5 ˆ = ˆ = +.5 ˆ = +.5. Pattern values quantize to, ˆ, { } ˆ ˆ = +.75 ˆ ˆ + = Fig. Bayer RGB R-Center B-Output patterns.

8 .. cent = = X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X Input X X [,] [,] [,] [,] [, ] = cent [,] [,] [,] [,] = = = = = = = = =.. X X X X Pattern Locations ˆ X X X X X X X X cent = Output Ajustment ˆ ˆ ˆ X X X X X X Fig. Bayer RGB GR-Center R-Output patterns. = ˆ ˆ = ˆ =.5 ˆ = ˆ = +.5 ˆ ˆ + = cent = = Notes X X X X X X. Pattern values quantize to, ˆ, { } X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X Input X X [,] [,] [,] [,] [, ] = cent [,] [,] [,] [,] = = = = = = = = = X X X X X X X X X X X X X X cent = = Pattern Output Ajustment Locations ˆ ˆ = ˆ ˆ ˆ =. 5 ˆ ˆ = ˆ ˆ = +.5 ˆ ˆ + = X X X X Notes Fig. 4 Bayer RGB GR-Center B-Output patterns. X X X X X X. Pattern values quantize to, ˆ, { } X X

9 .. cent = =.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X Input X X [,] [,] [,] [,] [, ] = cent [,] [,] [,] [,] = = = = = = = = =.. X X X X X X X X X X X X X X cent = = Pattern Output Ajustment Locations ˆ ˆ = ˆ ˆ ˆ =. 5 ˆ ˆ = ˆ ˆ = +.5 ˆ ˆ + = X X X X Notes Fig. 5 Bayer RGB GB-Center R-Output patterns. cent = = X X X X X X. Pattern values quantize to, ˆ, { } X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. X X Input X X [,] [,] [,] [,] [, ] = cent [,] [,] [,] [,] = = = = = = = = = X X X X X X X X X X X X cent = = Pattern Output Ajustment Locations ˆ ˆ = ˆ ˆ ˆ =. 5 ˆ ˆ = ˆ ˆ = +.5 ˆ ˆ + = X X X X X X Notes Fig. 6 Bayer RGB GB-Center B-Output patterns. X X X X X X. Pattern values quantize to, ˆ, { } X X

10 .. cent = = X - X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X - X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X.. cent = = Input Pattern Locations Output Ajustment Notes [,] [,] [,] = = = ˆ ˆ ˆ ˆ ˆ. 75 = =. Pattern values quantize to, ˆ {,} [, ] [, ] = cent [,] = = = ˆ ˆ ˆ =.5 ˆ =.5 [,] [,] [,] = = = ˆ ˆ ˆ ˆ ˆ = ˆ = +.5 ˆ = +.5 ˆ = +.75 ˆ ˆ + =.. Fig. 7 Bayer RGB B-Center R-Output patterns. cent = = X X X X X X X X X X X X X X X X XXXX X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X XXXX X X X X X X X X X X X X X X X X.. cent = = Input [,] [,] [,] = = = Pattern Locations ˆ Output Ajustment ˆ ˆ. 75 = ˆ ˆ. 5 = Notes. Pattern values quantize to, ˆ, { } [,] [, ] = cent [,] = = = ˆ ˆ. 5 ˆ = ˆ = [,] [,] [,] = = = ˆ ˆ +. 5 = ˆ ˆ +. 5 = ˆ ˆ = Fig. 8 Bayer RGB B-Center G-Output patterns.

11 each cente in the pattern, an each having two interpolate center color outputs. The patterns groupe accoring to their center sample colors are R-Center Patterns = { R-Center G-Output Patterns, (Fig. ) R-Center B-Output Patterns }, (Fig. ) GR-Center Patterns = { GR-Center R-Output Patterns, (Fig. ) GR-Center B-Output Patterns }, (Fig. 4) GB-Center Patterns = { GB-Center R-Output Patterns, (Fig. 5) GB-Center B-Output Patterns }, (Fig. 6) B-Center Patterns = { B-Center R-Output Patterns, (Fig. 7) B-Center G-Output Patterns }. (Fig. 8) The pattern number, P, is equal to the total number of input-output configurations, so P = 8 for this case. The figures show the patterns using a notation, ˆ {,}, where, ˆ = represents the smallest value,, ˆ = represents the largest value, an = X represents a value of input sample that is irrelevant to the interpolate color output value ˆ. The ˆ output values shown in the patterns are assigne accoring to the assumptions above. For a center sample value of either = or = in the patterns, the output ˆ is assigne a value of either ˆ = or ˆ =, such that ˆ is equal to the center sample value when the pattern shows a large luminance graient. The figures also show, using a iagonal line notation, the assigne values of neg an pos, the egree an irection of ajustment to the bilinear-interpolate values,. The iagonal lines inicate the ifferent egrees an irections of ajustment using ifferent line thicknesses an slopes. The line thickness is proportional to the egree of ajustment, an the line slope irection (positive or negative) is the same as the ajustment irection. For no ajustment; i.e., ~, ~ =, no line is shown. neg pos The egree an irection of ajustment are efine from the pattern sample values as the ifference between the assigne output value an the bilinear-interpolate value, shown by rearranging Equation as ~ ˆ for ˆ <, or neg = an () ~ ˆ for ˆ >, or pos =. () For sample pattern values quantize to, ˆ {,} as shown in Figures through 8, the possible bilinearinterpolate output values are {,.5,.5,.75,}. The possible ajustment values are therefore (from Equations an ) neg, pos {,.5,.5,.75,}. These values etermine the ajustment weights wnegpat an wpospat values in Equations 5 an 6, as follows. wnegpat an wpospat are efine as equal to the change in neg an pos versus the change in the input sample pattern functions fnegpat () an fpospat (), shown by rearranging Equations 5 an 6 as wnegpat = neg () fnegpat an ()

12 wpospat = pos () fpospat. (4) The neg an pos values above are equal to the change in neg an pos over the full range of fnegpat an fpospat(), as prouce by the entire range of associate input values, from the full pattern value, to the full complement of the same pattern value. Therefore, the wnegpat an wpospat values are numerically equivalent to the neg an pos values, i.e., wnegpat,wpospat = neg, pos {,.5,.5,.75,}. Also, the imension number D is equal to the total number of neg or pos values. For the Bayer mosaic uner consieration, D = 5. For each ifferent ajustment egree an irection neg an pos shown in the figures, separate inputoutput transfer function equations are evelope from the pattern values, as given by Equations an. These equations are then algebraically uce to minimize equations, as follows. With the pattern input values alreay quantize to {,}, the pattern ajustment output values are likewise quantize to {,}, with set to = for any non-zero value, an set to =. This allows the use of stanar Boolean algebraic minimization techniques to minimize the equations. The outputs of the minimize equations are summe an threshole as shown in Equations an 4. These equation outputs are then scale by their particular wnegpat an wpospat values, an either ae to or subtracte from the bilinear-interpolate value to form the final output value ˆ, as shown by Equations, 5, an 6. Eamination of Figures through 8 reveals that out of the eight ifferent input-output configurations, only three unique pattern sets eist. Grouping together ientical pattern sets, the three groups of unique pattern sets are Patterns Group = { R-Center G-Output Patterns, (Fig. ) B-Center G-Output Patterns }, (Fig. 8) Patterns Group = { R-Center B-Output Patterns, (Fig. ) B-Center R-Output Patterns }, (Fig. 7) Patterns Group = { GR-Center R-Output Patterns, (Fig. ) GR-Center B-Output Patterns, (Fig. 4) GB-Center R-Output Patterns, (Fig. 5) GB-Center B-Output Patterns }. (Fig. 6) This grouping allows the total number of interpolation kernels to be restricte to three, the total number of unique pattern sets. The patterns shown accoring to their group esignations are R-Center Patterns = { R-Center G-Output Patterns, (Group ) R-Center B-Output Patterns }, (Group ) GR-Center Patterns = { GR-Center R-Output Patterns, (Group ) GR-Center B-Output Patterns }, (Group ) GB-Center Patterns = { GB-Center R-Output Patterns, (Group ) GB-Center B-Output Patterns }, (Group ) B-Center Patterns = { B-Center R-Output Patterns, (Group ) B-Center G-Output Patterns }. (Group ) ()

13 This grouping inicates a requirement for two sets of harware implementing the Group pattern interpolation, in orer to concurrently obtain the two interpolate center color values of the GR-center an GB-center configurations.. RESULTS The proceure was simulate using The MathWorks MATLAB with the Lighthouse true-color (RGB, 8 bits-per-color) test image. 5 This image was ecimate an converte from a Tagge Image File Format image to a Microsoft Winows Bitmap image (Figure 9). The image was uce to a Bayer RGB color mosaic image by removing the appropriate two color samples at each sample location (Figure ). This color mosaic image was then interpolate in an attempt to recover the remove color samples. The resulting interpolate images were perceptually eamine for image quality an interpolation artifacts. Fig. 9: Lighthouse image Fig. : Lighthouse Bayer RGB mosaic. Figure shows the bilinear-interpolate image with no ajustment. Figure shows the bilinearinterpolate image with ajustment, using empirically obtaine activation threshol bias b values. The bias values were perceptually ajuste to approimately optimal values. Values set too high resulte in insufficient ajustment, with the image ehibiting ecessive color fringes. Values set too low resulte in ecessive ajustment, with the image ehibiting checkerboar artifacts. All ajuste images shown here use the same bias values.

14 Fig. : Lighthouse bilinear-interpolate. Fig. : Lighthouse bilinear-interpolate an ajuste. In Figure, some color fringe uction an image sharpening are present, but the image still contains substantial amounts of color fringes, as well as zipper noise artifacts an false colors. Another metho for the bilinear interpolation was trie to improve the resultant image. The new metho interpolate the image using the sample pattern values to irect the bilinear interpolation of colors having four samples in the local neighborhoo. For these colors, the interpolation is selecte from either the average of one of the two pairs of opposite-positione samples, or the average of all four samples. The interpolation irection is chosen base on the similarity of the sample values to the pattern values, with each pattern value associate with a prefer interpolation irection. The pattern values an interpolation irections are associate through the correlations of the pattern values. Pattern values with a high egree of correlation in a particular irection are associate with interpolation in that irection. The ajustment weights are moifie epening on whether the selecte interpolation uses two samples or four samples. Figure shows the interpolate image using the new interpolation metho, but without ajustment, an Figure 4 shows the same image with ajustment. Some uction in artifacts can be seen with this metho, but the image still contains significant amounts of false colors.

15 Fig. : Lighthouse irecte-linear-interpolate. Fig. 4: Lighthouse irecte-linear-interpolate an ajuste. To uce the false colors, meian filtering of the an e color values, reference to the green values, was use on the interpolate an ajuste image.6 The meian values of the color value ifferences ( green) an (e green) were etermine over the local neighborhoo an ae to the center green value to obtain new center an e values. These new an e values replace the eisting values. Figure 5 shows the results of this operation. The false colors are noticeably uce.

16 Fig. 5: Lighthouse irecte-linear-interpolate, ajuste, an meian-filte. 4. CONCLUSION This paper escribes a methoology for color emosaicing using a ata-riven connectionist approach. This methoology provie visible improvements over linear interpolation proceures. Although this paper presents the methoology in conjunction with bilinear interpolation, in theory it may be use in conjunction with other interpolation types to improve the resultant image quality. The methoology uses only simple integer an logical operations over a small local sample neighborhoo, thus facilitating its harware implementation, especially for real-time piel-rate harware. As currently efine, the methoology requires only binary shifts, complementation, aition, comparison, an ata selection, all having fairly simple harware implementations. Iterative, recursive, or learning operations are not requi. Goals for future investigation inclue: Development of a more theoretical founation for the methoology, performance optimization for general images or for special classes of images, optimization for harware implementation, an implementation of the optimize proceure in harware. Topics for future investigation inclue: Optimization of activation threshol bias values for the ajustment function. Nonlinear activation functions (e.g., sigmoi function) for the ajustment. Larger local neighborhoos; e.g., 5 5 samples. The primary isavantage of larger neighborhoos is the large increase in the proceure s esign compleity an computational requirements. Theoretical links to wavelet-base image reconstruction. The methoology bears a semblance to wavelet or wavelet-like image reconstruction. The linear interpolation resembles low-frequency

17 signal recovery through a Haar scaling function, an the ajustment to the linear interpolation resembles high-frequency signal recovery through a Haar wavelet function. 7 Interactions with characteristics of the image acquisition process an with characteristics of other image processing functions. Combinations of this methoology with other color mosaic interpolation methoologies. One propose application of this methoology is for proviing a final correction of last resort to an image previously processe by another interpolation methoology. ACKNOWLEDGMENTS The author woul like to thank Dr. To Sachs of Agilent Technologies for his helpful suggestions an comments. REFERENCES. A. M. Tekalp, Digital Vieo Processing, Prentice Hall, Upper Sale River, New Jersey (995), Chap. 7.. T. Chen, Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras, Psych/EE6 Course Project, Image Systems Engineering Program, Department of Electrical Engineering, Stanfor University (999), J. F. Hamilton an J. E. Aams, Averaging green values for green photosites in electronic cameras, Unite States Patent 5,596,67, Jan., J. E. Aams an J. F. Hamilton, Aaptive color plane interpolation in single sensor color electronic camera, Unite States Patent 5,65,6, Jul. 9, R. Kimmel, "Demosaicing: Image Reconstruction from Color CCD s," IEEE Trans. Image Processing 8(9), -8 (999). 6. S. B. Aison, Metho an system for color filter array multifactor interpolation, Unite States Patent 5,99,95, Nov., D. D. Muresan, S. Luke, an T. W. Parks, Reconstruction of Color Images from CCD Arrays, Teas Instruments DSPS Fest, Houston, Teas, (), 8. M. R. Gupta an T. Chen, Vector Color Filter Array Demosaicing, Proc. SPIE Sensors an Cameras Systems for Scientific, Inustrial, an Digital Photography Applications III 46, San Jose, California (). 9. F. Ahme, S. C. Gustafson, an M. A. Karim, "High-Fielity Image Interpolation using Raial Basis Function Neural Networks," Proc. IEEE 995 Aerospace an Electronics Conf., (995).. N. Plaziac, "Image Interpolation Using Neural Networks," IEEE Trans. Image Processing 8(), (999).. F. M. Canocia an J. C. Principe, "Superresolution of Images Base on Local Correlations," IEEE Trans. Neural Networks (), 7-8 (999), J. Go, K. Sohn, an C. Lee, "Interpolation Using Neural Networks for Digital Still Cameras," IEEE Trans. Consumer Electronics 46(), 6-66 ().. B. E. Bayer, Color imaging array, Unite States Patent,97,65, Jul., J. Aams, K. Parulski, an K. Spauling, Color Processing in Digital Cameras, IEEE Micro 8(6), - (998). 5. The School of Electrical an Computer Engineering, Cornell University, Thomas Parks Digital Signal Processing Lab, Color Images Database, 6. W. T. Freeman, Metho an apparatus for reconstructing missing color samples, Unite States Patent 4,66,655, May 5, E. J. Stollnitz, A. D. DeRose, an D. H. Salesin, Wavelets for Computer Graphics: Theory an Applications, Morgan Kaufmann, San Francisco (996).