Modelling, calibration and correction of nonlinear illumination-dependent fixed pattern noise in logarithmic CMOS image sensors

Transcription

1 IEEE Instumentation and easuement Technology Confeence Budapest, Hungay, ay 2 23, 200 odelling, calibation and coection of nonlinea illumination-dependent fixed patten noise in logaithmic COS image sensos Dileepan Joseph and Steve Collins Depatment of Engineeing Science Univesity of Oxfod, OX 3PJ, U Abstact At pesent, most COS image sensos use an aay of pixels with a linea esponse. Howeve, logaithmic COS sensos ae also possible, which ae capable of imaging high dynamic ange scenes without satuating. Unfotunately, logaithmic sensos suffe fom fixed patten noise (FPN). Wok epoted in the liteatue geneally assumes the FPN is independent of illumination. This pape develops a nonlinea model y = a + bln(c + x) of the pixel esponse y to an illuminance x showing that FPN aises fom vaiation of the offset a, gain b and bias c. Equations ae deived, which can be used to extact these paametes by calibation against a unifom illuminance of vaying intensity. Expeimental esults, demonstating paamete calibation and FPN coection, show that the nonlinea model outpefoms pevious models that assume eithe only offset o offset and gain vaiation. Illuminance x Output bias Optics V DD Pixel cicuit T V DD T3 T2 Column cicuit Row select eywods COS image sensos, logaithmic pixels, fixed patten noise. I. INTRODUCTION The CCD image senso, a dominant technology fo nealy thee decades, faces tough competition fom the COS image senso [], [2], [3], a moe ecent technology. Since thei fabication pocess is incompatible with conventional electonics, CCD sensos equie extenal cicuits to povide bias voltages, clock signals, contol logic, analogue-to-digital convesion and signal pocessing. COS technology, howeve, pemits the integation of these cicuits on the same die as the senso to educe the cost, powe consumption, size and weight of the final camea. Fundamentally, COS pixels scale well with shinking pocess geometies because moe electonics can be placed in each pixel to impove the output without affecting sensitivity o esolution. Fo these and othe easons, such as a highe quantum efficiency, less smea and blooming, bette yields and pice pessue fom moe competition, the electonics industy expects COS gadually to eplace CCD image sensos. This pape concens a subset of COS senso technology, namely logaithmic images. Unlike a linea pixel (CCD o COS), which integates the chage poduced by photon absoption ove a finite peiod, a logaithmic pixel, as in Figue, continuously convets incident photons into a voltage that is popotional to the logaithm of the light intensity ove moe than five decades of illuminance [4]. Such a non-integating senso can be andomly accessed in space and time, a featue useful in some industial applications fo which fame size and speed may be taded against each othe. As these pixels ae simple, consisting of thee tansistos and a diode, sensos have ADC Response y Output cicuit Column bias T6 T5 Column select Fig.. Fom an illuminance x to a esponse y in one pixel of a logaithmic COS image senso. been made with 2048 by 2048 pixels and acceptable yields [5]. In eality, light eflected by scenes spans many decades of illuminance, fom 0.00 lux at night to between and 000 lux in indoo lighting and up to 0,000 lux in bight sunlight [6]. Diect viewing and speculaities of bight souces, such as oncoming headlights o the sun, may lead to highe intensities. The advantage of a logaithmic senso is that, ove five decades of illuminance, ten bits of esolution ae sufficient to sense illuminance with one pecent accuacy. With a linea senso, 23 bits ae necessay to accomplish the same task. This would be costly fo still cameas and extemely difficult at video ates. Of couse, a linea senso with a smalle esolution could adapt ove a lage dynamic ange by apetue o integation-time contol. Howeve, satuated patches (black o white) would appea in images of scenes that span a high dynamic ange, such as most outdoo scenes in daylight o an indoo scene with a bight window. any non-logaithmic methods have been poposed to extend the dynamic ange of image sensos [6] but most esult in deceased esolution, sensitivity o fame ate /0/$ IEEE

2 One disadvantage of a logaithmic senso is a loss of the aveaging effect of integation, which impoves low-light sensitivity [4]. Howeve, the biggest poblem by fa is fixed patten noise (FPN), which is a distotion of the image due to vaiations in device paametes acoss the senso. Dieickx, Scheffe, Loose and othes have developed digital and analogue methods to coect this eo by assuming it is independent of illuminance [4], [5], [7]. Loose et al biefly consideed FPN as a linea function of illuminance but concluded that the dependence was not significant [7]. Howeve, Yadid- Pecht notes that FPN vaies nonlinealy with illuminance in a logaithmic senso but she neithe chaacteises no attempts to coect this distotion [6], which is the subject of this pape. Section II models the esponse of a logaithmic pixel to illuminance. Section III deives equations to calibate the model, fom images of a unifom scene, and to coect FPN. Section IV gives expeimental esults of calibation and coection. II. ODELLING Figue shows the pocess by which light, of illuminance x, falling on a pixel in a typical logaithmic COS senso is conveted to a digital esponse y. Befoe the light eaches the photodiode in the pixel, it is attenuated due to absoption and eflection by the apetue and lens of the camea, which may be epesented by gains G A and G L. Photons absobed by the photodiode fom electon hole pais that ae swept out by the electic field acoss the device to poduce a cuent I P,given in (). This photocuent is linealy elated to the incident light intensity ove many odes of magnitude. The elationship depends on the quantum efficiency, which may be epesented by a gain G Q, and the light-sensitive aea A of the photodiode. I P = G A G L G Q Ax () The photodiode in Figue is evese biased to pevent any cuent flowing to gound though it except fo the photocuent. Howeve, a small leakage cuent I S, known as the evese bias satuation cuent, also flows to gound though this diode. The total cuent I P + I S sets the gate voltage V T2 G, given in (2), of tansisto T2 via the diode-connected load tansisto T. Designed to opeate in the subtheshold egion, T has a logaithmic cuent-to-voltage elationship that is valid ove many decades of cuent amplitude. V T2 G = V DD ; nt kt q ln IP + I S I T on ; V T on (2) Tansisto T3 is a switch that is eithe an open o a shot cicuit between T2 and the output line fo a column of pixels. This output line is biased by tansisto. When T3 is off, T2 is disconnected fom the line and does not affect its voltage. When T3 is on, a simila switch is off fo all othe pixels in the column and the gate voltage VG T5 of tansisto T5, given in (3), equals the souce voltage VS T2 of T2. As T2 and have the same dain-souce cuent, when T3 is on, and as both opeate in satuation, thei gate-souce minus theshold voltages VGS T2 ; VT T2 and VGS ; VT ae linealy elated. VG T5 = V G T2 ; VT T2 ; ; V T2 GS ; VT (3) When a pixel is connected to the output line fo its column, all pixels in the same ow ae connected to thei espective output lines. Howeve, the analogue-to-digital convete (ADC) pocesses only one voltage at a time. Theefoe, the column outputs ae switched in sequence onto the ADC input line, which is biased by tansisto, using a two-tansisto cicuit simila to the one descibed above. When tansisto T6 is switched on, T5 is connected to the input line and the dain voltage VD of, given in (4), equals the souce voltage VS T5 of T5. V D = V G T5 ; VT T5 ; ; V T5 GS ; VT Rathe than getting into the details of ADC cicuits, equation (5) abstacts the digitisation of voltage VD by a clipping function, to limit the maximum and minimum output values, and by ounding off, which intoduces quantisation eo. Futhemoe, the ADC adjusts its input VD by an offset F ADC and gain G ADC to fit the domain of voltages to the ange of intege codes (e.g counts fo an 8-bit ADC). ; y = ound clip; FADC + G ADC V (5) (4) If the input voltage does not cause clipping, digitisation may be modelled by a quantisation eo tem Q, with a ange of one LSB (i.e. 0:5), that is added to the output. Futhemoe, the whole pocess in Figue will add noise components at vaious stages. Howeve, the noise shall be modelled by a single andom vaiable N added to the output. A futhe tem D is added to the output to account fo eo in the undelying device equations. Consideing these emaks, equation (6) gives the digital esponse y of a pixel. y = F ADC + G ADC V D + Q + N + D (6) Gouping the equations and physical paametes above, equations (7) () give the esponse y of a pixel as a logaithm of the illuminance x, with thee abstact paametes a, b and c, named the offset, gain and bias, and an eo tem. A vaiation fom pixel to pixel of a, b, c o a combination theeof causes FPN. Theefoe, these paametes must be detemined by calibation to coect FPN in an image. Futhemoe, the statistics of the output-efeed eo (the eo may also be efeed to the input x) must be estimated to validate the model and detemine the accuacy of calibation and coection. y = a + b ln (c + x)+ (7) D

3 a = F ADC + G ADC V DD + nt kt Ion T ln q G A G L G Q A ; VT T2 ; ; V T2 GS ; VT ; VT T5 ; ; V T5 GS ; VT n T kt b = ;G ADC q I S c = G A G L G Q A ; V T on (8) (9) (0) = Q + N + () III. CALIBRATION AND CORRECTION Since the complexity of calibation and coection depends on the numbe of paametes that vay spatially, no moe vaiables should be intoduced than ae necessay. Sections III-A, III- B and III-C conside the cases whee offset vaiation, offset and gain vaiation o offset, gain and bias vaiation cause FPN. Analytic solutions ae possible fo the fist two cases as coection, which esults in a logaithmic epesentation of the scene, is pefomed by a linea tansfom. The thid case must be solved by iteation as coection, which esults in a linea epesentation of the scene, is pefomed by a nonlinea tansfom. A. Offset vaiation Equation (2) models the esponse ^y ij of the j th pixel, in an aay of N pixels, to an illuminance x i, whee the offset a j vaies spatially. The gain b and bias c ae constant fo all pixels. ^y ij = a j + b ln(c + x i ) (2) The model paametes may be detemined by minimising the mean squae eo (SE), defined in (3) below, between the actual esponse y ij of the pixels and the model esponse ^y ij to diffeent but unifom illuminances x i. SE = N (y ij ; ^y ij ) 2 (3) The SE does not have a unique global minimum because (2) is invaiant unde tansfomations (4) (6), which means that all paametes cannot be detemined fom the data y ij. (a j b c x i ) (a j b c; x i + ) (4) (a j ; b ln b c x i ) (5) a j b 0 (c + x i) (6) Intuitively, the offset of a pixel is popotional to the diffeence between the pixel s aveage esponse and the aveage esponse of all pixels, a method by which Loose et al (amongst othes) coect FPN [7]. Equation (7) gives the aveage esponse y i of all pixels to illuminance x i, which elates to the paametes by (8) since y ij and ^y ij diffe, due to (7), by a andom vaiable. y i = N y ij (7) y i a + b ln (c + x i ) (8) a = N a j (9) Thus, the model esponse ^y ij is a linea function of the aveage esponse y i leaving one vaiable a 0 j pe pixel. ^y ij = a 0 j +y i (20) a 0 j = a j ; a (2) The minimum of the SE occus when the patial deivatives, given in (22), of the SE with espect to the vaiables a 0 j equal zeo. Solving fo a 0 j gives (23), whee y j and y ae the aveage esponse of the j th pixel and of all 0 j = ;2 N (y ij ; a 0 j ; y i ) (22) a 0 j =y j ; y (23) y j = y = y ij (24) y i (25) Thee vaiables a, b and c emain unknown, consistent with (4) (6). Nonetheless, offset FPN may be coected fo an image y j by subtaction, as in (26), leaving a esult y 0 j elated to the logaithm of the scene x j plus some eo. y 0 j = y j ; a 0 j =a + b ln (c + x j)+ (26) The vaiance 2 of the eo is estimated in (27) fom the calibation data y ij. The vaiance is like the SE in (3) except that the denominato equals the degees of feedom (DOF), poviding an unbiased estimate [8]. The DOF, given in (28), is the numbe of constaints N (i.e. y ij ) minus the numbe of vaiables + N (i.e. y i and a 0 j ) fitted to the same constaints. 2 = DOF (y ij ; ^y ij ) 2 (27)

4 DOF = N ; ; N (28) Equation (29) gives the vaiance i of the eo ove all pixels at the i th illuminance. Section IV uses this statistic to identify a dependence of eo on illuminance, if it exists. 2 i = DOF B. Offset and gain vaiation (y ij ; ^y ij ) 2 (29) Equation (30) models the esponse ^y ij of the j th pixel, in an aay of N pixels, to an illuminance x i, whee the offset a j and gain b j vay spatially. The bias c is constant fo all pixels. ^y ij = a j + b j ln(c + x i ) (30) As befoe, the model paametes may be found by minimising the SE in (3). The SE does not have a unique global minimum because (30) is invaiant unde tansfomations (4) (6) (with b eplaced by b j ). Intuitively, coecting the offset and gain vaiation should move the esponse of a pixel close to the aveage esponse of all pixels. This aveage y i in (7) elates to the paametes by (3), with a and b in (9) and (32). y i a + bln(c + x i ) (3) b = N b j (32) Compaing (30) and (3), ^y ij is a linea function, given in (33), of y i, leaving two vaiables a 0 j and b0 j pe pixel. ^y ij = a 0 j + b0 j y i (33) a 0 j = a j ; b 0 ja (34) b 0 j = b j b (35) inimising the SE between y ij and ^y ij is equivalent to solving N independent linea egession poblems whee the abscissae ae y i and the odinates ae y ij. Thus, a 0 j and b0 j ae given by (36) and (37) using the equations of linea egession [8], whee y j and y ae given in (24) and (25). a 0 j =y j ; b 0 j y (36) P b (y 0 j = ij ; y j )(y i ; y) P (y (37) i ; y) 2 Thee vaiables a, b and c emain unknown, as in Section III-A. Nonetheless, offset and gain FPN may be coected by subtaction and division, as in (38), giving a esult popotional to the logaithm of the scene illuminance plus some eo. y 0 j = y j ; a 0 j b 0 j =a + bln(c + x j )+ (38) Equations (27) and (29) give the vaiance of the eo using the DOF in (39), since thee ae two paametes pe pixel. C. Offset, gain and bias vaiation DOF = N; ; 2N (39) Although it begins the same way, thee paamete calibation diffes fom one and two paamete calibation. Equation (40) models the esponse ^y ij of the j th pixel, in an aay of N pixels, to an illuminance x i, whee a j, b j and c j vay spatially. ^y ij = a j + b j ln(c j + x i ) (40) The model paametes may be found by minimising the SE in (3). The SE does not have a unique global minimum because (40) is invaiant unde tansfomations (4) and (5) (with b and c eplaced by b j and c j ) but (6) does not apply because of bias vaiation. Unlike befoe, the intuitive appoach, i.e. fitting the esponse of each pixel to the aveage esponse, fails. Although the aveage y i in (7) elates to the paametes by (4), with a in (9), ^y ij cannot be witten as a linea function of y i because of the nonlinea effect of bias vaiation. y i a + N b j ln(c j + x i ) (4) Howeve, ^y ij is a linea function, given in (42), of l ij, defined in (43), but l ij depends on the unknown bias and illuminance. ^y ij = a j + b j l ij (42) l ij =ln(c j + x i ) (43) If l ij is known then minimising the SE between y ij and ^y ij is equivalent to solving N independent linea egession poblems with abscissae l ij and odinates y ij. Thus, a j and b j ae detemined by (44) (46), whee y j is given in (24). a j =y j ; b j l j (44) b j = l j = P (y ij ; y j )(l ij ; l j ) P (l (45) ij ; l j ) 2 l ij (46) Equations (44) and (45) show that, at the minimum of the SE, a j and b j ae functions of c j and x i. Theefoe, the SE is a

5 known function of only c j and x i, which educes the numbe of vaiables by almost two-thids. No analytic expession fo c j and x i that minimises the SE has been found because the patial deivatives of the SE with espect to c j and x i ae highly nonlinea. Howeve, the conjugate gadient algoithm [9] finds the minimum succesfully by iteation. inimisation yields biases c 0 j and illuminances x0 i within a linea tansfom of the actual values c j and x i because of (4) and (5). Nonetheless, FPN may be coected fo an image y j by a nonlinea tansfom, as in (47), whee a 0 j and b0 j ae found using (44) and (45). Unlike in sections III-A and III-B, the coected image x 0 j is a linea function of the scene x j, with an unknown offset and gain, plus an input-efeed eo. x 0 j = exp y j ; a 0 j b 0 j! ; c 0 j = + (x j + ) (47) Standad deviation of eo (counts) Offset model Offset and gain model Offset, gain and bias model Aveage esponse of pixels (counts) Fig. 2. Response eo vesus aveage esponse i y i fo one, two and thee-paamete calibation. Equations (27) and (29) give the vaiance of the eo using the DOF in (48), since thee ae thee paametes pe pixel. DOF = N; ; 3N (48) Offset model Offset and gain model Offset, gain and bias model Expeimental data IV. EXPERIENTAL RESULTS Expeiments wee done using a 52 by 52 pixel (N =52 2 ) Fuga 5d logaithmic image senso [4]. The camea, intefaced to a PC, had an 8-bit ADC with a pogammable offset voltage F ADC. By captuing seveal fames with diffeent offset settings, the esolution was inceased to 0 bits in softwae. Images of white pape unde unifom illumination povided calibation data y ij. The iis ing was otated, theeby changing the apetue, to vay the illumination eaching the focal plane. Paametes of the offset model, offset and gain model and offset, gain and bias model wee extacted accoding to Section III. A. Calibation The fist expeiment used an 800 Watt tungsten lamp, with a filte to simulate daylight, and involved 24 images ( =24)to pemit a detailed compaison of esults. Afte calibation, the standad deviation of the eo was 3.9,.9 and 0.9 counts fo the one, two and thee paamete models, which may not appea to be a significant diffeence. Loose et al also epoted a small diffeence between one and two paamete calibation, which is why they chose not to coect gain vaiation [7]. Howeve, when the eo is shown vesus illuminance, as in Figue 2, the models diffe makedly. The one paamete model has a minimum eo of two counts in the middle of the domain, with eo ising on each side to eight and fou counts. The two paamete model has a maximum eo of two counts in the middle, flanked by two minima of one count and ising to fou and thee counts at the sides. In contast, the thee paamete model has a flat eo of less than one count. The small Response of a pixel (counts) Aveage esponse of pixels (counts) Fig. 3. Theoetical ^y ij and expeimental y ij esponse of two pixels vesus aveage esponse y i fo one, two and thee-paamete calibation. but shap ises at the sides may be due to paamete ovefitting, which would favou the midange of data. These esults suggest that the nonlinea model fits the data vey well, at least in the estimated dynamic ange of thee decades. Figue 3 shows the theoetical and expeimental esponse of two pixels. While the offset model fits the top pixel s esponse well, it does not fit the bottom pixel s esponse because of a diffeent esponse slope fom one pixel to the othe. Instead, it intesects the esponse in the middle, minimising the SE, which explains the v-shaped cuve in Figue 2. The offset and gain model matches the esponse slopes of both pixels but intesects each esponse twice as the actual esponse follows a cuved path (especially in the bottom pixel). This accounts fo the w-shaped cuve in Figue 2. The offset, gain and bias model has no poblem following the cuved esponses and the model eo vaies andomly with illuminance, as in Figue 2.

6 This pape has modelled the esponse of a logaithmic COS pixel to illuminance. The model has numeous physical paametes but can be abstacted by a logaithmic function with only thee paametes an offset, gain and bias. A spatial vaiation of these paametes leads to fixed patten noise (FPN). Although it is well known that theshold voltage vaiation, in the pixel and column souce followe tansistos, leads to FPN, the model shows othe contibutions to offset vaiation and highlights possible souces of gain and bias vaiation. Bias vaiation makes the FPN a nonlinea function of illuminance. Using mean squae eo minimisation, equations to extact the model paametes fom images of unifom illuminance wee deived. Exact solutions wee given fo the case whee gain and bias vaiation ae ignoed and the case whee bias vaiation is ignoed. No exact solution was possible fo the case whee all thee paametes vay spatially but the numbe of unknowns was educed analytically by two-thids. The emaining paametes wee obtained by numeical optimisation. Expeimental esults validate the nonlinea model and demonstate calibation and coection. Although the aveage eo between one, two and thee paamete calibation does not diffe by much, the eo vesus illumination diffes substantially. The one and two paamete models have v-shaped and w- shaped eo cuves, poving they ae poo models. In contast, the thee paamete model has an eo less than one count and independent of illuminance ove appoximately thee decades. Fig. 4. A scene, using one, two and thee-paamete coection (left to ight), imaged with apetues of.8, 4, 8 and 6 f-stops (top to bottom). B. Coection Figue 4 shows fou scenes afte FPN coection, using the one, two and thee paamete models (calibated in fluoescent light with only five images of white pape, i.e. =5). The thee paamete coection was displayed on a logaithmic scale fo consistency. The fou scenes ae eally one scene made dake by changing the apetue fom.8 (wide open) to 4, 8 and 6 f-stops. The inte-scene dynamic ange is 38 db and the intascene dynamic ange is 29 db, fo a total of 67 db. Because the scenes ae the same going fom top to bottom in Figue 4 except getting dake, an ideal logaithmic image senso would give the same esponse with a pogessively moe negative offset. Since each image has been stetched linealy to fit the 8-bit ange of the display, images in a column should look identical. Thee paamete coection gives good esults and is bette than one o two paamete coection. Similaly, two paamete coection gives bette esults than one. V. CONCLUSION Whethe nonlinea coection poves to be a pactical appoach to coect FPN in logaithmic image sensos emains to be seen. Nonetheless, this pape shows that while analogue techniques to coect pixel and column offsets, such as coelated double sampling and delta diffeence sampling, ae useful to educe FPN (and =f noise), they will be inadequate fo a logaithmic COS senso opeating ove a high dynamic ange. The nonlinea effect of offset, gain and bias vaiation on FPN equies moe obust cicuits o nonlinea coection. Refeences [] eith Diefendoff, COS Image Sensos Challenge CCDs, Tech. Rep., icodesign Resouces, June 998, icopocesso Repot. [2] Suneta. endis, Sabina E. emeny, Russell C. Gee, Bedabata Pain, Caig O. Stalle, Quiesup im, and Eic R. Fossum, COS Active Pixel Image Sensos fo Highly Integated Imaging Systems, IEEE Jounal of Solid-State Cicuits, vol. 32, no. 2, pp , Feb [3] Tey Zanowski, Tom Vogelsong, and Jeff Zanowski, Inexpensive Image Sensos Challenge CCD Supemacy, Photonics Specta, pp , ay [4] Bat Dieickx, Danny Scheffe, Guy eynants, Wene Ogies, and Jan Vlummens, Random addessable active pixel image sensos, Poceedings of the SPIE, vol. 2950, pp. 2 7, 996, Advanced Focal Plane Aays and Electonic Cameas. [5] Danny Scheffe, Bat Dieickx, and Guy eynants, Random Addessable Active Pixel Image Senso, IEEE Tansactions on Electon Devices, vol. 44, no. 0, pp , Oct [6] Oly Yadid-Pecht, Wide-dynamic-ange sensos, Optical Engineeing, vol. 38, no. 0, pp , Oct [7]. Loose,. eie, and J. Schemmel, COS image senso with logaithmic esponse and self calibating fixed patten noise coection, Poceedings of the SPIE, vol. 340, pp. 7 27, 998, Advanced Focal Plane Aays and Electonic Cameas II. [8] Richad L. Scheaffe and James T. cclave, Pobability and Statistics fo Enginees, Wadswoth Publishing Company, Belmont, 995. [9] Chistophe. Bishop, Neual Netwoks fo Patten Recognition, Oxfod Univesity Pess, Oxfod, 995.