Optmal Schedulng of Capture Tmes n a Multple Capture Imagng System Tng Chen and Abbas El Gamal Informaton Systems Laboratory Department of Electrcal Engneerng Stanford Unversty Stanford, Calforna 9435, USA ABSTRACT Several papers have dscussed the dea of extendng mage sensor dynamc range by capturng several mages durng a normal exposure tme. Most of these papers assume that the mages are captured accordng to a unform or an exponentally ncreasng exposure tme schedule. Even though such schedules can be justfed by certan mplementaton consderatons, there has not been any systematc study of how capture tme schedules should be optmally determned. In ths paper we formulate the multple capture tme schedulng problem when the ncdent llumnaton probablty densty functon (pdf) s completely known as a constraned optmzaton problem. We am to fnd the capture tmes that maxmze the average sgnal SNR. The formulaton leads to a general upper bound on achevable average SNR usng multple capture for any gven llumnaton pdf. For a unform pdf, the average SNR s a concave functon n capture tmes and therefore well-known convex optmzaton technques can be appled to fnd the global optmum. For a general pece-wse unform pdf, the average SNR s not necessarly concave. The cost functon, however, s a Dfference of Convex (D.C.) functon and well-establshed D.C. or global optmzaton technques can be used. Keywords: Sgnal-to-Nose Rato(SNR), Dynamc Range(DR), multple capture, hgh speed magng, mage sensor. INTRODUCTION Hgh frame rate CMOS mage sensors wth non-destructve readout capabltes have been recently demonstrated., As dscussed n many papers, 3,4 ths capablty can be used to enhance the sensor dynamc range. The dea s to capture several mages durng a normal exposure tme shorter ntegraton tme mages capture the brghter areas of the scene whle longer ntegraton tme mages capture the darker areas of the scene. Ths method has been shown to acheve better SNR than other schemes such as logarthmc sensors and well capacty adjustng. 5 One mportant problem n the mplementaton of multple capture that has not receved much attenton s the selecton of the number of captures and ther tme schedule to acheve a desred mage qualty. Several papers 4,5 have assumed exponentally ncreasng capture tme schedules, whle others 6,7 have assumed unformly spaced capture tme schedules. Even though such schedules can be justfed by certan mplementaton consderatons, there has not been any systematc study of how optmal capture tme schedules can be determned. By fndng optmal schedules, one can acheve the mage qualty requrements wth less captures, whch s desrable snce reducng the number of captures reduces the magng system computatonal power, memory, and power consumpton requrements as well as the nose generated by the multple readouts. To determne the capture tme schedule, scene llumnaton nformaton s needed. Such nformaton may not be avalable before takng one or more captures of the scene. Thus n general an onlne schedulng algorthm that determnes the tme for the next capture based on updated nformaton about the scene llumnaton gathered durng prevous captures may be needed. Fndng such optmal onlne schedules appears ntractable. Instead, n ths paper, we nvestgate the tme capture schedulng problem assumng Correspondence: Emal: tngchen@sl.stanford.edu, abbas@sl.stanford.edu; Telephone: 65-75-9696; Fax: 65-73-8473
complete scene llumnaton nformaton. Ths can be vewed as an offlne verson of the problem. Optmal solutons to the offlne schedulng problem can provde gudance for developng onlne schedulng heurstcs as well as performance bounds on such heurstcs. In the followng secton we provde background on the mage sensor pxel model assumed, defne SNR and dynamc range, and brefly dscuss the multple capture method. In Secton 3, we formulate the multple capture tme schedulng problem when the ncdent llumnaton probablty densty functon (pdf) s completely known. We use average pxel sgnal SNR as an objectve functon snce t generally provdes good ndcaton of mage qualty. To smplfy the analyss, we assume that read nose s much smaller than shot nose and thus can be gnored. Wth ths assumpton the smple Last-Sample-Before-Saturaton (LSBS) algorthm for syntheszng hgh dynamc range mage from multple captures s optmal wth respect to SNR. 6 We use ths formulaton to establsh a general upper bound on achevable average SNR usng multple capture for any gven llumnaton pdf. In Secton 4 we show that for a unform pdf, the average SNR s a concave functon n capture tmes and therefore the global optmum can be easly found usng well-known convex optmzaton technques. In Secton 5 we show that for a general pece-wse unform pdf, the average SNR s not necessarly concave. The cost functon, however, s a Dfference of Convex (D.C.) functon and well-establshed D.C. or global optmzaton technques can be used.. BACKGROUND We assume mage sensors operatng n drect ntegraton, e.g., CCDs and CMOS PPS, APS, and DPS. Fgure depcts a smplfed pxel model and the charge Q(t) versus tme t for such sensors. Durng capture, each pxel converts ncdent lght nto photocurrent ph. The photocurrent s ntegrated onto a capactor and the charge Q(T ) (or voltage) s read out at the end of exposure tme T. Dark current dc and addtve nose corrupt the photocharge. The nose can be expressed as the sum of three ndependent components, () shot nose U(T ) N(,q( ph + dc )T ), where q s the electron charge, () readout crcut nose V (T ) (ncludng quantzaton nose) wth zero mean and varance σv, and () reset nose and FPN C wth zero mean and varance σc. Thus the output charge from a pxel can be expressed as { (ph + Q(T )= dc )T + U(T )+V(T)+C, for Q(T ) Q sat Q sat, otherwse, where Q sat s the saturaton charge, also referred to as well capacty. The SNR can be expressed as ( ph T ) SNR( ph )= q( ph + dc )T + σv + σ C for ph max, () where max Q sat /T refers to the maxmum non-saturatng photocurrent. Note that SNR ncreases wth ph, frst at db per decade when reset, FPN and readout nose domnate, then at db per decade when shot nose domnates. SNR also ncreases wth T. Thus t s always preferable to have the longest possble exposure tme. However, saturaton and moton mpose practcal upper bounds on exposure tme. Sensor dynamc range 8 s defned as the rato of the maxmum non-saturatng photocurrent max to the smallest detectable photocurrent mn = q T q dct + σv + σ C. Dynamc range can be extended by capturng several mages durng exposure tme wthout resettng the photodetector. 3,4 Usng the Last-Sample-Before- Saturaton (LSBS) algorthm 4 dynamc range can be extended at the hgh llumnaton end as llustrated n Fgure (b). Lu et. al., have shown how multple capture can also be used to extend dynamc range at the low llumnaton end usng weghted averagng. Ther method reduces to the LSBS algorthm when only shot nose s present. 6 3. PROBLEM FORMULATION We assume complete knowledge of the scene nduced photocurrent pdf and seek to fnd the capture tme schedule {t,t,..., t N } for N captures that maxmzes the average SNR wth respect to the gven pdf f I () Ths s equvalent to knowng the scene llumnaton pdf for a known sensor response.
V dd Q(t) Reset Q(t) Q sat Hgh lght ph + dc C Low lght τ τ 3τ 4τ T t (a) (b) Fgure. (a) Photodode pxel model, and (b) Photocharge Q(t) vs. Tme t under two dfferent llumnatons. Assumng multple capture at unform capture tmes τ,τ,...,t and usng the LSBS algorthm, the sample at T s used for the low llumnaton case, whle the sample at 3τ s used for the hgh llumnaton case. (see Fgure ). We assume that the pdf s zero outsde a fnte length nterval ( mn, max ). For smplcty we gnore all nose terms except for shot nose due to photocurrent. Let k be the maxmum non-saturatng photocurrent for capture tme t k, k N. Thus t k = Q sat, k and determnng {t,t,..., t N } s equvalent to determnng the set of currents {,,..., N }. The SNR as a functon of photocurrent s now gven by SNR() = Q sat q k for k+ < k and k N. To avod saturaton we assume that = max. Under these assumptons, the capture tme schedulng problem s as follows: Gven f I () and N, fnd {,..., N } that maxmzes the average SNR E (SNR(,..., N )) = Q sat q N k= k subject to: mn = N+ < N <...< k <...< < = max <. k+ k f I () d, () Upper bound: Note that snce we are usng the LSBS algorthm, SNR() Qsat q and thus for any N, max E (SNR(,,..., N )) Q sat q. Ths provdes a general upper bound on the maxmum achevable average SNR usng multple capture. Now, for a sngle capture wth capture tme correspondng to max, the average SNR s gven by E (SNR SC )= Q sat q max mn f I () d = Q sate(i), max q max where E(I) s the expectaton (or average) of the photocurrent for gven pdf f I (). Thus for a gven f I (), multple capture can ncrease average SNR by no more than a factor of max /E(I).
f I () t t N t 5 t 4 t 3 t t mn N 5 4 Fgure. Photocurrent pdf showng capture tmes and correspondng maxmum non-saturatng photocurrents. Note that the postons of capture tmes are not drawn n scale. 3 max 4. OPTIMAL SCHEDULING FOR UNIFORM PDFS In ths secton we show how our schedulng problem can be optmally solved when the photocurrent pdf s unform. In ths case, the schedulng problem becomes: Gven a unform photocurrent llumnaton pdf over the nterval ( mn, max ) and N, fnd {,..., N } that maxmzes the average SNR E (SNR(,..., N )) = Q sat q( max mn ) subject to: mn = N+ < N <...< k <...< < = max <. N ( k k+ ), (3) k Note that for k N, the functon ( k k+ k ) s concave n the two varables k and k+ (whch can be readly verfed by showng that the Hessan matrx s negatve sem-defnte). Snce the sum of concave functons s concave, the average SNR s a concave functon n {,..., N }. Thus the schedulng problem reduces to a convex optmzaton problem wth lnear constrants, whch can be optmally solved usng well known convex optmzaton technques such as gradent/sub-gradent based methods. Table 4 provdes examples of optmal schedules for to captures assumng unform photocurrent pdf over (, ]. Note that the optmal capture tmes are qute dfferent from the commonly assumed unform or exponentally ncreasng tme schedules. Fgure 3 compares the optmal average SNR to the average SNR acheved by unform and exponentally ncreasng schedules. To make the comparson far, we assumed the same maxmum exposure tme for all schedules. Note that usng our optmal schedulng algorthm, wth only captures, the E(SNR) s wthn 4% of the upper bound. Ths performance cannot be acheved wth the exponentally ncreasng schedule and requres over captures to acheve usng the unform schedule. 5. SCHEDULING FOR PIECE-WISE UNIFORM PDFS In the real world, not too many scenes exhbt unform llumnaton statstcs. On the other hand, the optmzaton problem for general pdfs appears to be qute ntractable. In ths secton we nvestgate the schedulng problem for pece-wse unform llumnaton pdfs. Snce any pdf can be approxmated by a pecewse unform pdf, solutons for pece-wse unform pdfs can provde good approxmatons to solutons of the general problem. Such approxmatons are llustrated n Fgures 4 and 5. The emprcal llumnaton pdf of the scene n Fgure 4 has two non-zero regons correspondng to drect llumnaton and the dark shadow regons, and can be reasonably approxmated by a segment pece-wse unform pdf. The emprcal pdf of the scene n Fgure 5, whch contans large regons of low llumnaton, some moderate llumnaton regons, k=
Capture Scheme t t t 3 Optmal Exposure Tmes (t k /t ) t 4 t 5 t 6 t 7 t 8 t 9 t Captures 3 Captures.6 3. 4 Captures.44.3 4.6 5 Captures.35.94 3. 6. 6 Captures.9.74.5 4 8 7 Captures.5.6.7 3.3 5 8 Captures..5.97.65 3.8 6..9 9 Captures..46.8.35 3.7 4.55 7.9 4.57 Captures.8.4.7.4.76 3.73 5.36 8.58 7.6 Table. Optmal capture tme schedules for a unform pdf over nterval (, ]. Upper bound.8 Optmal Unform E (SNR).6 Exponental.4. fi().5.5 4 6 8 4 6 8 Number of Captures Fgure 3. Performance comparson of optmal schedule, unform schedule, and exponental (wth exponent = ) schedule. E (SNR) s normalzed wth respect to the sngle capture case wth = max.
and small very hgh llumnaton regons s approxmated by a 3 segment pece-wse unform pdf. Of course better approxmatons of the emprcal pdfs can be obtaned usng more segments, but as we shall see, solvng the schedulng problem becomes more complex as the number of segments ncreases. x 4 True mage ntensty hstogram.5.5 5 5 5 Approxmated pece-wse unform pdf fi().5.5 mn max mn max..4.6.8. (normalzed to max ) Fgure 4. An mage wth approxmated two-segment pece-wse unform pdf 3 x 4 True mage ntensty hstogram 5 5 5 Approxmated pece-wse unform pdf fi() mn 4 max mn max mn max..4.6.8 (normalzed to max ) Fgure 5. An mage wth approxmated three-segment pece-wse unform pdf
Frst consder the schedulng problem for a two-segment pece-wse unform pdf. We assume that the pdf s unform over the ntervals ( mn, max ), and ( mn, max ). Clearly, n ths case, no capture should be assgned to the nterval ( max, mn ), snce one can always do better by movng such a capture to mn. Now, assumng that k out of the N captures are assgned to segment ( mn, max ), the schedulng problem becomes: Gven a -segment pece-wse unform pdf wth k captures assgned to nterval ( mn, max ) and N k captures to nterval ( mn, max ), fnd {,..., N } that maxmzes the average SNR E (SNR(,..., N )) = Q k sat c ( j j+ )+c ( k mn max N k+ )+c + c ( j j+ ), q j= j k k j j=k+ (4) where the constants c and c account for the dfference n the pdf values of the two segments, subject to: mn = N+ < N <...< k+ < max mn k <...< < = max <. The optmal soluton to the general -segment pece-wse unform pdf schedulng problem can thus be found by solvng the above problem for each k and selectng the soluton that maxmzed the average SNR. Smple nvestgaton of the above equaton shows that E (SNR(,..., N )) s concave n all the varables except k. Certan condtons such as c mn c max can guarantee concavty n k as well, but n general the average SNR s not a concave functon. A closer look at equaton (4), however, reveals that E (SNR(,..., N )) s a Dfference of Convex (D.C.) functon, 9, snce all terms nvolvng k n equaton (4) are concave functons of k except for c max/ k, whch s convex. Ths allows us to apply well-establshed D.C. optmzaton technques (e.g., see 9, ). It should be ponted out, however, that these D.C. optmzaton technques are not guaranteed to fnd the global optmal. In general, t can be shown that average SNR s a D.C. functon for any M-segment pece-wse unform pdf wth a prescrbed assgnment of the number of captures to the M segments. Thus to numercally solve the schedulng problem wth M-segment pece-wse unform pdf, one can solve the problem for each assgnment of captures usng D.C. optmzaton, then choose the assgnment and correspondng optmal schedule that maxmzes average SNR. One partcularly smple yet powerful optmzaton technque that we have expermented wth s Sequental Quadratc Programmng (SQP) wth multple randomly generated ntal condtons. Fgures 6(a) and (b) compare the soluton usng SQP wth random ntal condtons to the unform schedule and the exponentally ncreasng schedule for the two pece-wse unform pdfs of Fgures 4 and 5. Due to the smple nature of our optmzaton problem, we were able to use brute-force search to fnd the globally optmal solutons, whch turned out to be dentcal to the solutons usng SQP. Note that unlke other examples, n the 3-segment example, the exponental schedule ntally outperforms the unform schedule. The reason s that wth few captures, the exponental assgns more captures to the large low and medum llumnaton regons than the unform. 6. DISCUSSION The paper presented the frst systematc study of optmal selecton of capture tmes n a multple capture magng system. Prevous papers on multple capture have assumed unform or exponentally ncreasng capture tme schedules justfed by certan practcal mplementaton consderatons. It s advantageous n terms of system computatonal power, memory, power consumpton, and nose to employ the least number of captures requred to acheve a desred dynamc range and SNR. To do so, one must carefully select the capture tme schedule to optmally capture the scene llumnaton nformaton. To develop understandng of the schedulng problem and as a frst step towards developng onlne algorthms, we formulated the offlne schedulng problem,.e., assumng complete pror knowledge of scene llumnaton pdf, as an optmzaton problem where average SNR s maxmzed for a gven number of captures. Ignorng read nose and FPN and usng the LSBS algorthm, our formulaton leads to a general upper bound on the average SNR for any llumnaton pdf. For a unform llumnaton pdf, we showed that the average SNR s a concave functon n capture
E (SNR).8.6.4. fi() Upper bound Optmal Unform Exponental 4 6 8 4 6 8 Number of Captures E (SNR) 5 4.5 4 3.5 3.5.5 fi() Upper bound Optmal Unform Exponental 4 4 6 8 4 6 8 Number of Captures (a) (b) Fgure 6. Performance comparson of the optmal schedule, unform schedule, and exponental (exponent = ) schedule for the scenes n Fgures 4 and 5. E (SNR) s normalzed wth respect to the sngle capture case wth = max. tmes and therefore the global optmum can be found usng well-known convex optmzaton technques. For a general pece-wse unform llumnaton pdf, the average SNR s not necessarly concave. Average SNR s, however, a D.C. functon and can be solved usng well-establshed D.C. or global optmzaton technques. The offlne schedulng algorthms we dscussed can be drectly appled n stuatons where enough nformaton about scene llumnaton s known n advance. It s not unusual to assume the avalablty of such pror nformaton. For example all auto-exposure algorthms used n practce, assume the avalablty of certan scene llumnaton statstcs. Our results can also be used to develop heurstc onlne algorthms that can perform better than the offlne algorthm wth only partal nformaton of scene llumnaton statstcs. In stuatons where read nose and FPN s too hgh to neglect, our results may not be completely satsfactory, snce the LSBS algorthm s no longer optmal and dynamc range can also be extended at the low llumnaton end as shown by Lu et. al.. 6 Our upper bound on average SNR of Q sat /q stll holds, however. ACKNOWLEDGEMENTS The work reported n ths paper was partally supported by the Programmable Dgtal Camera (PDC) Project by HP, Aglent, Kodak and Canon. The authors would lke to thank the members of the PDC project for suggestons that greatly mproved the presentaton of the paper. REFERENCES. S. Klenfelder, S. Lm, X. Lu, and A. El Gamal, A, Frame/s.8um CMOS Dgtal Pxel Sensor wth Pxel-Level Memory, n ISSCC Dgest of Techncal Papers, pp. 88 89, (San Francsco, CA), February.. D. Handoko, S. Kawahto, Y. Todokoro, and A. Matsuzawa, A CMOS Image Sensor wth Non- Destructve Intermedate Readout Mode for Adaptve Iteratve Search Moton Vector Estmaton, n IEEE Workshop on CCD and Advanced Image Sensors, pp. 5 55, (Lake Tahoe, CA), June. 3. O. Yadd-Pecht and E. Fossum, Wde ntrascene dynamc range CMOS APS usng dual samplng, IEEE Trans. on Electron Devces 44(), pp. 7 73, 997. 4. D. Yang, A. El Gamal, B. Fowler, and H. Tan, A 64x5 CMOS mage sensor wth ultra-wde dynamc range floatng-pont pxel level ADC, IEEE J. Sold-State Crcuts 34(), pp. 8 834, 999.
5. D. Yang and A. El Gamal, Comparatve Analyss of SNR for Image Sensors wth Enhanced Dynamc Range, n Proceedngs of SPIE, vol. 3649, pp. 97, (San Jose, CA), January 999. 6. X. Lu and A. El Gamal, Photocurrent Estmaton from Multple Non-destructve Samples n a CMOS Image Sensor, n Proceedngs of SPIE, vol. 436, pp. 45 458, (San Jose, CA), January. 7. D. Handoko, S. Kawahto, Y. Takokoro, M. Kumahara, and A. Matsuzawa, A CMOS Image Sensor for Focal-plane Low-power Moton Vector Estmaton, n Symposum of VLSI Crcuts, pp. 8 9, June. 8. A. El Gamal, EE39b Classnotes: Introducton to Image Sensors and Dgtal Cameras, http://www.stanford.edu/class/ee39b, Stanford, Calforna,. 9. R. Horst and H. Tuy, Global optmzaton: determnstc approaches, Sprnger, New York, 996.. R. Horst, P. M. Pardalos, and N. V. Thoa, Introducton to global optmzaton, Kluwer Academc, Boston, Massachusetts,.