HVS-Aware Dynamic Backlight Scaling in TFT LCD s

Transcription

1 HVS-Awre Dynmic Bcklight Scling in TFT LCD s Ali Irnli, Wonbok Lee, nd Mssoud Pedrm Dept. of Electricl Engineering University of Southern Cliforni Los Angeles CA Abstrct: Liquid Crystl Displys hve ppered in pplictions rnging from medicl equipment to utomobiles, gs pumps, Lptops nd hndheld portble computers. These disply components present cscded energy ttenutor to the bttery of hndheld device which is responsible for bout hlf of energy drin t mximum disply intensity. As such, the disply components become the min focus of every effort for mximiztion of embedded system s bttery life-time. This pper proposes n pproch for pixel trnsformtion of the displyed imge to increse the potentil energy sving of the bcklight scling method. The proposed pproch tkes dvntge of humn visul system (HVS) chrcteristics nd tries to minimize distortion between the perceived brightness vlues of the individul pixels in the originl imge nd those of the bcklight-scled imge. This is in contrst to previous bcklight scling pproches which simply mtch the luminnce vlues of the individul pixels in the originl nd bcklight-scled imges. Furthermore, this pper proposes temporlly-wre bcklight scling technique for video strems. The gol is to mximize energy sving in the disply system by mens of dynmic bcklight dimming subject to video distortion tolernce. The video distortion comprises of (1) n intr-frme (sptil) distortion component due to frme-sensitive bcklight scling nd trnsmittnce function tuning nd (2) n interfrme (temporl) distortion component due to lrge-step bcklight dimming cross frmes modulted by the psychophysicl chrcteristics of the humn visul system. The proposed bcklight scling technique is cpble of efficiently computing the flickering effect online nd subsequently using mesure of the temporl distortion to ppropritely djust the slck on the intr-frme sptil distortion, thereby, chieving good blnce between the two sources of distortion while mximizing the bcklight dimming-driven energy sving in the disply system nd meeting n overll video qulity figure of merit. The proposed dynmic bcklight scling pproch is menble to highly efficient hrdwre reliztion nd hs been implemented on the Apollo Testbed II. Actul current mesurements demonstrte the effectiveness of proposed technique compred to the previous bcklight dimming techniques, which hve ignored the temporl distortion effect. 1

2 I. INTRODUCTION As the portble electronic devices become more intertwined with everydy life of people, it becomes necessry to put more functionlity into these devices, increse their performnce, nd limit their energy consumption. These devices, which re getting smller nd lighter, re often powered by rechrgeble btteries. Unfortuntely, the bttery cpcities re incresing t much slower pce thn the overll power dissiption of these devices. So it is criticl to develop power-wre design methodologies nd technique to bring the power dissiption growth of such devices in line with the bttery cpcity increse. Mjor sources of power dissiption in portble electronic device re mny nd vry s function of the device functionlity nd performnce specifiction. In relity, mny of these devices re equipped with Liquid Crystl Disply (LCD), which tends to ccount for significnt portion of the totl system power. For exmple, in the SmrtBdge system, the disply subsystem consumes 28.6% nd 50% of the totl power in the ctive/idle nd stndby modes, respectively [1] The dominnt bcklighting technology for LCD s is the Cold Cthode Fluorescent Lmp (CCFL), which uses low-voltge DC to high-voltge AC converter s the driver. This driver consumes the lrgest mount of power in the disply subsystem. The LCD displys currently vilble require two power sources, bcklight supply nd contrst supply. The disply bcklight is the single lrgest power consumer in typicl disply component. Bttery energy is lost in the electric-to-electric conversion to high voltge AC for driving the CCFL; where conversion efficiencies exceeding 90% re possible. Although the CCFL is the most efficient electric-tolight converter vilble tody, it hs losses exceeding 80%. Additionlly, the opticl trnsmission efficiency of present displys is under 50% for monochrome with color types much lower. Unfortuntely, s for other system resources, one cnnot tckle this energy hungry component with some form of power shutdown. This is due to the fct tht LCD subsystem must be continuously refreshed nd cnnot be turned off or put to sleep without significnt penlty in performnce nd Qulity of Service. There re two min clsses of techniques for lowering the power consumption of LCD subsystem. The first clss of techniques is focused on the digitl/nlog interfce between the grphics controller nd the LCD controller. These techniques try to minimize the energy consumption by tking dvntge of different encoding schemes to minimize the switching ctivity of the electricl bus. For instnce, reference [2] uses the sptil loclity of the video dt to reduce the number of trnsition on the DVI bus reducing its energy consumption by 75%. More recently, reference [3] hs extended the previous work by using the limited intr-word trnsition codes insted of single intr-word trnsition codes used in [2] nd including the DC blncing conditions in the coding scheme to chieve more thn 60% energy sving, on verge compred to the bsic trnsmission protocol. The second clss of techniques is focused on the video controller nd the bcklight to lower the energy consumption of the disply system. The key ide follows from the fct tht eye s perception of the light, which is emitted from the LCD pnel, is function of two prmeters, 1) intensity of the bcklight nd 2) trnsmittnce of the LCD pnel. Thus by crefully djusting these two prmeters one cn chieve the sme perception in humn eyes t different vlues of the bcklight intensity nd the LCD trnsmittnce. Now becuse the vrition in power consumption of the bcklight lmp for different output luminnce vlues is orders of mgnitude lrger thn power consumption of the LCD pnel for different pixel vlues, one cn sve energy by simply dimming the bcklight nd incresing the LCD trnsmittnce to compenste for the loss of bcklight luminnce. In [4], Chng et l. proposed Dynmic bcklight Luminnce Scling technique (DLS) to reduce the energy consumption of the LCD displys. This pproch suffers from two min drwbcks, ) it mnipultes every pixel on the screen one-by-one limiting the ppliction of this pproch to still imges or low-frme-rte videos; b) it chieves energy sving t the cost of loss in visul informtion. Reference [5] improved this simple pproch by eliminting the pixel-by-pixel trnsformtion of the displyed imge through minor hrdwre modifictions to the built-in LCD reference driver. These modifictions could implement ny single-bnd gryscle spreding function to djust the brightness nd contrst of the displyed imge, extending the pplicbility of the pproch to streming pplictions. Reference [6], 2

3 proposed severl pproches for power mngement of the disply sub-system bsed on vrible refresh rte, liquid crystl orienttion shift, nd bcklight dimming. However, in the proposed bcklight scling technique no pixel vlue trnsformtion method is used to compenste the loss of imge brightness. Reference [7] proposes disply power mngement method bsed on the user s ttention. In this method the disply is turn off or put to sleep mode whenever cmer bsed imge processing controller detects tht the user is not looking t the screen. This method suffers from lrge overhed in power nd performnce for clcultion nd detection of user ttention to the screen. Reference [8] proposed method for imge trnsformtion, whereby the dynmic rnge of the originl imge is reduced such tht the incurred imge distortion is no more thn pre-specified vlue. The uthors relied on very ccurte still imge qulity mesure. This work improved the previous methods in two key spects. Firstly, it presented globl histogrm equliztion lgorithm which enbled preservtion of most of the visul informtion in spite of imge trnsformtion, nd secondly, it described simple, yet effective, modifiction of the rchitecture of built-in LCD reference driver in order to produce ny piecewise liner imge trnsformtion function. The method however hd two disdvntges, ) the distortion chrcteriztion curve ws dependent on the displyed imge; b) the equliztion lgorithm required histogrm informtion of the displyed imge to clculte the imge trnsformtion function. Finlly, reference [9] presented bcklight scling technique, which is bsed on tone reproduction opertor. This opertor mps the originl imge χ to trnsformed imge χ` such tht the perceived brightness of the imge is preserved while its dynmic rnge is reduced. The opertor ws clculted without ny informtion bout individul pixels of the displyed imge or the imge histogrm. The proposed method ws the first to tke dvntge of some spects of the humn visul system to mximize power svings through dynmic bcklight dimming (see discussions in 4.1 for detils.) In ddition, this method is menble to highly efficient hrdwre reliztion becuse it does not require informtion bout the histogrm of the displyed imge. Although the forementioned techniques re effective bcklight scling pproches, they ll suffer from common shortcoming, i.e., temporl blindness. All previously proposed pproches for bcklight scling hve overlooked the temporl response of humn visul system; more specificlly they pply the bcklight scling scheme to ech frme of video sequence individully nd without considering flickering effects, which my result s consequence of frequent nd brupt chnges in the bcklight intensity. In the reminder of this pper, we will refer to ll of the prior work techniques for video s Frme-sensitive Bcklight Scling (FBS) techniques. In contrst, this pper proposes bcklight scling scheme which is sensitive to the sptio-temporl informtion of video sequence. The key ide is to decompose the mximum llowed distortion between the originl video nd the bcklight scled one into two components, 1) the sptil distortion which is the intrfrme luminnce distortion between the respective frmes of the originl nd bcklight scled video sequences, nd 2) the temporl distortion which is the inter-frme distortion due to (lrge-scle) chnges of luminnce over time when compring the originl nd bcklight scled video sequences. In this pper, we tke dvntge of n nlyticl model of the Criticl Fusion Frequency nd the temporl informtion bout the verge luminnce of ech frme in video sequence to cpture the effect of temporl distortions incurred by bcklight scling scheme. Next, we use this informtion to limit the mximum llowed sptil distortion tht the FBS scheme cn crete, nd therefore, limit the overll mximum video distortion below certin user specific limit. In the following, the bsic bckground on the humn visul system nd its temporl response, principles of photogrphic tone mpping, nd finlly the TFT LCD rchitecture nd prior work in dynmic bcklight scling will be discussed. Next, in section III temporlly-wre bcklight scling nd dynmic tone mpping pproch will be explined. Sections IV nd V will provide the supporting experimentl results nd conclusions of this technique. 3

4 II. BACKGROUND AND PRELIMINARIES II.A Photometric Definitions Although light is form of electromgnetic rdition, mesurement of luminous intensity from light source requires extr informtion bout the reltive sensitivity of the eye to different wvelengths. Photometry is the science of mesuring visible light in units tht re weighted ccording to the sensitivity of the humn eye. The luminous intensity of "white" light source is thus defined by multiplying the wtts emitted t ech wvelength by the efficiency of tht wvelength in exciting the eye, reltive to the efficiency t 555nm. This efficiency fctor is referred to s the V-lmbd curve [10]. The cndel is the luminous intensity per solid ngle, in given direction of source which emits monochromtic rdition t wvelength 555nm nd which hs rdint intensity in tht direction of 1/683 wtt per sterdin [11]. Humns perceive luminnce, which is n pproximte mesure of how bright surfce ppers when one views it from given direction. Luminnce is thus defined s luminous intensity per squre meter nd is mesured in cndel per squre meter. For exmple, the luminnce of sun, cloudy dull dy, typicl office, good street lighting, nd full moon re 5K, 300, 15, 1.2, nd 0.002cd/m 2, respectively. II.B Humn Visul System II.B.1 Sptil Chrcteristics When light reches eye, it hits the photoreceptors on the retin, which then send n electricl signl through neurons to the brin where n imge is formed. The photoreceptors in our retin, nmely rods nd cones, ct s the sensors for the HVS. Rods re extremely sensitive to light nd provide chromtic vision t scotopic levels of illumintion (10-6 to 10 cd/m 2 ), tht is why we cnnot see colors in drk surroundings. Cones (which comprise of three distinct types) re less sensitive, but provide color vision t photopic levels of illumintion (10-2 to 10 8 cd/m 2 ). Note tht both rods nd cones re ctive t light levels between 0.01 nd 10 cd/m 2. This rnge is clled the mesoptic rnge. The incoming light cn hve dynmic rnge of nerly 1:10 14, wheres the neurons cn trnsfer signl with dynmic rnge of only bout 1:10 3. The humn eye cn discern dynmic rnge of bout orders of mgnitude. As result, there is the need for some kind of dpttion mechnism in our vision. This mens tht we first dpt to some (unchnging) luminnce vlue, nd then perceive imges in rther smll dynmic rnge round this luminnce vlue. One of the most importnt chrcteristics tht chnges with different dpttion levels is the Difference Threshold or Just Noticeble Difference (JND.) The JND is the minimum mount by which stimulus intensity must be chnged in order to produce noticeble vrition in sensory experience. Let ΔL nd L denote the JND nd the dpttion luminnce, respectively. Blckwell [12] showed tht the rtio ΔL/L vries s function of the dpttion level, L nd thus, estblished the reltionship between L nd ΔL to be Δ LL ( ) = ( L ) Simply stted, Blckwell s eqution sttes tht if there is ptch of luminnce L +ε where ε ΔL on bckground of luminnce L, it will be discernible, but ptch of luminnce L +ε, where ε < ΔL will not be perceptible to the humn eye (1) 4

5 Figure 1. Brightness vs. luminnce chrcteristic of the HVS. Let us now consider the brightness perception. Brightness is the mgnitude of the subjective senstion which is produced by visible light. Although the rdince cn esily be mesured, the brightness cnnot exctly be quntified. Nevertheless, brightness is often pproximted s the logrithm of the luminnce, or the luminnce rised to the power of 1/2 to 1/3 depending on the context. Studies hve shown tht the brightness-luminnce reltion depends on the dpttion level to the mbient light. In prticulr, Stevens et l. [13] devised the brils units to mesure the subjective vlue of brightness. According to Stevens, one bril equls the senstion of brightness tht is induced in fully drk-dpted eye by brief exposure to 5- degree solid-ngle white trget of 1 micro-lmbert luminnce. (One lmbert is equl to 3,183 cd/m 2.) Let B denote brightness in brils, L the originl luminnce vlue in lmberts, nd L denote the dpttion luminnce of the eye. Then, where L B = λ L σ = 0.4 log ( ) L λ = 10 L Typicl perceived brightness chrcteristic curves re shown in Figure 1. The slope of ech curve represents the humn contrst sensitivity tht is the sensitivity of the HVS brightness perception to the chnges in the luminnce. Note tht s L is lowered, the humn contrst sensitivity decreses. Notice lso tht the HVS exhibits higher sensitivity to chnges in luminnce in the drker regions of n imge. Two imges with different luminnce vlues cn result in the sme brightness vlues, nd cn pper to the HVS s being identicl. Actully, ccording to eqution (2-) we re very poor judges of n bsolute luminnce vlue; ll tht we cn judge is the rtio of luminnces, i.e. the brightness II.B.2 Temporl Chrcteristics Studies of the dynmics of light perception cn be divided into two ctegories bsed on the stimultion method utilized for the HVS chrcteriztion. The first ctegory of techniques is bsed on periodic stimuli, which try to mesure the impulse response of HVS. In these experiments, the sensitivity of the HVS is mesured when brief test light is presented to the HVS before, during, or fter the presenttion of much stronger bckground light. The results show tht the sensitivity of the HVS is t its mximum when the test light is presented during the trnsition of the bckground light [14]. Although these experiments re very effective for understnding the time domin response of the HVS, they hve limited power in quntifying the distortion due to dynmic bcklight scling, which is perceived by the HVS during video plybck. This is becuse bcklight scling is pplied to every video frme t rte tht is too fst for the HVS to settle to some stedy stte bckground brightness. σ The second ctegory of techniques tries to mesure the Criticl Fusion Frequency (CFF) of the HVS t (2-) (2-b) 5

6 vrious mplitude sensitivity vlues (AS vlues.) These concepts re closely relted to the well-known notion of Temporl Contrst Sensitivity Function, [15]-[17].The CFF is the minimum frequency, f*, bove which the observer cnnot detect ny flickering effect when series of light flshes t tht frequency is presented to him/her, i.e., for frequencies lower thn f*, the HVS will notice the flickering. The AS t frequency f* is the rtio of the minimum required mplitude of the flsh light t frequency f* to the verge mbient luminnce such tht the flickering effect is perceived [14]. Generlly, s the mplitude sensitivity increses, the flickering becomes esier to perceive for series of flshes with fixed frequency. These metrics re suitble for quntifying distortions cused by the bcklight scling since they model the flickering effect of periodic bcklight scling. Figure 2. Temporl model of HVS In this pper, we dopt computtionl temporl response model of the HVS due to Weignd et l. [18]. Figure 2 depicts the schemtic representtion of this model. This model cn be used to determine the AS threshold of n observer when presented with vrying light intensities. Note tht the input to this model is sclr vlue representing the luminnce of the viewing re, wheres the output is the intensity of perceived luminnce. If the difference between the DC vlue nd the mplitude of given frequency, f*, in the output of the model exceeds fixed predefined threshold δ, then the flickering is perceived by the HVS t frequency f*. For exmple, in cse of bcklight scling, the input to the model is the verge luminnce of ech frme of video sequence, wheres the output of the model is the perceived intensity of the light. Bsiclly, this model comprises of cscde of two prmetric second order low pss filters nd dynmic gin control, followed by the non-liner sttic chrcteristic of HVS which trnsforms the luminnce vlues to perceived brightness vlues. Prmeters of the two qudrtic filters re controlled by first order low pss filter with cutoff frequency ner 0.1Hz. Figure 3 shows typicl frequency domin trnsfer function of this model when simple sinusoidl input with constnt mplitude nd vrying DC vlue is presented to it. Note tht for fixed DC vlue, i.e. fixed bckground luminnce, s we increse the frequency of the flickering light the required mplitude for the flickering to be perceived is lmost constnt for frequencies below 10Hz nd then increses exponentilly fter this frequency. Moreover, for given frequency of flshing light by incresing the bckground luminnce the minimum mplitude of the flshing light for which the flickering is perceived decreses. As described in [18], the control low pss filter is first order filter with following trnsfer function, 1 HC LP( f) = j2 π (1.59 f) + 1 wheres the cscde of two qudrtic low pss filters hs the generl trnsfer function of, (3-) 2 2 ( ) f 1 f 2 HLP f = f1 + jd1f1f f f2 + jd2f2f f with prmeters given by, (3-b) 6

7 1 2 1 ( r t ) 0.5 zc( t) = 1/ 1 + ( ) / c f ( t) = z ( t) f ( t) = z ( t) d ( t) = z ( t) d2( t) = zc ( t) Finlly, the controllble gin is given by, c c c (3-c) g= 2.2 ( rc( t) ) ( rc( t) ) (3-d) where r c (t) is the control signl t the output of the control low pss filter given in eqution (3-) (cf. Figure 2). It is worth noting tht the models of HVS re bsed on subjective experiments, nd therefore, their ccurcy vries from one individul to next. The key ide of this pper is to underscore the significnce of considering such humn perceptive models during the optimiztion process for bcklight scling. The commercil implementtion of such system cn hve multiple HVS models for users to select from. In section III we will tke dvntge of the forementioned model nd introduce metric for mesuring the distortion of originl nd bcklight scled video sequences in terms of incurred flickering Figure 3. Smple mplitude sensitivity of HVS for sinusoidl input with vrying DC vlue II.C Tone reproduction A clssic photogrphic tsk is the mpping of the potentilly high dynmic rnge of rel world luminnces to the low dynmic rnge of the photogrphic print. The rnge of light tht people experience in the rel world is vst. However, the rnge of light one cn reproduce on prints spns t best bout two orders of bsolute dynmic rnge. This discrepncy leds to the tone reproduction problem: how should one mp mesured/sensed scene luminnces to print luminnces nd produce stisfctory picture? The success of photogrphy hs shown tht it is possible to produce imges with limited dynmic rnge tht convey the ppernce of relistic scenes. This is fundmentlly possible becuse s mentioned in previous section humn eye is sensitive to reltive, rther thn bsolute, luminnce vlues. Consider typicl scene tht poses problem for tone reproduction in photogrphy, room illuminted by window tht looks out on sunlit lndscpe. A humn observer inside the room cn esily see individul objects in the room s well s fetures in the outdoor lndscpe. This is becuse the eye dpts loclly s we scn the different regions of the scene. If we ttempt to photogrph our view, the result is disppointing. Either the window is over exposed nd we cn t see outside, or the interior of the room is underexposed nd looks blck. In 1993, Tumblin et l. [19] introduced this concept to computer grphics community nd proposed 7

8 primitive tone mpping opertor. Since then gret del of work hs been done on the tone reproduction problem. Generlly speking the tone reproduction literture cn be divided into two min ctegories. The first ctegory which uses globl tone mpping opertor ignores the sptil informtion bout the luminnce of the originl scene nd dopts single non-decresing function s its tone mpping opertor. Reference [20] uses model of brightness perception to derive this mpping opertor. Reference [21] relies on globl multiplier to mintin the visibility threshold. In reference [22] nother globl opertor ws proposed bsed on histogrm djustment. This method uses n imge s histogrm to implicitly segment the imge so tht seprte scling fctors cn be used in different luminnce zones. The second ctegory tries to reproduce the visibility of different objects in the scene. This is done through multiple mpping functions which re dopted bsed on locl luminnce informtion of the originl scene. Chiu et l. [23] who used sptilly vrying exposure rmp over the imge ws the first to propose sptilly vrying dynmic rnge reduction opertor. Lter work by Pttnik et l. [24] developed the ultimte still imge opertor bsed on the humn visul system, incorporting color dpttion, locl contrst, nd dynmic rnge. More recently, References [25]-[27] hve proposed more successful tone mpping opertors in seprting the contrst differences tht mtter to vision from those tht do not. The bsic chllenge for sptilly vrying tone mpping opertor is tht it needs to reduce the globl contrst of n imge without ffecting the locl contrst to which humn visul system is sensitive. To ccomplish this, n opertor must segment the high dynmic rnge imge, either explicitly or implicitly, into regions tht humn visul system does not correlte during dynmic rnge reduction. Otherwise, the locl vrying opertors would led into disturbing reverse grdients typiclly seen s hlos round light sources. II.D LCD rchitecture nd Bcklight Scling Figure 4 shows the typicl rchitecture of the digitl LCD subsystem in microelectronic device. There re two min components in this subsystem: ) video controller nd frme buffer memory nd b) LCD controller nd pnel. The imge dt, which is received from the processing unit, is first sved into the frme buffer memory by the video controller nd is subsequently trnsmitted to the LCD controller through n pproprite nlog (e.g., VGA) or digitl (e.g., DVI) interfce.. LCD subsystem rchitecture b. LCD component rchitecture c. TFT cell schemtic Figure 4. TFT-LCD screen The LCD controller receives the video dt nd genertes proper gryscle i.e., trnsmissivity of the pnel for ech pixel bsed on its pixel vlue. All of the pixels on trnsmissive LCD pnel re illuminted from behind by the bcklight. To the observer, displyed pixel looks bright if its trnsmittnce is high (i.e., it is in the 'on' stte), mening it psses the bcklight. On the other hnd, displyed pixel looks drk if its trnsmittnce is low (i.e., it is in the 'off' stte), mening tht it blocks the bcklight. For color LCD s, different filters re used to generte shdes of three min colors (i.e. red, blue, nd green), nd then color pixels re generted by mixing three sub-pixels together to produce different colors. Figure 4b depicts the LCD controller in more detil. The dt received from the video bus is used to infer row of pixels timing informtion nd respective gryscle levels. Then, this timing informtion is used to select the pproprite row in the LCD mtrix. Next, the pixel vlues re converted to the corresponding voltge levels to drive the thin-film-trnsistors (TFT s) on different columns of the selected row. The bcklight bulb is powered with the id of DC-AC converter, to provide the required illumintion 8

9 of the LCD mtrix. Ech pixel hs n individul liquid crystl cell, TFT, nd storge cpcitor (cf. Figure 4c). The electricl field of the cpcitor controls the trnsmittnce of the liquid crystl cell. The cpcitor is chrged nd dischrged by the TFT. The gte electrode of the TFT controls the timing for chrging/dischrging of the cpcitor when the pixel is scnned (or ddressed) by the trcer for refreshing its content. The (drin-) source electrode of the TFT controls the mount of chrge. The gte electrodes nd source electrodes of ll TFT s re driven by set of gte drivers nd source drivers, respectively. A single gte driver (clled gte bus line) drives ll gte electrodes of the pixels on the sme row. The gte electrodes re enbled t the sme time the row is trced. A single source driver (clled source bus line) drives ll source electrodes of the pixels on the sme column. The source driver supplies the desired voltge level (clled gryscle voltge) ccording to the pixel vlue. In other words, idelly, the pixel vlue trnsmittnce, t(x), is liner function of the gryscle voltge v(x), which is in turn liner function of the pixel vlue X. The trnsfer function of source driver which mps different pixel vlues, X, into different voltge levels, v(x) is clled the gryscle-voltge function. If there re 256 gryscles, then the source driver must be ble to supply 256 different gryscle voltge levels. For the source driver to provide wide rnge of gryscles, number of reference voltges re required. The source driver mixes different reference voltges to obtin the desired gryscle voltges. Typiclly, these different reference voltges re fixed nd designed s voltge divider. Mthemticlly speking, in trnsmissive TFT-LCD monitor, for pixel with vlue X, the luminnce I(X) of the pixel 1 is: I ( X) = b. t( X) (4) where t(x) is the trnsmissivity of the TFT-LCD cell for pixel vlue X, nd b [0,1] is the (normlized) bcklight illumintion fctor with b=1 representing the mximum bcklight illumintion nd b=0 representing no bcklight. Note tht t(x) is liner mpping from [0,255] domin to [0,1] rnge. In bcklight scled TFT-LCD, b is scled down nd ccordingly t(x) is incresed to chieve the sme imge luminnce. Let X nd X'= Φ(X, β) denote the originl nd the trnsformed imge dt, respectively. Moreover, let D(X, X') nd P(X', β) denote the distortion of the imges X nd X' nd the power consumption of the LCDsubsystem while displying imge X' with bcklight scling fctor, β. Dynmic Bcklight Scling (DBS) Problem: Given the originl imge X nd the mximum tolerble imge distortion D mx, find the bcklight scling fctor β nd the corresponding pixel trnsformtion function X'=Φ(X, β) such tht P(X', β) is minimized nd D(X, X') D mx. The generl form of DBS problem s stted bove is difficult to solve due to the complexity of the distortion function, D, nd lso the non-liner function minimiztion step tht is required to determine Φ(X, β). From now on, in this pper we will use Φ(X, β) to denote the trnsformed imge, nd Φ(x,β) to denote single trnsformed pixel in the imge X. Reference [4] describes two bcklight luminnce dimming techniques. Let x denote the normlized pixel vlue, i.e., ssuming n 8-bit color depth, x=x/255. The uthors of [4] scle the bcklight luminnce by fctor of β while incresing the pixel vlues from x to Φ(x, β) by two mechnisms. Clerly, Φ(x, β)=x denotes the identity pixel trnsformtion function (cf. Figure 5.) The bcklight luminnce dimming with brightness compenstion technique uses the following pixel trnsformtion function (cf. Figure 5b): Φ ( x, β) = min(1, x+ 1 β) (5-) wheres the bcklight luminnce dimming with contrst enhncement technique uses this trnsformtion function (cf. Figure 5c): 1 Illuminnce cn be used to chrcterize the luminous flux emitted from surfce. Most photogrphic light meters mesure the illuminnce. In terms of visul perception, we perceive luminnce. It is n pproximte mesure of how bright surfce ppers when we view it from given direction. Luminnce is mesured in lumens per squre meter per sterdin. The mximum brightness of CRT or LCD monitor is described by luminnce in its specifiction. 9

10 x Φ ( x, β ) = min(1, ) (5-b) β In these schemes, to clculte the distortion rte, n imge histogrm estimtor is required for clculting the sttistics of the input imge. Note tht the imge histogrm simply denotes the mrginl distribution function of the imge pixel vlues. Reference [4] uses the number of sturted pixel vlues due to forementioned trnsformtions s their mesure of imge distortion. Reference [5] proposes different pproch in which the pixel vlues in both drk nd bright regions of the imge re used to enble further dimming of the bcklight. The key ide is to first truncte the imge histogrm on both ends to obtin smller dynmic rnge for the imge pixel vlues nd then to spred out the pixel vlues in this rnge (by pplying n ffine trnsformtion) so s to enble more ggressive bcklight dimming while mintining the contrst fidelity of the imge. The pixel trnsformtion function is given s (cf. Figure 5d): 0, 0 x g l 1 1 g Φ ( x, β ) = cx+ d, g x g c ; d l l u = = = gu g β gu g l l 1, gu x 1 where (g l,0) nd (g u,1) re the points where Φ(x, β)=cx+d intersects Φ(x, β)=0 nd Φ(x, β)=1, respectively. In this technique the imge distortion is mesured bsed on the number of pixel vlues tht re preserved fter the ppliction of the trnsformtion function to the originl imge. The implementtion of this pproch is quite simple nd requires miniml chnges to the built-in Progrmmble LCD Reference Driver (PLRD). Notice tht PLRD llows clss of liner trnsformtions on the bcklight-scled imge, resulting in both brightness scling nd contrst scling. Clerly, brightness nd contrst re the two most importnt properties of ny imge. The previous pproches cnnot fully utilize the power sving potentil of the dynmic bcklight scling scheme, due to the fct tht their mesure of distortion between the originl nd the bcklightscled imge is n overestimtion. This is becuse these pproches simply either minimize the number of sturted pixel vlues [4] or mximize the number of pixel vlues tht re preserved [5]. The min disdvntge of these cost functions is tht they trets the gry scle vlues in drk nd white regions of the imge the sme wy, which is in contrst to the perceived brightness chrcteristic of the HVS s shown in Figure 1. (6). identity b. gryscle shift 10

11 c. gryscle spreding d. single-bnd gryscle spreding Figure 5. Pixel trnsformtion functions The forementioned techniques for bcklight scling hve proven to be quite effective for still imges; however, if one tries to pply these techniques on per-frme bsis, i.e. FBS, for video strems, the output video qulity my significntly be degrded. This is due to two fcts: 1) they overlook very importnt deciding fctor in their optimiztion process, tht is, the humn visul system chrcteristics. All of these pproches rely on the luminnce vlues of pixels of the displyed imge s their optimiztion vribles. However, luminnce vlue of light source is not the sme s its perceived brightness; 2) different frmes in video clip my hve different chrcteristics, nd therefore, dynmic bcklight scling (DBS) pplied in per-frme bsis to video sequence my result in different pixel trnsformtion functions (Φ), nd thus, different bcklight scling fctors (β). The brupt chnges in Φ nd β my result in lrge flickering effect over time, which in turn dversely impcts the overll video qulity. This pper improves the previous techniques in two wys: 1) it proposes bcklight scling technique which is bsed on tone reproduction opertor. This opertor mps the originl imge X to trnsformed imge X' such tht the perceived brightness of the imge is preserved while its dynmic rnge is reduced. This reduction in the dynmic rnge of the imge will further increse the potentil for bcklight scling, nd therefore, mximize the energy sving. Moreover, the proposed opertor cn be clculted without ny informtion bout the individul pixels of the displyed imge or its histogrm, further improving the video frme-rte nd power svings due to elimintion of ny hrdwre/softwre support for imge histogrm genertion; 2) it presents temporlly-wre bcklight scling method in which the inter-frme chnges in Φ nd β re limited such tht the perceived flickering is minimized while the mximum power sving is chieved. III. TEMPORALLY AWARE BACKLIGHT SCALING (TABS) In FBS techniques, the distortion between ech bcklight scled frme nd the originl frme, i.e. the sptil distortion, is upper bounded by user specified mximum llowed vlue (cf. Figure 7). However, this sptil distortion is not sufficient to quntify the overll qulity of bcklight scled video. The second component which is importnt in quntifying the video qulity is the chnges in the luminnce of the bcklight-scled video compred to the originl video, i.e. the temporl distortion. This component of the video distortion cptures the unwnted vritions in overll luminnce of different frmes of bcklight scled video sequence. Let X nd X' denote the originl nd the bcklight scled versions of video sequence with totl number of frmes, N. Then, we define the distortion between two video sequences X nd X' s, D( XX', ) = α D ( X,X' ) + (1 α) D ( X,X' ) spt tmp 2 ( ) ( { } ) 2 ( ) α { } { } = α mx D ( X, X' ) + (1 ) F V ( X) F V ( X' ) F V ( X) i spt i i j j j j j where D spt (X, Y) is the sptil distortion between still imges, X nd Y; V(.) is the perceived brightness when n input is given to the HVS (cf. Figure 2), nd F{.} is the Fourier trnsform opertor. Finlly, α is (6) 11

12 the weighting coefficient. The first term in eqution (6) cptures the sptil distortion between the respective frmes of video sequences X nd X', while the second term is the normlized men squre error between the spectrl power density of the brightness of originl video sequence nd the bcklight scled one. This term, ctully, cptures the temporl distortion of two video sequences (cf. section II.2.B). The video distortion function presented in eqution (6) is hrd to evlute becuse the first term is in time domin while the second term is defined in the frequency domin. To circumvent this problem, eqution (6) cn be simplified using the Prsevl s theorem, which simply sttes tht integrl of squred signl is equl to integrl of its spectrl power density, therefore, ( { } { } ) { } ( ) 2 2 D ( XX', ) = FV( X) FV( X' ) FV( X) tmp j j j j j = = 1 + ( V( X' )) ( V( X )) 2 V( X) V( X' ) ( V( X )) N N N 1 2 ( V( X' j )) 2 V( X0) V( X' 0) N j = ( V ( Xj )) N ( ) 2 2 ( F{ V( X) } ) ( { ( )} ) 2 ( { ( )} ) ( { ( )} ) { ( )} j F V X' V V V j F X F X' j j F X' j j j j j j j j= 0 j j j j j Bsed on this video distortion mesure we propose Temporlly-Awre Bcklight Scling for Video (TABS). Figure 6 shows the block digrm of this pproch. The key ide is to mesure the temporl distortion of bcklight scled video nd then feedbck this informtion to dynmiclly chnge the mximum llowed sptil distortion of the frmes. 2 (7) Figure 6. Temporlly-wre bcklight scling To reduce the sptil distortion between the corresponding frmes of the originl nd bcklight scled video sequences, we propose simple, yet effective pixel trnsformtion function bsed on the sttic chrcteristic of HVS nd tone mpping theory. Let L nd L mx mx denote the mximum luminnce of the originl imge nd the dynmiclly tone-mpped nd bcklight-scled frme, respectively. Moreover, let X t nd X' t denote the pixel vlue informtion of the originl nd bcklight scled frmes t time t. Then, the perceived imge distortion between imges X t nd X' t cn be quntified by function D spt (X t, X' t ). Converse Tone Mpping (CTM) Problem: Given n originl imge X t nd mximum llowble imge distortion D mx, find the tone mpping opertion ψ :[0, L ] [0, L mx mx ] such tht L mx is minimized while D spt (X t, X' t ) D mx (8) where X' ψ ( X ). The forementioned problem is the converse of the tone mpping problem, becuse in the tone 12

13 mpping problem, the gol of optimiztion is to find the mpping opertor Ψ such tht for given mximum disply luminnce, the imge distortion is minimized [19]. In contrst, in the CTM problem, the gol of optimiztion is to find the minimum of mximum luminnce vlue tht gurntees given mximum imge distortion level. Unfortuntely, due to complexity of HVS,, nd therefore the complexity of the imge distortion function, D spt, neither the CTM problem nor the tone mpping problem hve closed form solutions. To solve the CTM problem, this pper proposes heuristic pproch bsed on pixel brightness preservtion. The key ide is to mke sure tht the JND in the bcklight scled imge nd tht in the originl imge re equl. In this wy, the imge perception is preserved, i.e., both imges hve the sme discernible detils. Mthemticlly speking, let L nd L denote the dpttion luminnce for the originl nd the bcklight scled imges. Bsed on eqution (1), the JND for the originl imge is ΔLL ( ) nd the JND for the bcklight scled imge will be Δ L( L ). Therefore, to preserve the discernible detils of the imge, we ought to find tone mpping function,ψ, such tht, Δ L( L ) = ψ ( Δ L( L )). Figure 7. Sptil vs. Temporl distortions As simple solution, one cn ssume vrible scling function where the scling fctor chnges depending upon the locl luminnce vlue. Subsequently, the lighter regions of the imge will be scled more non-linerly thn the drker regions so s to tke dvntge of the decresing humn contrst sensitivity from drk to light regions of the imge (cf. Sec. II.B.1) However, this pproch requires mnipultion of individul pixel vlues, which my be undesirble for rel-time implementtion. Therefore, this pper first doptsψ, to be constnt scling function ψ ( x) = κ x, where κ cn be clculted from eqution (1). Consider two pixels in the originl imge, x 1 nd x 2, which re discernible therefore, x1 x2 = Δ L( L ). If we wnt to preserve the discernible detils of the imge in the trnsformed imge we need to hve ψ( x1) ψ ( x2) = Δ L( L ), substituting the pixel trnsformtion function with ψ ( x) = κ x one cn clculte the ψ s function of L nd L, 13

14 ψ ( x1) ψ( x2) = ΔL( L ) κ ( x1 x2) = ΔL( L ) κ Δ LL ( ) =ΔLL ( ) ΔLL ( ) κ = ΔLL ( ) κ = where L nd L my be pproximted by hlf of the mximum bcklight luminnce before nd fter bcklight scling, i.e., 0.5L nd 0.5L. Plese note tht the dpttion luminnce level, L mx mx, (cf. section II.B.1) is key prmeter in determining the perception of brightness. There re mny techniques vilble for mesuring the mbient light, e.g. by mens of light sensors. However, these techniques re power consuming nd use up resources. In this pper, for our experimentl results, we use simple method for pproximting the mbient light s one hlf of the mximum luminnce of the disply system itself. Our pproch cn be esily modified to consider ny form of dpttion luminnce (mbient light) mesurement or estimtion. Eqution (9) represents the key difference between our pproch nd those reported previously in references [4]-[8]. In those pproches, κ is essentilly set s L mx / L mx in order to preserve the pixel luminnce vlues, wheres in our pproch κ is given by eqution (9). In ddition, to cpture the humn contrst sensitivity (cf. Sec. II.B.1, Figure 1), we will use functionl form for the trnsformtion function, Ψ, which is similr to tht of the humn brightness perception function, i.e. (cf. eqn. 2), ( L ) ( L ) γ ( L, L ) 2.5 X ψ( X ) = κ( L, L ) (10) L where κ ( L, L ) is simply the luminnce intensity djustment fctor s given by eqution (9) nd γ ( L, L ) is the humn contrst sensitivity chnge between the originl imge nd the bcklight scled imge, tht is, σ γ ( L, L ) = (11) σ The motivtion behind introduction of prmeter γ ( L, L ) is to ffect lrge nd smll luminnce vlues differently. More precisely, if only the κ ( L, L ) fctor ws used, in the trnsformed bcklight scled imge the contrst between two pixels would hve been incresed uniformly with respect to tht of the originl imge; however, with introduction of γ ( L, L ), s the contrst between two pixels in the originl imge increses the contrst between sme two pixels in the bcklight scled imge would increse but, grow more slowly for smller pixel luminnce vlues. Therefore, the result would be single tone mpping function which tkes into ccount the sensitivity sturtion of HVS (cf. Figure 1.) To mesure the temporl distortion, D tmp, of the video sequence we use the following procedure, 1. For ech originl frme, X t, nd bcklight scled frme, Y t, t time t, clculte the men brightness vlue, X(t) nd Y (t) of ll pixels in the respective frmes. 2. Filter signls X(t) nd Y (t), using the temporl response model of HVS to get perceived luminnce signls X (t) nd HVS Y HVS (t), respectively. 3. Clculte eqution (7) for X (t) nd Y (t) to get D HVS HVS tmp. Next, we use D tmp to modify the mximum llowed sptil distortion, D mx spt (9), in FBS techniques (cf. 14

15 Figure 6.) IV. IMPLEMENTATION AND EXPERIMENTAL RESULTS IV.A Hrdwre chrcteriztion IV.A.1 Cold Cthode Fluorescent Lmp (CCFL) The CCFL illumintion is complex function of the driving current, mbient temperture, wrm-up time, lmp ge, driving wveform, lmp dimensions, nd reflector design [28]. In the trnsmissive TFT-LCD ppliction, only the driving current is controllble. Therefore, we model the CCFL illumintion s function of the driving current only nd ignore the other prmeters. Accounting for the sturtion phenomenon in the CCFL light source [29], we use two-piece liner function to chrcterize the power consumption of CCFL s function of the bcklight fctor: A. β + C 0 β C lin lin s P ( β ) = bcklight (12) A. β + C C < β 1 st st s Reltionship between the CCFL illumintion (i.e., luminous flux incident on surfce per unit re) nd the driver s power dissiption for the CCFL in LG Philips trnsmissive TFT-LCD LP064V1 [30] is shown in Figure 8.. The CCFL illumintion increses monotoniclly s the driving power increses from 0 to 80% of the full driving power. For vlues of driving power higher thn this threshold, the CCFL illumintion strts to sturte. The sturtion phenomenon is due to the fct tht the incresed temperture nd pressure inside the tube dversely impct the efficiency of emitting visible light [31]. After interpoltion, we obtin the following coefficient vlues for the CCFL in LG Philips trnsmissive TFT-LCD LP064V1: C s =0.8234, A lin =1.9600, C lin = , A st =6.9440, nd C st = IV.A.2 TFT-LCD disply mtrix The hydrogented morphous silicon (-Si:H) is commonly used to fbricte the TFT in disply pplictions. For TFT-LCD pnel, the -Si:H TFT power consumption cn be modeled by qudrtic function of pixel vlue x [0,1] [32] 2 ( ).. TFT Pnel P x = x + b x + c (13) We performed the current nd power mesurements on the LG Philips, LP064V1 LCD. The mesurement dt re shown in Figure 6b. The regression coefficients re thus determined s: = , b= , nd c= Normlly white TFT-LCD pnel consumes less power while displying brighter (luminnce-enhnced) imge. Thus, this type of pnel dds dditionl power sving to the bcklight scling technique becuse it generlly enhnces the luminnce of the imge tht is displyed on the pnel to compenste for the loss of the brightness fter bcklight dimming. In contrst, power consumption of normlly blck TFT-LCD pnel increses slightly s its globl trnsmittnce increses (which increses power svings of bcklight dimming pproch.) However, for either type of the TFT-LCD, the chnge in power consumption s function of the trnsmittnce is so smll tht it cn be ignored. IV.A.3 Imge distortion chrcteriztion To del with the complexity of imge distortion function, D, first the imge distortion function is chrcterized for set of benchmrk imges s function of the dynmic rnge of the tone-mpped imges. Next stndrd curve fitting tools re used to generte n empiricl imge distortion curve bsed on this dt. Finlly, this empiricl curve is utilized s the imge distortion function D to find the minimum required dynmic rnge for ny given imge to chieve the mximum imge distortion of D mx fter tonempping. We dopted the universl imge qulity index proposed in [33] s our distortion mesure nd used set of benchmrk imges from the USC SIPI Imge Dtbse (USID) [34]. The USID is considered the de- 15

16 -fcto benchmrk suite in the signl nd imge processing reserch field [35]. Figure 9 depicts the resulting distortion vlues for these imges when the dynmic rnge of the trnsformed imge is set to twelve different vlues. Figure 10 is subset of benchmrks reported to provide visul reference for the distortion mesure. Next, we used stndrd curve fitting tools provided in MATLAB version 7 relese 14 to find the best verge nd worst-cse globl fits to these distortion vlues. The result is n empiricl curve depicted in Figure 9, which mps trget dynmic rnge of trnsformed imges to the observed distortion vlues CCFL illuminnce (i.e., bcklight fctor b) versus driver s power consumption for CCFL source 1 Actul mesurements Qudrtic fit b. Pixel trnsmittnce, versus power consumption of pixel in the normlly white TFT-LCD pnel Figure 8. LCD Component Power Consumption Chrcteriztion 16

17 Figure 9. Imge distortion vs. Dynmic Rnge Originl Dynmic rnge=200 Dynmic rnge = 150 Originl Dynmic rnge=200 Dynmic rnge = 150 Normlized power =1 Distortion=4.3% Power sving=36.19% Distortion=10.6% Power sving=45.24% Normlized power =1 Distortion=5% Power sving=35.72% Distortion=12% Power sving=46.49% Normlized power =1 Distortion=3.3% Power sving=37.16% Distortion=7.4% Power sving=48.28% Normlized power =1 Distortion=3.6% Power sving=32.21% Distortion=5.1% Power sving=42.57% Figure 10.Smple imges nd their corresponding trnsformed versions IV.B Implementtion To generte the experimentl results we used setup s shown in Figure 11. The Apollo test-bed II is used s the experimentl pltform [36]. This test-bed uses Intel X-scle processor nd includes different peripherl devices such s, video cpture device, GPS, nd wireless connection. Moreover, this test-bed hs 6.4 LCD screen with 6 bit/color/pixel t 640x480 resolution, nd provides enough hrdwre/softwre cpbilities for chnging the intensity of the bcklight nd online clcultion of the imge histogrm. The processor runs t 733MHz nd t this speed cn decode nd ply MPEG-1 video strem t the rte of 15 frmes per second. We modified the MPEG-1 decoder ppliction to incorporte the FBS bcklight scling technique bsed on the tone mpping function described in Section III. 17

18 Figure 11. Experimentl Setup To implement the proposed temporlly-wre bcklight scling, we mde two observtions, 1) the HVS model presented in section II.B.2 is continuous-time model while the verge luminnce of ech frme in the displyed video is clculted on frme by frme bsis, resulting in discrete-time signl. Therefore, we trnsformed the continuous-time filters given by equtions (1-, d) to discrete-time filters. 2) the output of the control low-pss filter in Figure 2 does not chnge significntly. This is due to limited rnge of output luminnce vlues for typicl off-the-shelf LCD. Therefore, we ssumed the output of control filter to be constnt, i.e., rc(t)=r0 (cf. Figure 2). This ssumption results in fixed qudrtic filters nd gin blocks in the temporl model of the HVS. Using these observtions nd clculting the output of the control low-pss filter, r 0, we get the following digitl filter, HLP( f ) = z + z z+ z (14) Next, we implemented this digitl filter nd then clculted the output of the HVS model for the verge luminnce vlue of ech frme in the video sequence. For the online clcultion of the temporl distortion, D tmp, we pproximted the verge vlue clculted in eqution (7), by verging the signl over limited number of previous filtered verge luminnce vlues, i.e., pproximting the overll verge by moving verge. Then, we used this vlue nd the user specified mximum llowed video distortion, D mx, in eqution (6) to clculte the mximum mx mx llowed sptil distortion, D. Next, we use D s the constrint for the proposed tone mpping bsed spt spt bcklight scling to clculte the pixel trnsformtion function for the originl imge with dynmic rnge DR. To clculte the pixel trnsformtion function we first lookup the corresponding llowed dynmic mx rnge, DR, for the given D using the distortion curves in Figure 9, nd then clculte the pixel spt DR trnsformtion function using eqution (10). This is done by clculting L s L = L. Then, we DR pply the clculted trnsfer function using technique similr to reference [8]. IV.B.1 Sttic Mesurements To show the effectiveness of proposed tone mpping bsed pixel trnsformtion function, we mesured the power sving for different still imges from USC SIPI dtbse. The results re shown in tble 1. These power svings re generted for three different vlues of distortion levels. Clerly, by incresing the mximum tolerble distortion level the power sving should increse, which is lso confirmed with listed results. Plese note tht the verge power sving of 45% is chieved only for mere distortion level of 10%. This is bout 10% increse in power sving compring to the results reported in [4] nd [5]. Nme Distortion = 5% Power sving (%) Distortion = 10% Distortion = 20% 18