A Markov Random Field Model for the Restoration of Foggy Images

Transcription

1 Internatonal Journal of Advanced Robotc Systems ARTICLE A Markov Random Feld Model for the Restoraton of Foggy Images Regular Paper Fan Guo 1, Jn Tang 1 and Hu Peng 1,* 1 School of Informaton Scence and Engneerng, Central South Unversty, Changsha, Chna * Correspondng author E-mal: hupeng@mal.csu.edu.cn Receved 26 Jan 2014; Accepted 30 Apr 2014 DOI: / The Author(s). Lcensee InTech. Ths s an open access artcle dstrbuted under the terms of the Creatve Commons Attrbuton Lcense ( whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgnal work s properly cted. Abstract Ths paper presents an algorthm to remove fog from a sngle mage usng a Markov random feld (MRF) framework. The method estmates the transmsson map of an mage degradaton model by assgnng labels wth a MRF model and then optmzes the map estmaton process usng the graph cut-based α-expanson technque. The algorthm employs two steps. Intally, the transmsson map s estmated usng a dedcated MRF model combned wth a blateral flter. Next, the restored mage s obtaned by takng the estmated transmsson map and the ambent lght nto the mage degradaton model to recover the scene radance. The algorthm s controlled by just a few parameters that are automatcally determned by a feedback mechansm. Results from a wde varety of synthetc and real foggy mages demonstrate that the proposed method s effectve and robust, yeldng hghcontrast and vvd defoggng mages. In addton to mage defoggng, survellance vdeo defoggng based on a unversal strategy and the applcaton of a transmsson map are also mplemented. Keywords Foggy Image, Defoggng, Markov Random Feld, Label Assgnment, Transmsson Map 1. Introducton Image defoggng s an mportant ssue n the feld of computer vson. There are many crcumstances n whch defoggng algorthms are needed, such as automatc montorng systems, automatc guded vehcle systems, outdoor object recognton and vsual navgaton n low vsblty envronments, etc. However, the qualty of mages taken n foggy weather condtons s easly undermned by the aerosols susped n the medum, whch have an effect on the mage such that the contrast s reduced and the surface colours become fant. Such degraded mages often lack vsual vvdness and offer a poor vew of the scene contents. The goal of defoggng algorthms s to recover the detals of scenes from foggy mages. Snce the process of removng fog from an mage deps on the depth of the scene, the essental problem that must be solved for most mage defoggng methods s scene depth estmaton. Ths s not trval, and requres pror knowledge. In ths paper, we propose a new method that can produce a good defoggng effect for varous foggy mages. The man motvaton of ths research s to mprove the vsual qualty of mages for the bulk of automatc systems and outdoor photos taken n poor weather condtons. For Fan Guo, Jn Tang and Hu Peng: A Markov Random Int J Adv Feld Robot Model Syst, for 2014, the Restoraton 11:92 do: of /58674 Foggy Images 1

2 example, n foggy weather, the qualty of mages captured by a classc n-vehcle camera s drastcally degraded, whch makes current n-vehcle applcatons relant on such sensors very senstve to weather condtons. An n-vehcle vson system should take fog effects nto account f t s to be more relable. A soluton s to remove fog effects from the mage beforehand. Ths s also the case for other applcatons, such as survellance, ntellgent vehcles, remote sensng and aeral photography, etc. Therefore, restorng foggy mages s hghly desrable n both computer vson applcatons and consumer photos. Usually, computer vson algorthms assume that the nput mage characterzes the scene radance. The performance of vson algorthms (e.g., feature detecton, flterng, object recognton and photometrc analyss) wll nevtably suffer from the based, low-contrast scene radance. Removng fog can sgnfcantly ncrease the vsblty of the scene and correct the colour shft caused by the ambent lght to make the vson algorthms more effectve and the appearance of foggy photos more pleasng. In ths paper, the proposed defoggng method combnes the MRF model wth transmsson map (scene depth) estmaton, and the graph-cut based α-expanson method s used here to optmze the map estmaton process. Ths provdes a new way to solve the mage defoggng problem. The man contrbuton of ths paper can be descrbed as follows: - A novel MRF-based method s proposed whch apples an optmzaton lbrary to estmate a transmsson map. Experments on both synthetc mages and real-world mages show the effectveness of the proposed method. Compared wth exstng defoggng methods, the proposed algorthm can remove fog more thoroughly wthout producng any halo artefacts, and the colour of the restored mages s natural n most cases. - We ext our proposed method to foggy vdeo applcatons usng a unversal strategy, whch greatly mproves computatonal effcency and enhances the vsual effect. The applcaton of our transmsson map, such as fog smulaton, s also mplemented based on the estmated transmsson map. - The adaptve adjustment of the algorthm s parameters usng a defoggng effect measurement ndex s realzed n ths paper. Thus, a statc, openloop parameter estmaton ssue s transformed nto a dynamc parameter adjustment ssue. In addton, the performance of the defoggng algorthms s effectvely measured usng approprate qualtatve and quanttatve evaluatons. The organzaton of ths paper s as follows. We begn by revewng exstng works on mage defoggng. In Secton 3, we ntroduce the MRF model and the outdoor geometry of a foggy mage. In Secton 4, we propose a defoggng algorthm based on the MRF model. In Secton 5, we ext our algorthm to vdeo applcatons and our transmsson map s also presented. In Secton 6, we present some expermental results. Fnally, n Secton 7, we make some concludng remarks. 2. Prevous Works Gven the mportance of defoggng algorthms, many studes on defoggng have been conducted. Prevous defoggng research can be dvded nto two categores: mage enhancement methods and mage restoraton methods [1]. Image enhancement methods t to ncrease the dynamc range and contrast of mages degraded by fog. Classc mage enhancement algorthms nclude hstogram equalzaton and a Retnex algorthm. Image restoraton methods cover the ntrnsc lumnance of an object usng addtonal nformaton or pror nformaton. Representatve algorthms nclude the dark channel algorthm [2] and the fast flter algorthm [3]. The dark channel algorthm [2] s recognzed as one of the most effectve ways to remove fog. The algorthm estmates the transmsson map of each patch as the mnmum colour component wthn that patch and employs a soft mattng algorthm to refne the map. The fast flter algorthm [3] has been proven to be faster than most other algorthms for outdoor scenes. The algorthm uses a fast medan flter to nfer the atmospherc vel and further estmate the transmsson map. The man advantage of ths method s ts speed. However, The defoggng algorthm n [2] s based on an mage prordark channel pror, whch s a knd of statstcs of hazefree outdoor mages, and the dark channel pror wll be nvald when the scene objects are nherently smlar to the ambent lght and no shadow s cast on them; n addton, the defoggng method n [3] s unable to remove the fog between small objects and the colour of the scene objects s unnatural for some stuatons. Graphcal models (GMs) are probablstc models combnng probablty wth a graph, and comprse an mportant means for solvng ths problem. Such models can be dvded nto two categores: drected graphs and undrected graphs. Generally, a drected GM s a Bayesan network (BN) when the graph s acyclc, meanng there are no loops n the drected graph. The relatonshps n a BN can be descrbed by local condtonal probabltes [4]. In [5, 6], a Bayesan defoggng method that jontly estmates the scene albedo and depth from a sngle foggy mage s ntroduced by leveragng ther latent statstc structure. The undrected graph refers to a MRF. Snce a MRF s undrected and may be cyclc, t can represent certan depences that a BN cannot, provdng a new means for mage defoggng due to the depences exstng between the neghbourng pxels. The defoggng algorthm n [7] s based on the observaton that the surface Lambertan 2 Int J Adv Robot Syst, 2014, 11:92 do: /58674

3 shadng factor and the scene transmsson are locally ndepent. Thus, the fog can be separated from the scene. Then, a Gaussan MRF s used to smooth the ntensty value of the transmsson map. In [8], a cost functon s developed wthn the framework of MRF to enhance the vsblty of mages. However, the results obtaned by ths method t to have larger saturaton values than those n the actual clear-day mages. In [9], scene geometry and the α-expanson optmzaton technque are employed to mprove the robustness of a sngle mage dehazng algorthm. Recently, mage defoggng based on the MRF model has made sgnfcant progress [10-12]. In [10], the mage defoggng problem s decomposed nto two steps: frst, the atmospherc vel s nferred usng a dedcated MRF model, and second the restored mage s estmated by mnmzng another MRF energy whch models the mage defoggng n presence of nosy nputs. In ths MRF model, the flat road assumpton s ntroduced to acheve better results on road mages. In [11], a MRF model for both stereo reconstructon and defoggng problems s combned nto a unfed MRF model to take advantage of both stereo and atmospherc vel depth cues. Thus, the stereo reconstructon and mage defoggng of daytme fog can be solved usng the new MRF model. In [12], a mult-level depth estmaton method based on a MRF model s presented for mage defoggng. The method ntegrates the characterstcs of a dark channel pror nto the MRF model n order to estmate an accurate depth map. The MRF s appled, here, to label the depth level n adjacent regons to compensate for wrongly estmated regons. The textures n the scene are the crtcal element, servng as the smoothng term n the MRF model. These fog removal algorthms are the most representatve of MRF defoggng methods, and they are all physcally sound. However, the colour and the profle of the scene objects can sometmes look unnatural for the defogged results. To solve the problem, we ntroduce an mage assessment ndex to the MRF model to optmze the parameters of the proposed method. Thus, vsually pleasng defoggng results can be obtaned. 3. Background 3.1 Markov Random Felds Many vson problems can be solved naturally usng the MRF technque. MRF theory s a branch of probablty theory for analysng the spatal or contextual depences of physcal phenomena. It s often used n vsual labellng to establsh the probablstc dstrbutons of nteractng labels. Here, we use an MRF to estmate the transmsson map n an mage degradaton model. It s an undrected graph, and adjacent nodes are connected to determne the depth of a real scene [12]. We assocate a hdden layer wth the dense level of fog and an observaton layer wth the ntal transmsson map, and then a MRF model s added to a cost functon, such that:, (1) E( f ) = D ( f ) + V ( f, f ) p p p, q p q p P ( p, q) N In (1), f = {f p p P} s a labellng of mage P, f p s the label of pxel p n mage P, and f p = {1,2,3,..., k}. In addton, q s the neghbour of p, N s the set of pars of pxels defned over the standard four-connecton neghbourhood, E(f) s for mnmzng the sums of two types of terms, and the frst term D p ( ) s a data functon. The smaller the dfference between a pxel and ts label, the smaller D p (.) wll be. D p (.) penalzes a label f p assgned to pxel p f t s too dfferent from the observed data I p. The second term V p,q (.) s a smoothng functon (or a dscontnutypreservng functon) [13, 14]. The smaller the dfference among the labels of the pxels n set N, the smaller V p,q (.) wll be. V p,q (.) encourages the ntegrty of an mage by penalzng two neghbourng labels f p and f q f they are too dfferent. The choce of V p,q (.) s a crtcal ssue, and n the proposed defoggng method we apply the outdoor geometry to obtan ths term. Wth the smoothng term, the saturated colours at each pxel can be computed wth reasonable smoothng. Thus, for the transmsson map estmaton, the data functon represents the probablty of pxel p havng a transmsson assocaton wth label f p. The smoothng functon encodes the probablty whereby neghbourng pxels should have a smlar depth. A graph cut s used here to mnmze the energy functon of the MRF. The method transforms an mage represented by a set of pxels nto a graph wth an augmented set of nodes, and then cuts the graph nto dfferent sets. The cuts correspond to some assgnment of pxels to labels. If the edge weghts are approprately set based on the parameters of the energy functon [see Eq. (1)], a mnmum cost cut wll be obtaned by labellng each pxel accordng to the mnmum value of ths energy functon. Thus, n fndng a cut that has the mnmum cost among all cuts, the mnmum value of the energy functon of the MRF can be obtaned and a proper label can be assgned to each mage pxel, as shown n Fgure 1. Therefore, the graph cut technque transforms the energy mnmzaton problem to an equvalent problem concernng fndng an effectve way to partton a specal graph constructed accordng to the prmal mnmzaton problem nto dfferent sets. α-expanson algorthm s used to solve the graph cut problem wth good computatonal performance. For the transmsson map estmated usng the MRF model, the smaller value of the label on behalf the deeper depth n the scene, whle the lager value correspondng to the scene ponts whch near the camera or observer.the relabelng results would consttute the ntal transmsson map of the proposed method. However, there remans certan redundant detals that need to be removed. Fan Guo, Jn Tang and Hu Peng: A Markov Random Feld Model for the Restoraton of Foggy Images 3

4 Fgure 1. Label assgnment by energy mnmzaton 3.2 Outdoor geometry for the foggy mage In ths secton, we present the outdoor geometry that s used n the transmsson map estmaton of the proposed algorthm. Lght passng through a scatterng medum s attenuated and dstrbuted n other drectons. Ths can happen anywhere along the path and leads to a combnaton of radances ncdent towards the camera, as shown n Fgure 2. Fgure 2. Scatterng of lght by atmospherc partcles Formally, to express the relatve porton of lght that managed to survve passage along the entre path between the observer and a surface pont wthn the scene, the defned transmsson map t combnes the geometrc dstance d and the medum extncton coeffcent β (the net loss from scatterng and absorpton) nto a sngle varable [15]: (a) (b) (c) (d) Fgure 3. The dstance and ntensty relatonshp of any scene pont. (a) Input foggy mage and three scene ponts. (b) The transmsson map for (a) and the scene ponts. (c) The relatonshp between the heght poston of the observaton pont and ts dstance. (d) The relatonshp between the heght poston of the observaton pont and ts ntensty n relaton to the transmsson map. 4. The Proposed Algorthm 4.1 The algorthm flowchart Specfcally, the proposed algorthm employs three steps n removng fog from a sngle mage. The frst one nvolves computng the ambent lght accordng to the three dstnctve features of the sky regon. The second step nvolves the computng of the transmsson map wth the MRF model and the blateral flter. The goal of ths step s to assgn an accurate pxel label usng the graph cut-based α-expanson and to remove any redundant detals usng the blateral flter. Fnally, wth the estmated ambent lght and the transmsson map, the scene radance can be recovered accordng to the mage degradaton model. The flowchart of the proposed method s depcted n Fgure 4. d t = e β (2) Accordng to (2), the followng outdoor geometry s reasonable: assumng that β s constant over the mage, the varatons n transmsson are due to the dstance d between the scene pont and the camera such that, the greater the dstance, the lower the ntensty n the transmsson map. For most outdoor mages, an object whch appears closer to the top of the mage s usually further away. Thus, the dstance along the ground to the object s a monotoncally ncreasng functon of the mage plane heght, whch starts from the bottom of mage gong up to the top. For example, from Fgure 3(a) one can clearly see that the dstance between the scene pont R and the camera s smaller than that between scene pont S or T and the camera. In addton, the ntensty at pont R n the transmsson map s hgher than that of pont S or T, as shown n Fgure 3(b). Fgures 3(c-d) show the relatonshp between the heght poston of the observaton pont and ts dstance or ntensty n relaton to the transmsson map. Fgure 4. Flowchart of the algorthm 4.2 Ambent lght and transmsson map estmaton The presence of aerosols n the lower atmosphere means that the lght may scatter and be absorbed whle travellng through the medum [16]. Ths can happen anywhere along the path, and t can lead to a combnaton of radances ncdent towards the camera. The mage degradaton model that s wdely used to descrbe the formaton of foggy mages s as follows [2]: I ( x) = J ( x) t( x) + A(1 t( x )) (3) where I(x) s the observed ntensty correspondng to the pxel x=(x, y), the nput foggy mage J(x) s the scene radance, the fog removal mage A s the ambent lght, and t(x) s the transmsson map, whch s the key factor 4 Int J Adv Robot Syst, 2014, 11:92 do: /58674

5 for mage defoggng. In (3), the frst term J(x)t(x) s called the drect attenuaton model and the second term A(1- t(x)) s called the ambent lght model. Theoretcally, the goal of fog removal s to recover J(x) from the estmated A, t(x) and the orgnal mage I(x) Ambent lght estmaton Estmatng ambent lght A should be the frst step n restorng the foggy mage. To estmate the ambent lght, three dstnctve features of the sky regon are consdered here, whch s a more robust approach than that of the brghtest pxel method. The dstnctve features of the sky regon are: () a brght mnmal dark channel, () a flat ntensty, and () an upper poston. For the frst feature, the pxels that belong to the sky regon should satsfy I mn (x) > T v, where I mn (x) s the dark channel and T v s 95% of the maxmum value of I mn (x). For the second feature, the pxels should satsfy the constrant N edge (x) < T p where N edge (x) s the edge rato map and T p s the flatness threshold. Due to the thrd feature, the sky regon can be determned by searchng for the frst connected component from top to bottom. Thus, the atmospherc lght A s estmated as the maxmum value of the correspondng regon n the foggy mage I(x) Intal transmsson map estmaton Transmsson map estmaton s the most mportant step for mage defoggng. Here, we use the graph cut-based α- expanson method to estmate the map t(x), as t s able to handle regularzaton and optmzaton problems, and has a good track record n energy mnmzaton [17]. Specfcally, each element t of the transmsson map s assocated wth a label x, where the set of Labels L = {0,1,2,... l} represents the transmsson values {0,1 / l,2 / l,...,1}. Before labellng, we frst convert the nput RGB mage nto a greyscale mage. Thus, the number of labels s 32, snce the labellng unt of a pxel value s set as 8 and l = 31. The most probable labellng x* mnmzes the assocated energy functon: (4) E( x) = E ( x ) + E ( x, x ) j j P (, j) N where P s the set of pxels n an unknown transmsson t, and N s the set of pars of pxels defned over the standard four-connect neghbourhood. The unary functon E (x ) s the data term representng the probablty of pxel havng transmsson t assocated wth label x. The smooth term E j (x, x j ) encodes the probablty whereby neghbourng pxels should have a smlar depth. For data functon E (x ), whch represents the probablty of pxel havng transmsson t assocated wth label x, we frst convert the nput RGB mage I nto a grey-level mage I ', and then compute the absolute dfferences between each pxel value and the label value. The process can be wrtten as: E x = I L x (5) ' ( ) ω ( ) ' In (5), I s the ntensty of a pxel n the grey-level mage (0 I ' 1), L(x ) denotes each element n the set of labels L= {0,1 / l,2 / l,...,1}. The parameter ω s ntroduced to ' ensure that I and L(x ) have the same order of magntude. The smooth functon E j (x, x j ) encodes the probablty whereby neghbourng pxels should have a smlar depth. Inspred by the work n [8], we use a lnear cost functon, whch s solved by α-expanson: E ( x, x ) = w x x (6) j j j From the outdoor geometry, we know that objects whch appear closer to the top of the mage are usually further away. Thus, f we consder two pxels and j, where j s drectly above, we have d j > d accordng to the outdoor geometry. Thus, we can deduce that the transmsson t j of pxel j must be less than or equal to the transmsson t of pxel by usng Eq. (2), that s x j x. For any par of labels whch volate ths tr, a cost c > 0 can be assgned to punsh ths pattern. Thus, the smoothng functon n Eq. (6) can be wrtten as: c f x < x j, Ej ( x, x j ) = w x x j otherwse. The parameters w and c are used to control the good or bad of the defoggng effect. The value of w controls the strength of the detal enhancement, and s set usually between 0.01 and 0.1. The cost c controls the strength of the colour recovery, and s usually set between 100 and 1,000. The two parameters are useful as a compromse between hghly enhanced detals where colours may appear too dark, and less restored detals where colours are brghter. Besdes, the weghts assocated wth the graph edge should be determned. If the ntenstes of two neghbourng pxels n the nput foggy mage I are less than 15 n each channel, whch means that the two pxels have a hgh probablty of sharng the same transmsson value. Thus, the cost of the labellng s ncreased by ffteen-fold to mnmze the artefacts due to the depth dscontnutes n ths case. Takng the data functon and the smoothng functon nto the energy functon equaton (4), the pxel label of the transmsson map can be estmated by usng graph cut-based α-expanson. In our (7) Fan Guo, Jn Tang and Hu Peng: A Markov Random Feld Model for the Restoraton of Foggy Images 5

6 method, the gco-v3.0 lbrary [17], developed by O. Veksler and A. Delong, s adopted for optmzng multlabel energes va the α-expanson. It supports energes wth any combnaton of unary, parwse and label-cost terms [18, 19]. Thus, we use the lbrary to estmate each pxel label n an ntal transmsson map. The pseudocode of the estmaton process usng the gco-v3.0 lbrary s presented n Fgure 5. Algorthm 1: Label assgnment usng optmzaton lbrary Input: Input foggy mage I Output: Each pxel label x n ntal transmsson map obtan each pxel label x. Next, a proper ntensty value of the ntal transmsson map can be assgned to each mage pxel. Specfcally, for each label x, we have: t n ( x ) = 255 ( x 1) 8 (8) In Fgure 6, we show a synthetc example n whch the mage conssts of fve grey-level regons. The mage can be accurately segmented nto fve label regons usng the proposed MRF method, whereby the fve labels are represented by fve ntensty values, whose results are shown n Fgure 6. Step 1 // Create new object Set NumStes = M N and NumLabels = 32; h = GCO_Create(NumStes, NumLabels); Step 2 // Compute data term 2.1 Convert colour mage I nto a grey mage and express t n one-dmensonal array data; 2.2 for = NumLabels for j = NumStes DataTerm(, j) = Data( j) ω (( ) / NumLabels) * 255 ; 2.3 GCO_SetDataCost(h, DataTerm); Step 3 // Compute smooth term 3.1 for = NumLabels for j = NumLabels f > = j SmoothTerm(, j) = w * j ; else SmoothTerm(, j) = c ; 3.2 GCO_SetSmoothCost(h, SmoothTerm); (a) Fgure 6. Synthetc example. (a) Input grey-level mage. (b) Output mult-label mage. The MRF-based algorthm can also be appled to estmate the ntal transmsson map for real-world mage. An llustratve example s shown n Fgure 7. In the fgure, Fgure 7(b) shows the ntal transmsson map estmated usng the algorthm presented above - ts correspondng restored result s shown n Fgure 7(c). One can clearly see that the appearance of the scene objects n the restored mage looks one-dmensonal. (b) Step 4 // Compute the cost of the labellng 4.1 for = NumStes - 1 f Data( + 1) Data( ) < 15 NeghborCost (, + 1) = 15 ; 4.2 GCO_SetNeghborCost(h, NeghborCost); (a) (b) (c) Step 5 // Compute optmal labellng va α-expanson 5.1 GCO_Expanson(h); 5.2 Label = GCO_GetLabelng(h); 5.3 Covert label whch s a one-dmensonal array nto a M N array, that s the output pxel label x. Fgure 5. The pseudo-code of the label assgnment usng the gcov3.0 lbrary In Fgure 5, M and N are the heght and wdth of the nput foggy mage, and ω, w and c are the parameters n Eqs. (5) and (7). By usng the functons defned n the optmzaton lbrary (e.g., GCO_SetDataCost, GCO_SetSmoothCost and GCO_GetLabelng), we can (d) Fgure 7. True example. (a) Input mage. (b) Intal transmsson map. (c) Restored result obtaned usng (b). (d) Blateral flter to (b). (e) Restored result obtaned usng (d) Refned transmsson map estmaton As shown n Fgure 7, there s an obvous defcency n the recovered mage n the dscontnutes of the transmsson map obtaned by the MRF model. For example, the red brcks and the gaps between them should have the same (e) 6 Int J Adv Robot Syst, 2014, 11:92 do: /58674

7 depth values. However, as shown n Fgure 7(b), one can clearly see the gaps between the brcks n the transmsson map estmated by the MRF-based algorthm. In order to handle these dscontnutes, many works adopt a blateral flter to refne the transmsson map estmaton, such as local albedo-nsenstve dehazng [20], flterng-based dehazng [21] and mage dehazng usng an teratve method [22], etc. In ths work, we also apply a blateral flter to our algorthm, snce such a flter can smooth mages whle preservng edges [23]. Thus, the redundant detals of the transmsson map t n estmated by the algorthm presented above can be effectvely removed, whch mproves the restored result wth better detal enhancement capablty. Ths process can be wrtten as: value of t 0 s too large, the result has only a slght defoggng effect, and f the value s too small, the colour of the fog removal result seems oversaturated. Experments show that when t 0 s set to 0.2, we can get vsually pleasng results n most cases. An llustratve example s shown n Fgure 8. In the fgure, Fgure 8(a) shows the nput foggy mages, Fgure 8(b) shows the transmsson map estmated by usng our MRF-based method, and Fgure 8(c) s the fnal defoggng result. t( u) = W ( p u ) W ( t ( u) t ( p) ) t ( p) c s n n n p N ( u) W ( p u ) W ( t ( u) t ( p) ) c s n n p N ( u) (9) (a) (b) (c) Fgure 8. Image defoggng example. (a) The nput mage. (b) Our transmsson map. (c) The fog removal mage. where t n (u) s the ntal transmsson map correspondng to the pxel u=(x, y) and N(u) denotes the neghbours of u. The spatal doman smlarty functon W c (x) s a Gaussan 2 2 x /2σ c flter wth the standard devaton σ c : Wc ( x) = e, and the ntensty smlarty functon W s (x) s a Gaussan flter wth the standard devaton σ s (t can be defned as: 2 2 x /2σ s ( x) = e ). In our experments, the values of σ c and Ws σ s are set as 3 and 0.4, respectvely. Thus, we can obtan the fnal refned transmsson map, as shown n Fgure 7(d-e), whch s the restored result obtaned usng the refned map. From Fgure 7(e), one can see that the restored result obtaned usng the blateral flter has more layers and ts stereoscopc depth percepton seems more evdent compared wth the result (Fgure 7(c)) obtaned wthout usng the flter. However, t takes about eght seconds to refne an ntal transmsson map of sze by executng MATLAB on a PC wth a 3.00 GHz Intel Pentum Dual-Core Processor. In addton, although n our experment we fnd that the flter can refne the map wthout creatng sgnfcant errors n the restored mage for our testng database, t may cause a gradent effect for some mages due to the fxed parameter values σ and σ for dfferent szes of mages. c s 4.3 Scene radance recovery Snce, now, we already know the nput haze mage I(x), the fnal refned transmsson map t(x) and the ambent lght A, we can obtan the fnal fog removal mage J(x) accordng to the mage degradaton model. The fnal defoggng result J(x) s recovered by: I( x) A J ( x) = + A max( t( x), t ) 0 (10) where t 0 s applcaton-based and s used to adjust the fog remanng at only the farthest reaches of the mage. If the 5. Extenson to Vdeo and Applcaton 5.1 Vdeo defoggng Gven the mportance of the fog removal method, many researchers have studed algorthms for sngle mage defoggng. However, research nto vdeo defoggng s rare n the lterature. Vdeo processng takes nto consderaton not only the pxel values n a sngle statc frame but also the temporal relatons between frames. For survellance camera systems, the camera s fxed and often postoned hgh up, such that the background of each frame s unchangeable and the dfference n the transmsson map between a foreground object and the background s usually small. Thus, we can regard the foreground object as mage nose and use some denosng algorthms - such as a blateral flter - to elmnate the nose and produce a unversal transmsson map. Fgure 9 shows the man dea behnd our vdeo processng method. Durng the defoggng process, the transmsson map s only calculated once for the background mage of the nput vdeo and then appled to more frames wth a tolerable error. Specfcally, we defne the statc part of the scene as the background part and the movng objects n the scene as the foreground part. The background mage can be obtaned by usng a frame dfferental method. Next, our method estmates the transmsson map of the background mage by usng the algorthm mentoned above as the unversal transmsson map, and apples the map to a seres of vdeo frames to obtan the restored mages, as shown n Fgure 10. The parameter values of the blateral flter used n our vdeo defoggng method are set to a smaller value of σ c and a larger value of σ s ( σ c = 1, σ s = 0.9 ) compared wth sngle mage defoggng, whch cause the foreground Fan Guo, Jn Tang and Hu Peng: A Markov Random Feld Model for the Restoraton of Foggy Images 7

8 nose and the redundant detals of the transmsson map to be effectvely removed. Generally, no sgnfcant errors wll be ntroduced nto the restored mage by usng the unversal transmsson map, as shown n Fgure 10. where t(x) s the estmated transmsson map. Once t(x) s computed, we can take the map nto Eq. (10) to create the smulated fog scenes by adjustng parameter λ, as shown n Fgure 11. (a) (b) (c) Fgure 9. The man dea behnd our vdeo defoggng process: Input vdeo frames (top), extracted background mage (mddle) and estmated unversal transmsson map (bottom). In the nput vdeo frames, the foreground objects are denoted by crcles, squares and trangles. These objects are regarded as mage nose and are elmnated usng a blateral flter durng the estmaton process of the unversal transmsson map. Fgure 11. Fog smulaton based on our transmsson map. (a) Input mage. (b) and (c) Smulated foggy mages wth λ = 2 and λ = 4, respectvely. 6. Expermental Results 6.1 Parameter settng The proposed MRF model [see Eq. (7)] s manly parameterzed by w and c, whch are the weghts of the smooth term. From Fgure 12, one can clearly see that the dents n the haystack are more obvous when w s close to 0.1 [see Fgure 12(a)], and the colour of the fog removal results seems less saturated when c s close to 1,000 w [see Fgure 12(b)]. (a) Fgure 10. Vdeo results. Frst row: estmated background mage and two orgnal frames from the vdeo. Second row: unversal transmsson map and the enhanced frames obtaned by usng the same transmsson map. 5.2 Applcaton After acqurng the transmsson map, we can add some vsual effects on the fog removal mage, such as fog smulaton. Fgure 11 shows the fog smulaton results. The vrtual foggy scene can be smulated by multplyng the extncton coeffcent β by a factor of λ. Specfcally, accordng to Eq. (2), ths s acheved by applyng the followng smple power law transformaton of the transmsson values: λ β d ( x) λ ( e ) = t ( ) x (11) (b) Fgure 12. Defoggng results wth dfferent parameter values. (a) From left to rght: orgnal mage, the results obtaned wth w = 0.02 and c = 200w, w = 0.1 and c = 200w. (b) From left to rght: orgnal mage, the results obtaned wth w = 0.05 and c = 200w, w = 0.05 and c = 1000w. The value of w and c are applcaton-based - we thus adopt the measurement presented n [24] to determne the proper value for the two parameters. For the CNC ndex proposed n that work, three components - contrast, mage naturalness and colourfulness - are combned to yeld an overall defoggng result measure. Therefore, the CNC ndex between the orgnal foggy mage x and the fog removal mage y s defned as: CNC( x, y) = e( x, y) CNI ( y) + CCI ( y) CNI( y) (12) 8 Int J Adv Robot Syst, 2014, 11:92 do: /58674

9 where e measures the contrast by the number of vsble edges n mage sgnals x and y, CNI s the mage colour naturalness that descrbes the degree of correspondence between human percepton and the external world, and CCI s the mage colour colourfulness that presents the degree of colour vvdness [25]. Good results are descrbed by a hgh value of CNC. We use the CNC ndex as a feedback sgnal to determne the optmal value for the two parameters. Thanks to the feedback mechansm, the statc, open-loop parameter estmaton ssue can be transformed nto a dynamc parameter adjustment ssue. Fgure 13 shows the average results of the CNC ndex wth a dfferent w and c for 128 testng mages. One can clearly see that the best result, correspondng to the hghest CNC value (1.2005), s obtaned when w = 0.05 and c = 200w (see pont M) for the testng mages. Therefore, the optmal values for w and c are approxmately 0.05 and 200w. Fgure 13. Average results of CNC wth dfferent parameter values We thus fx c = 200w and set an ndcator of w over the range [0.1, 0.01] by a certan nterval, whch set as The reason for us to choose CNC as a parameter adjustment ndex s that the ndex covers mage contrast, naturalness and colourfulness. Besdes, t s easy to mplement and has quck computatonal speed. Assume that the ndex values obtaned durng a step are represented as CNC1 and CNC2, then the nteracton condtons can thus be defned as: f CNC1 CNC2, teraton contnues. Otherwse, stop the teraton and obtan the value of w n ths condton. The defoggng mage produced wth w and c ( c = 200w ) s our fnal result. Fgure 14 shows the pseudo-code of the parameter estmaton process. 6.2 Synthetc Images M To evaluate the performance of varous defoggng algorthms, we rely on synthetc mages due to the dffculty of acqurng a scene wth and wthout fog. 66 synthetc mages wth unform fog from the database FRIDA2 [26] are used here. Ths database contans ground-truth no-fog mages as the target mages to compare varous defoggng methods. The sample results obtaned usng the proposed MRF-based algorthm on the FRIDA2 databases are shown n Fgure 15. One can see how far the extent to whch the buldngs and exts further n the restored mages. Algorthm 2: Parameter estmaton usng CNC ndex Input: Input foggy mage Iy Output: w, c, fog removal mage Ih Step 1 Output Im g1 = Iy ; // Set ntal condton for nteracton Step 2 for w = 0.1 : : 0.01 c = 200w ; // Fog removal usng the proposed method Output Im g1 = MRFDefog(Iy, w, c ) ; // Move a step to removng mage fog agan Output Im g2 = MRFDefog(Output Im g1, (w ), c ) ; // Usng CNC ndex to evaluate the two nput mages CNC_ndex = CNC (Iy, Output Im g1) // Obtan the CNC ndex value CNC1 = CNC_ndex ; CNC_ndex = CNC (Iy, Output Im g2) CNC2 = CNC_ndex ; // If the prevous CNC value s greater or equal to the current CNC value f CNC1 CNC2 // Set the varable for the next nteracton Output Im g1 = Output Im g2 ; Contnue ; else // Obtan the fnal defoggng result Iy = Output Im g1 ; Break ; Fgure 14. The pseudo-code of the parameter estmaton usng CNC Fgure 15. Defoggng results on synthetc mages from the FRIDA2 database. Frst row: the synthetc mages wth fog. Second row: the obtaned restored mages usng the proposed method. We compared the proposed algorthm wth the two classc enhancement algorthms: hstogram equalzaton and the Retnex algorthm, as shown n Fgure 16. Table 1 shows some of the results of the average absolute error (AAE) for the restored mage and the target mage wthout fog, and for three defoggng methods on 66 synthetc mages wth fog. In the evaluaton, good results are descrbed by a small value for the AAE. From Table 1, we can see that the proposed algorthm outperforms all Fan Guo, Jn Tang and Hu Peng: A Markov Random Feld Model for the Restoraton of Foggy Images 9

10 the other algorthms. The AAE of the proposed algorthm s 30.71, whle the next best result s for the Retnex algorthm, whch demonstrates that the results obtaned wth the proposed algorthm and the Retnex algorthm can effectvely remove the fog. However, the remote object n our results seems much clearer than that seen usng the Retnex method. colour nformaton and varance. If the fog s dense, the colour nformaton used n that method s not enough to relably estmate the transmsson. Fgure 18. From left to rght: the nput mage and the results obtaned by Fattal [7] and our method Fgure 16. Comparson of synthetc mages. From left column to rght column: orgnal foggy mage, results usng hstogram equalzaton, Retnex algorthm and the proposed algorthm. In addton, we compare our method wth Tan s work [8] n Fgure 19. The colours of Tan s result can sometmes over-saturate or dstort. For example, the colour of the sky and the road regon n Tan's result s turned yellow, as shown n Fgure 19. Algorthm Nothng Hstogram equalzaton Retnex algorthm Proposed method Mean error (n grey-levels) Fgure 19. From left to rght: the nput mage and the results obtaned by Tan [8] and our method Table 1. Average absolute error between the restored mage and the target mage wthout fog 6.3 Camera Images The algorthms proposed n ths paper work well for a wde varety of real captured foggy mages. Fgure 17 shows some examples of the defoggng effects obtaned usng the proposed MRF-based algorthm. One can clearly see that the mage contrast and detal are greatly mproved compared wth the orgnal foggy mages. We also compare our method wth He s work [2] n Fgure 20. He s algorthm can acheve a good enhancement effect for most outdoor mages. However, when the scene objects are nherently smlar to the ambent lght, the dark channel pror used n He s method wll be nvald. In ths case, the defoggng result of He s algorthm s not vsually pleasng, as shown n Fgure 20. Fgure 20. From left to rght: the nput mage and the results obtaned by He [2] and our method. Fgure 17. Defoggng results for real captured foggy mages usng the proposed method. Frst row: the real captured mages wth fog. Second row: the obtaned restored mages usng the proposed method. We also compared our defoggng algorthm wth several other state of the art algorthms. Fgure 18 shows a comparson between the results obtaned by Fattal [7] and our algorthm. As can be seen n Fgure 18, Fattal s method can produce a vsually pleasng result. However, the method s based on statstcs and requres adequate The comparson between the results obtaned by Tarel [3] and our algorthm s shown n Fgure 21. We can see that the colour n Tarel s result seems unnatural and that t also has many halo artefacts, whereas our method has no such problems. Fgure 21. From left to rght: the nput mage and the results obtaned by Tarel [3] and our method. Fgure 22 shows a comparson between the results obtaned by Carr [9] and our algorthm. It can be seen 10 Int J Adv Robot Syst, 2014, 11:92 do: /58674

11 that our algorthm ts to enhance detals better than Carr s result, and that the colour of our result seems closer to the orgnal nput mage. Fattal, wth better colour fdelty and fewer halo artefacts than compared to Tan, Tarel, Caraffa and Wang. Meanwhle, we also fnd that - depng on the mage - each algorthm s a trade-off between colour fdelty and contrast enhancement. Fgure 22. From left to rght: the nput mage and the results obtaned by Carr [9] and our method Fgures 23 and 24 show the results of our method and Caraffa s methods [10, 11]. From these mages, we can see that although the results we get are unable to thoroughly remove the fog n very dense fog regons compared wth Caraffa s methods (e.g., the buldngs and the trees n the dstance), our results appear natural n terms of both colour and the profle of the scene objects. To quanttatvely assess and rate the nne restoraton algorthms (Fattal s method, Tan s method, He s method, Tarel s method, Carr s method, Caraffa s two methods, Wang s method and the proposed MRF-based method), we use the CNC ndex [24] to measure the defoggng effect. Fgure 18 to Fgure 25 gve some example results obtaned usng the above defoggng algorthms, and ther correspondng CNC results are shown n Table 2. From the table, we can see that the hghest values of CNC are obtaned usng the proposed method. Ths llustrates that the proposed method can acheve as good or even better results n most real-world foggy stuatons as compared to other defoggng algorthms. Fgure 23. From left to rght: the nput mage and the results obtaned by Caraffa [10] and our method Fgure Image defoggng methods [7] [8] [2] [3] [9] [10] [11] [12] Our Table 2. CNC ndex computed for the nne compared methods Fgure 24. From left to rght: the nput mage and the results obtaned by Caraffa [11] and our method Fgure 25 shows a comparson between the results obtaned by Wang [12] and our method. One can clearly see that the colour of the sky regon n Wang s result seems a lttle nconsstent wth that of the orgnal foggy mage. To better evaluate the proposed method, an assessment method dedcated to vsblty restoraton proposed n [27] s also used here to measure the contrast enhancement of the defogged mages. We frst transform the colour-level mage to a grey-level mage and use the three ndcators to compare two grey-level mages: the nput mage and the fog removal mage. The vsble edges n the mage before and after restoraton are selected by a 5% contrast threshold accordng to the meteorologcal vsblty dstance proposed by the Internatonal Commsson of Illumnaton. To mplement ths defnton of contrast between two adjacent regons, the method for the segmentaton of vsble edges proposed n [28] has been used. Fgure 25. From left to rght: the nput mage and the results obtaned by Wang [12] and our method Results on a varety of haze and fog mages also show that the results obtaned wth our algorthm seem vsually close to the results obtaned by Carr, He and Once the map of the vsble edges s obtaned, we can compute the rate e of edges that are newly vsble after restoraton. Next, the mean r over these edges of the rato of the gradent norms both before and after restoraton s computed. Ths ndcator r estmates the average vsblty enhancement obtaned by the restoraton algorthm. Fnally, the percentage of pxels σ whch become completely black or completely whte after restoraton s computed. Fan Guo, Jn Tang and Hu Peng: A Markov Random Feld Model for the Restoraton of Foggy Images 11

12 These ndcators e, r and σ are evaluated for Fattal [7], Tan [8], He [2], Tarel [3], Carr [9], Caraffa [10, 11], Wang [12] and our method on eght mages (see Table 3). Indcator e r σ e r σ Method Fgure 18 ( ) Fgure 19 ( ) Fattal [7] Tan [8] He [2] Tarel [3] Carr [9] Caraffa [10] Caraffa [11] Wang [12] Our Method Fattal [7] Tan [8] He [2] Tarel [3] Carr [9] Caraffa [10] Caraffa [11] Wang [12] Our Method Fattal [7] Tan [8] He [2] Tarel [3] Carr [9] Caraffa [10] Caraffa [11] Wang [12] Our Method Fattal [7] Tan [8] He [2] Tarel [3] Carr [9] Caraffa [10] Caraffa [11] Wang [12] Our Fgure 20 ( ) Fgure 22 ( ) Fgure 24 ( ) Fgure 21 ( ) Fgure 23 ( ) Fgure 25 ( ) the defogged mages may have halos near some edges and the colour followng defoggng seems unnatural. Ths confrms our observatons regardng Fgure 18 to Fgure Vdeo defoggng results Expermental results wth vdeos of traffc scenes taken under foggy condtons are offered n Fgures 26 and 27. The two vdeo clps are used to evaluate the proposed algorthm for traffc montorng. One clp has 350 frames, wth RGB colour mages coded wth 24 bts per pxel. The other s a resoluton 200-frame vdeo. As descrbed n prevous experments, we frst obtan the background mage of the nput foggy vdeo sequences and compute the unversal transmsson map for the background mage. Next, the contrasts of the road, trees and movng vehcles were restored for each frame of the vdeo usng the unversal map. Notce the sgnfcant ncrease n contrast and the mprovement n colour (see Fgures 26 and 27). In our current mplementaton, fog removal was appled to the nput foggy vdeo whle offlne. Fgure 26. Restoraton results for vdeo clp #1. Frst row: estmated background mage and two orgnal frames from the vdeo. Second row: unversal transmsson map and the enhanced frames obtaned usng the unversal map. Table 3. Comparson of the state of art defoggng algorthms wth the three ndcators Under each method, the am s to ncrease the level of contrast wthout losng vsual nformaton. Hence, accordng to [27], good results are descrbed by hgh values of e and r and low values of σ. From Table 3, we deduce that, depng on the mage, Tan s algorthm generally has more vsble edges than the other algorthms. Moreover, we can order the algorthms n decreasng order wth respect to the average ncrease n the degree of contrast on the vsble edges: Tan, Tarel, He, Fattal, Caraffa, our own, Carr and Wang. However, n the experment we fnd that the algorthms wth more vsble edges probably ncrease the contrast to much, such that Fgure 27. Restoraton results for vdeo clp #2. Frst row: estmated background mage and two orgnal frames from the vdeo. Second row: unversal transmsson map and the enhanced frames obtaned usng the unversal map. 6.5 Computaton tmes The computatonal tme s measured by executng MATLAB on a PC wth a 3.00 GHz Intel Pentum Dual- Core Processor. 12 Int J Adv Robot Syst, 2014, 11:92 do: /58674

13 For an mage of sze s x s y, the fastest algorthm s Tarel s method. The complexty of Tarel s algorthm s O(s x s y ), whch mples that the complexty s a lnear functon of the number of nput mage pxels. Thus, only two seconds are needed to process an mage of sze For He s method, ts tme-temporal complexty s relatvely hgh, snce the Mattng Laplacan matrx L used for the method s so large; therefore, for an mage of sze s x s y, the sze of L s s x s y s x s y ; accordngly, 20 seconds are needed to process a pxel mage. The computatonal tmes of Fattal s and Tan s methods are even greater than that of He s method. They take about 40 seconds and fve to seven mnutes to process a foggy mage of the same sze, respectvely. For our proposed algorthm, t takes about two mnutes to process a pxel mage. Ths can be mproved usng a GPU-based parallel algorthm. Notce that when the mage sze s small, the proposed method has a relatvely faster speed. For example, only three seconds are needed to process a pxel mage usng our method, whle t takes about sx seconds to process an mage of the same sze usng He s method. 7. Conclusons Image defoggng s an mportant ssue n computer vson. In ths paper, a new defoggng algorthm was presented based on a MRF model. The problem was formulated as the estmaton of a transmsson map wth α-expanson optmzaton. The algorthm was mplemented n two stages. Frst, the transmsson map was estmated usng a dedcated MRF model and a blateral flter. Second, once the map was nferred, the restored mage could be obtaned accordng to the mage degradaton model. The expermental results demonstrated that the proposed algorthm can produce vsually pleasng defoggng results and that t ts to enhance the mage contrast, whch s better than prevous technques. The man contrbutons of ths work are as follows: (1) A novel MRF-based method was proposed whch apples an optmzaton lbrary to estmate a transmsson map. Experments on both synthetc mages and real-world mages showed the effectveness of the proposed method. (2) We exted our proposed method to foggy vdeo applcatons usng the unversal strategy and mplemented the fog envronment smulaton based on the estmated transmsson map. (3) Due to the feedback mechansm proposed n ths paper, the statc open-loop parameter estmaton ssue can be transformed nto a dynamc parameter adjustment ssue. However, the colour of our defoggng results sometmes seemed over-saturated, and some fog removal results may have a gradent effect. Nevertheless, we mght mprove the overall qualty of a foggy mage by enhancng the man detals, and the algorthm could be further mproved by employng better mage pror for the smoothng functon of the MRF model. In the future, we nt to nvestgate nstances of varous knds of fog and speed up the proposed algorthm for real-tme processng. 8. Acknowledgments The authors would lke to thank Dr. Fan for provdng hs paper [12], Dr. Tarel for provdng the MATLAB code for hs approach, and Dr. Fattal, Dr. Tan, Dr. He and Dr. Carr for provdng the defoggng mages on ther webstes. Ths work was supported by the Natonal Natural Scence Foundaton of Chna ( , and ), the Internatonal Scence and Technology Cooperaton Programme of Chna (2011DFA10440), the Collaboratve Innovaton Centre of resource economcal and envronment frly socety, the Chna Postdoctoral Scence Foundaton (No. 2014M552154), the Hunan Postdoctoral Scentfc Program (No. 2014RS4026), and the Postdoctoral Scence Foundaton of Central South Unversty (No ). 9. References [1] Yu B N, Km B S, Lee K H, (2012) Vsblty enhancement based real-tme Retnex for dverse envronments. The 8th Internatonal Conference on Sgnal Image Technology and Internet Based Systems (SITIS), Sorrento, Naples, Italy. USA: IEEE [2] He K M, Sun J, Tang X O, (2011) Sngle mage haze removal usng dark channel pror. IEEE Transactons on Pattern Analyss and Machne Intellgence. 33(12): [3] Tarel J P, Hautere N, (2009) Fast vsblty restoraton from a sngle color or gray level mage. IEEE Internatonal Conference on Computer Vson (ICCV), Kyoto, Japan. USA: IEEE [4] L S Z, (2009) Markov random feld modelng n mage analyss. UK: Sprnger-Verlag London Lmted. 21 [5] Nshno K, Kratz L, Lmbard S, (2012) Bayesan defoggng. Internatonal Journal of Computer Vson. 98(3): [6] Kratz L, Nshno K, (2009) Factorzng scene albedo and depth from a sngle foggy mage. IEEE Internatonal Conference on Computer Vson (ICCV), Kyoto, Japan. USA: IEEE [7] Fattal R, (2008) Sngle mage dehazng. ACM Transactons on Graphcs. 27(3): 1-9 [8] Tan R T, (2008) Vsblty n bad weather from a sngle mage. IEEE nternatonal conference on computer vson and pattern recognton (CVPR), Anchorage, Alaska, USA. USA: IEEE. 1-8 Fan Guo, Jn Tang and Hu Peng: A Markov Random Feld Model for the Restoraton of Foggy Images 13