IMAGE super-resolution (SR) [2] is the problem of recovering

Transcription

1 1 SoftCuts: A Soft Edge Smoothness Pror for Color Image Super-Resoluton Shengyang Da*, Student Member, IEEE, Me Han, We Xu, Yng Wu, Senor Member, IEEE, Yhong Gong, Aggelos K. Katsaggelos, Fellow, IEEE. Abstract Desgnng effectve mage prors s of great nterest to mage super-resoluton, whch s a severely under-determned problem. An edge smoothness pror s favored snce t s able to suppress the jagged edge artfact effectvely. However, for soft mage edges wth gradual ntensty transtons, t s generally dffcult to obtan analytcal forms for evaluatng ther smoothness. Ths paper characterzes soft edge smoothness based on a novel SoftCuts metrc by generalzng the Geocuts method [1]. The proposed soft edge smoothness measure can approxmate the average length of all level lnes n an ntensty mage. Thus the total length of all level lnes can be mnmzed effectvely by ntegratng ths new form of pror. In addton, ths paper presents a novel combnaton of ths soft edge smoothness pror and the alpha mattng technque for color mage super-resoluton, by adaptvely normalzng mage edges accordng to ther α-channel descrpton. Ths leads to the adaptve SoftCuts algorthm, whch represents a unfed treatment of edges wth dfferent contrasts and scales. Expermental results are presented whch demonstrate the effectveness of the proposed method. Index Terms SoftCuts, super-resoluton, edge smoothness, α- channel descrpton. I. INTRODUCTION IMAGE super-resoluton (SR) [2] s the problem of recoverng hgh resoluton (HR) mages from low resoluton (LR) nputs. Ths problem s of great practcal nterest, especally to vdeo communcaton, object recognton, HDTV, mage compresson, etc. Ths s a very challengng task, especally when only one sngle low resoluton nput mage s avalable (see for example [3]), as s very often the case. Ths paper manly focuses on ths case. In theory, the generaton process of low resoluton mages can be characterzed by smoothng and down-samplng the hgh resoluton scenes wth low-qualty mage sensors. Recoverng the orgnal hgh resoluton mage from low resoluton nputs s an nverse process. One ntutve approach to address ths nverse problem s to mnmze the reconstructon error, whch s defned as the dfference between the observaton and the result obtaned by puttng the recovered HR mage through the same generaton process. In other words, the result whch can produce the closed low resoluton mage to the Shengyang Da, Yng Wu, and Aggelos K. Katsaggelos are wth the Department of Electrcal Engneerng and Computer Scence, Northwestern Unversty, Evanston, IL 60208, Unted States, Tel: (847) , Fax: (847) , Emal: {sda690, yngwu, aggk}@eecs.northwestern.edu. Me Han s wth Google Inc., Mountan Vew, CA 94043, Unted States, Tel: (650) , Fax: (650) , Emal: mehan@google.com. We Xu and Yhong Gong are wth NEC Laboratores Amerca, Inc., Cupertno, CA 95014, Unted States, Tel: (408) , Fax: (408) , Emal: {xw, ygong}@sv.nec-labs.com. observed one s preferred. Such a reconstructon error can be effcently optmzed by the back-projecton method [4] n an teratve way. However, researchers have found that the superresoluton problem s essentally under-determned [5], [6]. Gven the LR nput or nputs, there mght be multple solutons that can mnmze the reconstructon error. Thus, by smply mnmzng the reconstructon error, the result may converge to an unsatsfyng soluton. To overcome ths dffculty, t s necessary to regularze the under-determned nverse problem by ncorporatng effectve mage prors. Smple nterpolaton-based methods, such as blnear or bcubc nterpolaton tend to produce hgh resoluton mages wth jagged edges, whch s also a common artfacts for many super-resoluton algorthms. Ths observaton motvates us to desgn an edge smoothness pror that favors an HR mage wth smooth edges. In addton, ths pror s also consstent wth human percepton. However, desgnng and ncorporatng an effectve edge smoothness pror s a very challengng task due to the followng two man dffcultes n practce: 1. It s very dffcult to quanttatvely characterze the smoothness of an edge, especally for edges n natural mages. In most real stuatons, an mage edge exhbts a gradual ntensty transton, whch s n general much more complex than a smple geometrc curve. In ths paper, we refer to such an edge as a soft edge (see Fg. 1(b) for an example), as opposed to a hard edge, as shown n Fg. 1(a). Workng wth soft edges poses an extra dffculty n measurng and quantfyng the edge smoothness n an analytcal form. 2. Color mage edges are determned by the nformaton from all three color channels smultaneously. Besdes, edges n natural mages exhbt a large varaton wth dfferent contrasts and scales. How to explore the 3D color nformaton and treat those varous edges n a unfed way s of great mportance to color mage SR. Ths paper s manly focused on addressng the above two ssues. The man contrbutons are as follows: 1. To measure and quantfy the edge smoothness, we ntroduce the Geocuts method [1], whch can approxmate the Eucldean length of a hard edge wth a cut metrc on the mage grd. More mportantly, n order to handle soft edges, we propose an extenson of the SoftCuts method based on a soft edge cut metrc, whch can measure the smoothness of soft edges n an ntensty mage. Ths extenson s sgnfcant because t leads to a new analytcal form for the soft edge smoothness pror. The new smoothness measure s used to regularze the objectve functon of the SR task, and produce mpressve results. Ths metrc also has a nce geometrc

2 2 (a) (b) (c) Fg. 1. Examples of (a) hard edge, (b) soft edge, and (c) level lnes (boundares between dfferent ntenstes) for (b) wth quantzaton step sze equal to 64 on mage ntenstes. property, snce t approxmates the average length of all level lnes n the ntensty mage, where a level lne represents the boundary between pxels wth ntenstes smaller and larger than a gven value (an example s shown n Fg. 1(c)). 2. To handle varous edges n color mages, a novel mage representaton scheme s proposed based on the alpha mattng technque from the computer graphcs lterature. We transform the problem of color edge super-resoluton to a combnaton of alpha mattng decomposton and α-channel super-resoluton. Ths makes possble the applcaton of the soft edge smoothness pror to natural color mages. 3. To process the entre mage n a unfed way, an adaptve strategy s further proposed to utlze the SoftCuts for all mage edges smultaneously by usng an α-channel mage descrpton obtaned wth a multple layer mage mattng decomposton method [7]. Due to ths decomposton, the SoftCuts regularzaton term for dfferent edges s normalzed, such that the smoothness measure of all level lnes n the α channel has the same weght. The proposed SR algorthm has the followng benefts: (1) Due to the geometry property of the proposed SoftCuts measure, the length of all mage level lnes can be mnmzed smultaneously for the super-resoluton task. Thus, results wth smooth edges can be obtaned. In the mean tme, the edgepreservng property of the proposed pror term can also make the resultng edges have sharp transtons. (2) All three color channels are utlzed smultaneously wth the α-channel SR scheme, and the adaptve strategy provdes a unfed treatment of edges wth dfferent contrasts and scales. In ths paper, the related work s summarzed n Sec. II. The SoftCuts method s proposed n Sec. III, and the geometrc explanaton s presented. Ths SoftCuts metrc can be used as a regularzaton term n the super-resoluton problem to favor soft smooth edges. To make use of ths term adaptvely, we frst ntroduce the α-channel edge/mage descrpton method n Sec. IV. Based on that, an adaptaton scheme s derved n Sec. V, whch apples the SoftCuts regularzaton term wth the same weght on dfferent edge transtons. Experments are shown n Sec. VI and Sec. VII concludes ths paper. II. RELATED WORK Due to the under-determned nature of the super-resoluton problem, an mage pror needs to be utlzed to regularze the restoraton process. Extensve work has been done on superresoluton gven multple LR mages [2], [4], [8], [9]. The pror term (or mage regularzaton term) n these works could also be appled to the sngle mage SR problem drectly. So n ths secton, we wll also revew the methods used when multple LR mages are avalable. In general, havng multple LR nput mages mposes more constrants on the lkelhood term (or mage fdelty term). Thus, t s argued that the use of an effectve mage pror s even more crtcal for sngle mage SR. Exstng works on the use of mage prors for superresoluton tasks can be placed nto two categores: prors from mage modelng and prors from learnng. In the frst case, based on the observaton that neghborng pxels tend to have the same color, the mage smoothness pror s wdely explored, such as n varous flterng or nterpolaton algorthms (e.g., blnear and bcubc nterpolaton). Such methods usually produce blurry mages, snce the mage edges are also smoothed. Edge-preservng regularzaton terms [9], [10], [11] are desgned to address the over-smoothness problem at mage boundares. In [9], the blateral pror s used for sngle channel mages. The hgh order neghborhood s also nvestgated n ths work, but from a dfferent perspectve from ours. Ths work s extended to color mages [12] by optmzng a mult-term cost functon, whch s a combnaton of regularzaton terms on llumnance and chromnance. In [10], the sparse dervatve pror s appled, whch can preserve sharp edges by modelng the mage dervatve wth a heavy-taled dstrbuton, nstead of tradtonal Gaussan dstrbuton. In [11], a TV pror s used along wth a varatonal approxmaton for obtanng the soluton. Another popular approach s usng edge drected nterpolaton [13], [14], [15], whch s tryng to nfer sub-pxel edges postons, to further prevent cross-edge nterpolaton. Locatng the edge poston wth hgh precson s of great mportance for those methods. Another common observable artfact of super-resoluton algorthms s the jagged edge effect. It s reasonable to assume an edge smoothness pror wthout any other pror knowledge on the mage. Ths pror s also consstent wth human percepton, whch seems to also favor smooth curves n natural mages. Based on ths assumpton, a lot of algorthms have been proposed to obtan smooth edges. Varous technques are nvestgated n the lterature to obtan smooth mage boundares, such as Level-set [16], multscale tensor votng [17], and Snake-based vectorzaton [18] technques. Instead of modelng the mage pror explctly, many researchers try to learn the pror from natural mages, especally by learnng from mage exemplars drectly. One common way s to nfer the mssng hgh frequency nformaton for the HR mages for each poston based on the mddle frequency nformaton. Ths approach s adopted n [19] for the mage restoraton problem. The Markov Random Felds model [20], [21], [22] usually s utlzed to enforce spatal consstency. Extenson of ths research drecton ncludes vdeo SR [23] and learnng from doman-specfc exemplars [24]. In such methods, two key problems usually need to be addressed. The frst s how to search HR canddate patches effcently. Localty Senstve Hashng [25] and KD-tree [24] are appled to speed up the searchng. The second s how to solve the optmzaton problem effcently. Belef propagaton can be used, and the mage prmal sketch [26] method can smplfy the problem to a chan structure. Other learnng algorthms can also be used

3 3 to nfer the hgh frequency nformaton, such as locally lnear embeddng [27]. Ths work s an extenson of the authors prevous work on α-channel edge super-resoluton [28]. Ths paper represents a more detaled and comprehensve treatment of the topc, and ntroduces the local adaptve scheme under the framework of α-channel super-resoluton, n order to enable a unfed treatment of the entre mage. III. SOFT EDGE SMOOTHNESS PRIOR In ths secton, we frst revew the Geocuts method whch s desgned for mage segmentaton tasks, and then present our work of the SoftCuts method for measurng the soft edge smoothness. A. Geocuts Our work s partally motvated by the Geocuts method [1], whch s brefly summarzed n ths secton. The basc dea of Geocuts s to use a cut metrc to approxmate the Eucldean length of a curve. The cut metrc s defned on the mage grd graph, wth edges connectng pxels n a large neghborhood system, and proper edge weghts. Snce such a cut metrc has an analytc form, the smooth hard edges can be obtaned by mnmzng ths metrc. More formally, the weghted grd-graph G = V, E s desgned n the followng way. V s the set of all dscretzed mage pxel postons. The edge set E s defned accordng to the neghborhood system represented by a set of vectors {e k 1 k n G }, where n G s the neghborhood order, and the e k s are chosen as the relatve poston (takng nteger values as ts components, and the unt s the grd nterval) of the nearest n G neghbors wth dfferent drectons. They are ordered by ther correspondng angle φ k w.r.t. the +x axs, such that 0 φ 1 < φ 2 <... < φ ng < π. For example, when n G = 4, we have e 1 = (1, 0), e 2 = (1, 1), e 3 = (0, 1), and e 4 = ( 1, 1). The set of pxel pars wth relatve poston e k s denoted by N k, assumng that N = n G k=1 N k. All pxel pars wth relatve poston nsde the set N are connected by edges. Fgure 2 shows some example neghborhood systems. After defnng the edges, the edge weghts correspondng to N k are defned as w k = δ2 φ k 2 e k, (1) where φ k = φ k+1 φ k (set φ ng +1 = π), and δ s the sze of the grd nterval. Gven the weghted grd-graph G and a curve C n R 2 overlayed on G, we denote by E C the set of edges ntersectng ths curve. The cut metrc of C s defned as C G = e E C w e, (2) where w e s the weght for edge e. In another words, C G s the summaton of the weghts of all edges ntersectng C. Denotng by C E the Eucldean length of curve C, the followng theorem s derved n [1]: e e 3 4 e 2 δ e1 δ Δ φ 3 Fg. 2. Neghborhood system for n G = 2, 4 (left) and n G = 12 (rght, only the neghbors on the upper plane are shown). Theorem 1: If C s a contnuously dfferentable regular curve n R 2 ntersectng each straght lne a fnte number of tmes then C G C E as δ, sup k φ k, and sup k e k go to zero [1]. Theorem 1 means that the length of a curve can be approxmated by ts cut metrc on a weghted mage grd graph. It s derved from the Cauchy-Crofton formula n Integral Geometry, whch relates the Eucldean length of a curve wth the expectaton of the number of ntersectons wth a random lne. Ths expectaton s further approxmated by the cut metrc defned n Eqn. 2 wth the mage grd graph and approprately chosen edge weghts. Roughly speakng, the approxmaton error converges to zero as the grd sze goes to zero and a larger neghborhood s consdered. Ths method can be generalzed to 3D and arbtrary Remannan metrc. The global mnmum of the cut metrc can be obtaned n a close-to-lnear tme by the Graph Cuts method [29], [30]. As ts name suggests, Geocuts reveals the underlnng relatonshp between two well-known segmentaton algorthms,.e., Geodesc actve contours and Graph Cuts. Geocuts also provdes a prncpled way to choose the edge weghts for usng hgher order neghborhoods. By ntegratng the cut metrc nto an objectve functon, the hard edge smoothness pror can be added. Curves wth smaller Eucldean length are preferred by mnmzng such an objectve functon, thus smooth curves are obtaned. B. SoftCuts for smoothness measure of soft edges Now, we present the proposed SoftCuts method as a generalzaton of the Geocuts method. A cut metrc can be defned on any set of dsjont closed curves C, or equvalently, a bnary valued characterstc functon χ C on R 2 whch equals 1 nsde C, and 0 otherwse. Geocuts s only applcable to a bnary valued functon χ C on the mage plane. To handle soft edges, whch are gradual transtons n an ntensty mage, we frst rewrte the defnton of the cut metrc n Eqn. 2 w.r.t. curve C (or equvalently, functon χ C ) as follows C G = χ C G (3) ( = wk χ C p χ C ) q, (4) (p,q) N k 1 k n G where p and q are pxel postons on the grd G; they are used as subscrpts to ndcate the functon value at that pxel φ 3

4 4 for smplcty throughout ths paper. We smplfy the above equaton as C G = w pq χ C p χ C q, (5) (p,q) N where w pq s the weght for neghborng pxel par (p, q), whch s determned by the relatve poston of p and q. In other words, for (p, q) N k, w pq = w k. Although Eqn. 5 s equvalent to Eqn. 2, the former s easer to be generalzed to a real valued functon S on R 2. We defne the soft cut metrc for S w.r.t. the grd-graph G as follows S G = w pq S p S q. (6) (p,q) N Equaton 6 takes the same form as Eqn. 5, wth the only dfference that contnuous values are allowed for S n Eqn. 6, nstead of bnary value for χ C n Eqn. 5. By unformly quantzng the functon values wth step 1 n, S can be approxmately by S d, whch takes values from {0, 1 n, 2 n,..., 1}. The soft cut metrc of Sd can be smlarly defned by Eqn. 6, by replacng S wth S d. Moreover, S d can be equvalently descrbed by a set of level lnes l 1, l 2,..., l n, where l s the boundary between regons wth S d values that S d < n and Sd n n R2. From Theorem 1, we know that the length of l can be approxmated by ts cut metrc l G. Based on ths, we have the followng theorem (the proof s gven n Appendx A). Theorem 2: Assume that S s a contnuous dfferentable regular functon on R 2, whch ranges n [0, 1], and S d dscretzes S wth step 1 n. Then the average length of all level lnes of S d w.r.t. 1 n can be approxmated by the soft cut metrc of S d,.e., S d G 1 l E (7) n 1 n under the same condtons of Theorem. 1 Theorem 2 generalzes Theorem 1 to soft edges from hard boundares. It mples that by mnmzng the soft cut metrc n Eqn. 6, the sum of lengths of dscrete level lnes can be mnmzed. So addng ths metrc as a regularzaton term can help us obtan results wth smaller length of mage level lnes. Thus the soft smoothness pror can be easly ncorporated nto the SR objectve functon. There are several related works n the lterature. The levelset method [16] and multple-scale tensor votng [17] has been used to ncorporate the edge smoothness pror. Image gradent on a large neghborhood s also used n [9] as a regularzaton term. Compared wth these exstng works, the beneft of the proposed SoftCuts method s that we have an explct analytc term to characterze the edge smoothness, wth a specfc geometrc explanaton. When n G = 2, Eqn. 6 becomes an approxmaton to the total varaton (TV) regularzaton term [31], [32], [33], whch s very powerful n edgepreservng mage reconstructon. IV. α CHANNEL IMAGE DECOMPOSITION For a color mage I, mattng s the technque that separates ts foreground objects from the background. As the pxels on the object boundary tend to be mxtures of both foreground and background color components, the separaton needs to be done softly. A color mage can be treated as a lnear combnaton of the foreground and background mages. Specfcally, for each pxel p, we have I p = α p F p + (1 α p )B p, (8) where I p, F p, and B p are the pxel color vectors of the nput, foreground, and background mages, respectvely, and α p [0, 1] determnes the weghts of the convex combnaton of foreground and background at poston p. Alpha mattng tres to recover F, B, and α smultaneously for each pxel, gven an nput color mage I. An example s shown n Fg. 3. For a color mage edge, f we consder the two sdes of ths edge one as the foreground layer F and the other as the background layer B, then the regon close to ths edge can also be decomposed by alpha mattng. Once havng the decomposton, nstead of usng the orgnal pxel colors, we use α to represent ths edge, and refer to t as α-channel edge descrpton. The dea of usng the α-channel s attractve due to ts specal propertes as descrbed next. Frst of all, the α-channel provdes a normalzaton of the edge, that s, they transton from 0 to 1 between the two sdes of the edge, nstead of a large range of possble values n the color channels. More mportantly, the α-channel actually preserves the edge nformaton n a sngle channel as we wll see n Sec. V-C. So t can be consdered as a normalzed verson of the mage edge. In addton, usng the α-channel can explore the relatonshp among three color channels, and naturally combne all color nformaton from three channels. In fact, color channels are closely related to each other. One example s the lnear color model [34], whch means that the pxels colors n a local mage patch tend to form a lne n the 3D color space. Such nformaton s totally gnored by methods that process three color channels separately. The α-channel s extracted n a way such that all color nformaton s taken nto account smultaneously. Thus, the color channel relatonshp nformaton s mplctly ntegrated. The dea of usng the α-channel has been successfully appled to mage deblurrng [35], [36], [37]. Mattng s also used n [17] to extract sub-pxel locaton of the curve for super-resoluton. A two color mage pror, whch n essence very smlar to the mattng decomposton, s used n [38] for demosacng. Gven an nput mage I, solvng for α, F, and B smultaneously s obvously an under-determned problem. Image prors are needed to regularze the problem. One commonly used pror s the mage smoothness pror, whch assumes local color smoothness for both foreground and background. Fgure 3 shows an example of a mattng soluton wth ths pror, where we can clearly see that F and B are locally smooth along the mattng boundary, and α retans most of the edge nformaton. Ths smoothness pror s extended to the local lnear color model n [34]. Based on ths assumpton, a closed-form soluton s derved. User nteracton can be consdered as another knd of pror, where some pure foreground / background pxels

5 5 (a) (b) (c) (d) (e) Fg. 3. An example of the alpha mattng technque: (a) nput color mage I, (b) human nteracton specfyng pure foreground and background pxels wth black and whte strokes, (c) extracted α channel (whte represents 1, and black represents 0), (d) foreground patch F, (e) background patch B. Fg. 4. Multple layer mage representaton. The top left mage s the nput, the other fve mages represent the α value for the fve layers. all the propertes of the α-channel edge descrpton can be drectly appled to the α-channel mage descrpton scheme n those regons. Although ths property s not satsfed for pxels correspondng to more than two layers, such pxels only represent a very small percentage of the total number of pxels [7]. Moreover, the α-channel mage descrpton ntegrates the nformaton of all mage edges smultaneously, nstead of one sngle edge for the α-channel edge descrpton. Ths property makes the α-channel mage descrpton more sutable snce t enables a unfed mage processng soluton, as shown later n Sec. V-C. are ndcated manually by provdng a trmap or usng a brush tool (Fg. 3(b)) [39]. Tradtonally, the alpha mattng technque decomposes an nput mage nto two components,.e., a foreground layer and a background layer. Very recently, the spectral mattng technque [7] was proposed to generalze ths concept by decomposng a color mage nto a lnear combnaton of multple mage layers as follows I = n α L, (9) =1 where L s the -th mage layer, α s the correspondng combnaton weght, satsfyng α = 1 for each pxel, and n s the total number of layers. Ths method not only enables multple layer mage representaton, but also leads to a fully automatc and unsupervsed soluton for α by the spectral clusterng technque. Thus, human nteracton that specfes partal foreground or background pxels s not necessary anymore. An example s shown n Fg. 4. Each layer roughly corresponds to a homogeneous mage regon, and the edge nformaton s captured by at least one of those α channels. The defnton of α-channel edge descrpton can be easly extended to an α-channel mage descrpton as a = (α 1, α 2,, α n ), gven the multple-layered mage decomposton. Smlar to the α-channel edge descrpton, mage edges n the α-channel mage descrpton are also normalzed n the range [0, 1] over one sngle channel. Pxels along the border of two neghborng mage layers have two non-zeros components n ther α-channel mage descrpton, and these two values should sum up to 1 for each pxel. The α-channel edge descrpton and the α-channel mage descrpton contan the same edge transton nformaton for those pxels. Thus V. SoftCuts FOR COLOR IMAGE SUPER-RESOLUTION In ths secton, we frst apply the SoftCuts method drectly to sngle channel mage super-resoluton. Then we use the SoftCuts on α channel to super-resolve color mage edges. Fnally, the adaptve SoftCuts method s presented to process the entre color mage n a unfed framework. A. Sngle channel super-resoluton by SoftCuts Theoretcally, the generaton process of an LR mage can be modeled by the combnaton of varous blurs, such as atmospherc blur, moton blur, camera blur, out-of-focus blur, and down-samplng. By combnng the varous blurs nto a sngle flter G for the entre mage, the generaton process of a sngle channel LR mage can be formulated as follows I l = (I h G) +n, (10) where I h and I l represent the HR and LR sngle channel mages respectvely, G the mpulse response of the blurrng flter, the convoluton operator, the down-samplng operator, and n the addtve nose. We propose the SoftCuts method by utlzng the soft cut metrc for sngle channel SR as the regularzaton term. The objectve functon s now defned as follows I h = arg mn I ( I l (I G) λ I G ), (11) where the frst term on the rght sde represents the fdelty to the data, the second term represents our pror knowledge about the orgnal mage defned by Eqn. 6, and λ s the regularzaton parameter controllng the contrbuton of the two terms. Whle the l 2 norm s used for the data fdelty term (stemmng also from a Gaussan assumpton for the nose term n a stochastc formulaton of the problem), the l 1 norm s used for

6 6 the regularzaton term. Its geometrc meanng s explaned by Theorem 2. Besdes, the l 1 norm does not severely penalze large local gradents n general, thus allowng for sharp edges n the restored mage (the l 2 norm results n gradual ntensty transton across edges). The objectve functon s optmzed by the steepest descent algorthm. Denotng by p re and p s the gradents of the frst and second term n Eqn. 11, we obtan the followng teraton I t+1 = I t β(p re + p s ), (12) where p re = ( (I t G) I l) G, (13) p s = λ ( w k {sgn I t D ek I t) D ek sgn ( I t D ek I t) }, 1 k n G (14) β s the descent step sze and denotes the up-samplng operator. p re s smlar to the updatng functon of the backprojecton [4] method, except that the back-projecton kernel s the same as the blur flter. D ek s the dsplacement operator, whch translates the entre mage by e k (an nteger valued vector defned n Sec. III-A), and sgn s the sgn ndcaton functon. p s s the dervatve of the soft cut metrc defned by Eqn. 6. In fact, each term n Eqn. 6 wll produce a (+w k ) or ( w k ) change for the two correspondng pxels. Ths updatng strategy s the same as n [9]. In our experments, I 0 s equal to the bcubc nterpolaton result. B. Color edge SR by SoftCuts For natural color mage SR, a nave soluton s to apply the above sngle channel SR algorthm to all three color channels ndependently. However, ths approach tends to fal for a couple of reasons. Frst, the SoftCuts method s senstve to the value of λ. The selecton of ths parameter s related to the local edge propertes. Some edge strength normalzaton mechansm s needed to make possble a unfed treatment for all edges. Second, n order to determne the exact edge poston, nformaton from all three color channels s requred. Decsons made on each channel separately mght be erroneous and nconsstent. To address the above ssues for color mage superresoluton, n ths secton, we propose to super-resolve color edges by super-resolvng ts α channel. Assume that an LR mage edge I l can be decomposed as I l = α l F l + (1 α l )B l. Based on the local smoothness assumpton of F l and B l mentoned before, t s easy to see that super-resolvng the edge I l can be acheved by superresolvng ts α channel α l. Thus the mage super-resoluton can be performed by processng each mage edge separately. To be more specfc, for each edge, we frst perform alpha mattng to a nearby regon I l, to obtan α l, F l, and B l. Then ther HR counterparts α h, F h, and B h can be recovered accordngly from them. Recoverng α h from α l = (α h G) s exactly the sngle channel SR problem dscussed n Sec. V-A, whle F h and B h can be nterpolated usng for example the bcubc method gven ther down-sampled versons due to the smoothness assumpton. Fgure 5 llustrates the dea of α-channel SR for one mage edge. The LR patch s decomposed nto two mage patches and an LR α channel. Fg. 5(e) shows the recovered HR α channel by the proposed SoftCuts method. Combnng the mages n Fg. 5(b)(c)(e) usng Eqn. 8 wll produce the sharp and smooth edge, shown n Fg. 5(f). The SoftCuts edge SR method has the followng benefts. Frst, wth α-channel edge descrpton, each edge s normalzed to a unfed scale. The same value of λ s appled to super-resolve the α-channel descrptons of all edges, thus the problem of parameter selecton for the SoftCuts method can be avoded. The underlyng ratonale s that we want to apply equal weghts to all level lnes over the α channel of dfferent edges. Besdes, the alpha mattng technque can extract an edge by combnng color nformaton from all three channels, thus more precse results can be obtaned. However, applyng ths edge SR method on an entre color mage s not easy [28]. It heavly reles on successful extracton of mage edges for the low resoluton nput mage. In addton, to enable edge decomposton by alpha mattng, hard constrants for foreground and background pxels need to be automatcally specfed, whch s also qute challengng, and sometmes even mpossble. C. Adaptve SoftCuts for color mage SR In ths secton, based on the α-channel mage descrpton, an adaptve method s proposed to overcome the dffcultes of applyng the α-channel super-resoluton strategy mentoned above. Based on that, the entre mage can be super-resolved n a unfed way by mplctly processng all mage edges smultaneously. We start by consderng an mage edge, and rewrtng Eqn. 8 as follows α = 1 F c B c Ic Bc F c B c, (15) where c {r, g, b} s the ndex for the color channels, and the RGB color space s used n ths work. The subscrpt p n Eqn. 8 s omtted for smplcty. For a sngle mage edge, the assumpton s made that both F and B are locally smooth (Please refer to Sec. IV for more detals). Thus, Eqn. 15 shows that the α-channel edge descrpton s a lnear functon of the 1 orgnal mage ntensty, wth scalng factor F c B, and an c approxmately constant shft F c B. Ths also explans how c the edge nformaton s preserved n the α-channel, and why the α-channel s a normalzed descrpton of the mage edge. F c B c n the denomnator serves as the normalzaton factor. More mportantly, from Eqn. 15, we have that the followng equaton holds locally: α G = B c 1 F c B c Ic G, (16) where α G and I c G are the SoftCuts terms defned by Eqn. 6 on the α-channel and the sngle color channel, and the constant term s removed due to the smoothness assumpton. Eqn. 16 suggests a locally adaptve strategy for applyng the SoftCuts regularzaton term to the entre mage nstead of a sngle edge. More specfcally, for dfferent edges, applyng the

7 7 (a) (b) (c) (d) (e) (f) Fg. 5. An example of α-channel edge SR, (a) LR nput and the edge needed to be processed, (b) (c) F and B on two sdes of ths edge segment, (d) LR α channel, (e) HR α channel usng SoftCuts, (f) Smooth edge generated wth mattng equaton (Eqn 8). 1 edge adaptaton factor F c B c onto the color channel c can normalze the dfferent edges to the α-channel, whch always has a fxed contrast value of 1, nstead of a large number of possble values n the orgnal color channels. Thus by applyng ths adaptaton scheme, the same weghts are placed on the soft edge smoothness pror for dfferent edges,.e., the smoothness measures of all the level lnes on the α channel are equally weghted. Now we consder the entre nput mage wth ts multple layer decomposton defned n Eqn. 9. Based on the above dscusson, for each pxel p wth two non-zero components n ts α-channel mage descrpton a, the local adaptve factor on the color channel c s µ c 1 p = L c,p (17) Lc j,p, where and j are the layer ndces for those two nonzeros components. Snce the alpha mattng technque does not provde a soluton for mage layers L, alternatvely, from Eqn. 16, we use the followng equaton as an estmator of the above adaptve factor µ c p = a p, (18) Ip c 2 + γ where γ s a small postve number to avod numercal problems, s the l norm, whch returns the largest absolute value of the α-channel gradent among all layers. Snce most pxels do have one or two non-zero α components [7], the above normalzaton factor can successfully normalze most part of the mage n the same way as n Eqn. 16, and also gves reasonable results for other parts, as shown n the experments. Due to the smoothness assumpton of mage layers, µ c should also be smooth. To suppress mage nose, n practce, we convolve the weght map gven by Eqn. 18 wth a Gaussan kernel (σ = 1.8 n our experments). Please notce that gven an LR nput mage, the resultng alpha descrpton s also n LR. The LR adaptve factor obtaned by Eqn. 18 s then up-sampled wth bcubc nterpolaton to get the HR adaptve factor. Here, bcubc nterpolaton does not ntroduce artfacts due to the smoothness property of the adaptve factor. Besdes, snce dfferent samplng rates do not change the value of the RHS of Eqn. 18 at the same poston, so we do not need to scale the value durng up-samplng. Input LR mage I l and the scalng factor s. Output HR mage I h 1) Decompose I l by usng the spectral mattng algorthm [7] to get ts α-channel mage descrpton. 2) Compute the LR adaptve factor based on Eqn. 18 for each pxel n each color channel. 3) Compute the HR adaptve factor by up-samplng wth the bcubc nterpolaton and the scalng factor s. 4) For each color channel, optmze the adaptve Soft- Cuts objectve functon n Eqn. 20, to obtan the hgh resoluton result I h. Fg. 6. Adaptve SoftCuts algorthm for color mage super-resoluton. Fnally, the adaptve SoftCuts regularzaton term for color channel c {r, g, b} s defned as follows I c G = (p,q) N ( wpq µ pq I c p I c q ), (19) where µ pq = 1 2( µ c p +µ c q), and µ c p and µ c q are the local adaptve factors defned n Eqn. 18. The fnal objectve functon for adaptve SoftCuts on each channel c s I h,c ( = arg mn I l,c (I c G) 2 I c 2 + λ I c ) G. (20) Smlar teratve method as n Sec. V-A s appled for optmzaton. The entre algorthm s summarzed n Fg. 6. Applyng the adaptve SoftCuts regularzaton term to each color channel provdes a unfed soluton by processng all mage edges smultaneously. The relance on the success of the edge extracton algorthms and hard constrant selecton for mattng are avoded. The benefts, however, of performng SR wth the α-channel edge descrpton s stll materalzed. VI. EXPERIMENTS In ths secton, we frst present the SR results of applyng SoftCuts on sngle channel to demonstrate ts ablty to generate smooth edges, and then show the results by usng the adaptve SoftCuts method for sngle color mage SR.

8 8 (a) (b) (c) (d) Fg. 7. (a) LR nput mage, (b), (c), (d) are the SR results ( 3) wth soft edge smoothness pror when n G = 2, 4, 12 respectvely (λ = 0.01). (a) (b) (c) Fg. 10. SR results ( 3) wth soft edge smoothness pror; 1 st column: LR nputs; 2 nd column: SR results (λ = 0.01, n G = 12); 3 rd column: bcubc nterpolaton. (d) (e) (f) Fg. 8. Comparson of SR results by soft edge smoothness pror wth dfferent parameters ( 3), (a) LR nput mage (20 20), (b) λ = 0.01, n G = 12, (c) λ = 0.001, n G = 12, (d) bcubc nterpolaton, (e) λ = 0.01, n G = 2, (f) λ = 0.1, n G = 12. A. Sngle channel SR by SoftCuts Fgure 7 shows a proof-of-concept experment, llustratng the necessty of usng hgher order neghborhood. Jaggy effects can be observed for small n G, especally n (b) for a 4- neghborhood system. There are some 45 o artfacts n (c), snce 8-neghborhood system s used for t. The soft edge s much smoother n (d) wth n G = 12. Fgure 8 shows the result comparson of dfferent parameter settngs wth an LR con mage (con mage SR s also studed n [18]). Larger n G s appled n (b) than n (e), thus smoother edges are produced. In (c), a smaller λ s used than n (b), thus a smaller weght s placed on the smoothness pror. Ths makes the result look over-sharpened on hgh contrast edges, whle a better result s obtaned n other parts (such as the foot). In (f), a larger λ s used than n (b), the edge smoothness pror s over-weghted. All boundares are very smooth, but the result s very blurry. The effect of the parameters can be summarzed as follows: (1) a larger n G wll produce smoother boundares, and s also more computatonal expensve. In all of the later experments, n G s set equal to 12, wth the neghborhood system shown n Fg. 2. (2) The value of λ s crtcal. As revealed by Eqn. 18, the desrable weght should consder both mage gradent and α-channel gradent. Besdes, the flter G n the generaton model (Eqn. 10) also nfluences the qualty of the result. However, estmatng G s beyond the scope of ths paper. We fx t as a Gaussan flter wth σ = 1.4 throughout ths paper. Fgure 9 shows an example of how each term n the objectve functon (n Eqn 11) changes durng the teraton (Eqn. 12). In the frst several steps (about 20 n ths example), Fg. 11. Vsualzaton of the adaptaton weght. From left to rght: one LR color channel, LR adaptaton weghts for ths channel (large values of weghts are represented by brghter ntensty value), color LR nput mage, and result of the proposed adaptve SoftCuts method. the change of the reconstructon error domnates, and the Soft cut metrc may ncrease. After that, although the reconstructon error can hardly be further reduced, the result stll looks jaggy as shown n (e). The Soft cut metrc s further reduced by the teraton, and results n the fnal result wth smooth and sharp edges n (h). Addtonal results are shown n Fg. 10. The resultng edges are smooth and wth sharp transtons, even when the qualty of the LR nput mages s very low. However, for natural color mages lke the one n the thrd row of Fg. 10, although much smoother edges are obtaned, the mage looks unnatural. Some subtle edges are smoothed out. As dscussed, the reason s that t s dffcult, or even mpossble, to fnd a sngle value of λ to address all edges n an mage. That s why we need to work on the α channel to normalze dfferent edges, and further develop the proposed adaptve SoftCuts method. B. Adaptve SoftCuts for color mage SR Fgure 11 dsplays the adaptaton weghts for one color channel. Heaver weghts are placed on low contrast edges (e.g., those n the regons of the hat and nose), and smaller weghts are placed on hgh contrast edges. Thus the nfluence of the smoothness pror on dfferent edges s balanced. Please also notce that a zero weght s assgned to smooth regons, thus some subtle mage fluctuaton n those regons could stll

9 Reconstructon error Soft cut metrc Total value # Iteraton # Iteraton # Iteraton (a) I l (I G) 2 2 (b) λ I G (c) I l (I G) λ I G (d) (e) (f) (g) (h) Fg. 9. An example of the teratve optmzaton process n Sec. V-A. The value of the reconstructon error, the Soft cut metrc, and the entre objectve functon durng the teraton process are plotted n (a), (b), and (c) respectvely. The LR nput mage s shown n (d), and the result after 20, 40, 60, and 200 teratons are shown n (e), (f), (g), and (h) respectvely. Fg. 12. Mattng components extracted by the spectral mattng algorthm. The orgnal mage s shown n Fg. 11. be present. Ths makes the result look much more natural than the result wth the non-adaptve SoftCuts method shown n Fg. 10. The ntermedate result from the spectral mattng decomposton s shown n Fgure 12, where the mage edge nformaton s clearly extracted. Fgure 13 shows an example for comparson wth other wdely used algorthms, ncludng bcubc nterpolaton, bcubc followed by unsharp maskng (Photoshop), and the backprojecton method n [4]. Fgure 14 shows a zoomed n mage patch. Blurry edges can be observed wth the result of bcubc nterpolaton. The unsharp maskng (a large radus of 10, s used to better mprove the mage contrast) method and the back-projecton method can ncrease the mage contrast, but the results stll look jaggy. Both blurry and jaggy artfacts are successfully removed by the proposed adaptve SoftCuts method, even for very fne mage structures. Fgure 15 compares the proposed method wth an exemplarbased algorthm [21]. The exstng exemplar-based methods can produce very sharp edges, but rely heavly on effectve tranng data. Compared wth t, smoother boundares can be archved by our method, thus makng the result look natural. Fgure 16 shows a zoomed n patch to better llustrate the effectveness of the proposed method. The mage boundares of our result are both smooth and wth sharp transtons at the same tme. Fgure 17 compares the proposed method wth some exstng commercal / free softwares desgned specfcally for mage reszng, ncludng IrfanVew [40] Lanczos Interpolator, VSO [41] Image Reszer, Genune Fractals 5.0 [42], and PhotoZoom Pro 2 [43]. A close-up vew s also shown. Image edges n our result looks smooth, sharp, and natural. Addtonal results are shown n Fg Varous mages are tested, and vsually appealng results are obtaned by the adaptve SoftCuts algorthm. Please notce that the same set of parameters are appled for all of these mages, except that the weght of the regularzaton term λ s set equal to 0.01 for natural mages, and 0.02 for graphc mages. The reason s that the graphc mages usually have much smoother edges, thus a larger weght should be placed on the regularzaton. Quanttatvely, the RMS error (root-of-mean-square error) for bcubc nterpolaton, back-projecton, and the proposed SoftCuts method s shown n Table I. The back-projecton method [4] s effectve n reducng the RMS error by enforcng the reconstructon constrant. The RMS error for our algorthm s roughly the same as the back-projecton method, snce a smlar reconstructon constrant s enforced. However, the back-projecton algorthm ams at mnmzng only the data fdelty term, whch s reflected n the RMS error. Therefore, n certan cases, the back-projecton algorthm results n a smaller RMS error, although the vsual qualty of the results s not as good as the one provded by the proposed algorthm. Due to the addtonal soft edge smoothness pror, some mprovement s observed for mages wth smooth and sharp edges, such as Zebra, Fonts, and Mckey. The ERMS (edge RMS) errors are also presented, snce mage edges are more mportant for vsual percepton. Smlar observaton can be made wth the ERMS error. For complexty, we run our experments on a PIV3.4G PC wth 1G RAM by Matlab mplementaton. For an LR nput mage of sze pxels, the spectral mattng algorthm for mage decomposton takes 120 seconds, and the adaptve SoftCuts method takes 35 seconds for 30 teratons. 1 More on

10 10 TABLE I ERROR COMPARISON FOR BICUBIC INTERPOLATION, BACK-PROJECTION [4], AND THE PROPOSED ALGORITHM (FOR EACH BOX WITH TWO NUMBERS, THE FIRST IS THE RMS ERROR, THE SECOND IS THE ERMS ERROR). Lena Boy Zebra Monarch Peppers Fonts Mckey Bcubc BP [4] Ours (a) (b) (c) Fg. 15. Comparson results wth exemplar-based methods ( 4): (a) LR nput, (b) result n [21], (c) result of the proposed method. (a) (b) (c) Fg. 16. A close-up vew of the comparson results n Fg. 15: (a) LR nput, (b) result n [21], (c) result of the proposed method. The selected mage patch s hghlghted n Fg. 15(b) by a blue box. VII. CONCLUSION In ths paper, a novel sngle mage super-resoluton algorthm s proposed. A soft edge smoothness measure s defned on a large neghborhood system, whch s an approxmaton of the average length of all level lnes n the mage. To extend ths method to natural color mage SR, an adaptve SoftCuts method s proposed based on a novel α-channel mage descrpton. It enables a unfed treatment of edges wth dfferent contrasts on the α channel. Promsng results for a large varety of mages are obtaned by ths algorthm. thus and From Eqn. 5, we have APPENDIX A PROOF OF THEOREM 2 l G = k l G = = k ( wk k N k χ l p χ l q ), ( wk χ l p χ l q ) N k N k ( wk χ l p χ l q ), χ l p χ l q = #{ χ l p χ l q } = n S d p S d q, resultng n l G = ( wk n Sp d S d ) q k N k = n ( wk Sp d S d ) q N k k = n S d G (from Eqn. 6). Thus from Theorem 1, we have S d G = 1 l G 1 l E. n n ACKNOWLEDGMENT Ths work was supported n part by Natonal Scence Foundaton Grants IIS REFERENCES [1] Y. Boykov and V. Kolmogorov, Computng geodescs and mnmal surfaces va graph cuts, n ICCV, [2] A. Katsaggelos, R. Molna, and J. Mateos, Super resoluton of mages and vdeo, Synthess Lectures on Image, Vdeo, and Multmeda Processng. Morgan & Claypool, [3] S. Chaudhur and J. Manjunath, Moton-Free Super-Resoluton. Sprnger, [4] M. Iran and S. Peleg, Moton analyss for mage enhancement: resoluton, occluson and transparency, JVCIP, [5] S. Baker and T. Kanade, Lmts on super-resoluton and how to break them, IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 9, pp , [6] Z. Ln and H.-Y. Shum, Fundamental lmts of reconstructon based super-resoluton algorthms under local translaton, IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 1, pp , [7] A. Levn, A. Rav-Acha, and D. Lschnsk, Spectral mattng, n CVPR, [8] M. Elad and A. Feuer, Restoraton of sngle super-resoluton mage from several blurred, nosy and down-sampled measured mages, IEEE Trans. Image Process., vol. 6, no. 12, pp , [9] S. Farsu, M. D. Robnson, M. Elad, and P. Mlanfar, Fast and robust multframe super resoluton, IEEE Trans. Image Process., vol. 13, no. 10, pp , [10] M. F. Tappen, B. Russell, and W. T. Freeman, Explotng the sparse dervatve pror for super-resoluton and mage demosacng, n IEEE Workshop on Statstcal and Computatonal Theores of Vson, [11] S. D. Babacan, R. Molna, and A. K. Katsaggelos, Total varaton super resoluton usng a varatonal approach, n ICIP, [12] S. Farsu, M. Elad, and P. Mlanfar, Mult-frame demosacng and super-resoluton of color mages, IEEE Trans. Image Process., vol. 15, no. 1, pp , [13] J. Allebach and P. W. Wong, Edge-drected nterpolaton, n ICIP, [14] X. L and M. Orchard, New edge-drected nterpolaton, IEEE Trans. Image Process., vol. 10, no. 10, pp , [15] D. D. Muresan, Fast edge drected polynomal nterpolaton, n ICIP, [16] B. S. Morse and D. Schwartzwald, Image magnfcaton usng level set reconstructon, n CVPR, [17] Y.-W. Ta, W.-S. Tong, and C.-K. Tang, Perceptually-nspred and edgedrected color mage super-resoluton, n CVPR, [18] V. Rabaud and S. Belonge, Bg lttle cons, n CVAVI, 2005.

11 11 (a) Fg. 13. (b) (c) (d) (e) Comparson results: (a) LR nput, (b) bcubc, (c) bcubc followed by unsharp maskng, (d) back-projecton [4], (e) proposed method ( 3). (a) (b) (c) (d) (e) Fg. 14. A close-up vew of the comparson results n Fg. 13: (a) LR nput, (b) bcubc, (c) bcubc followed by unsharp maskng, (d) back-projecton [4], (e) proposed method. The selected mage patch s hghlghted n Fg. 13(b) by a blue box. Fg. 17. (a) LR nput mage (b) IrfanVew Lanczos Interpolaton (c) VSO Image Reszer (d) Genune Fractals 5.0 (e) PhotoZoom Pro 2 (f) Our result Comparson wth some exstng commercal / free softwares desgned specfcally for mage reszng.

12 12 Fg. 18. Addtonal results wth the proposed adaptve SoftCuts method. For each par of mages, the upper one s the LR nput, and the lower one s our result ( 3). [19] R. Nakagak and A. K. Katsaggelos, A vq-based blnd mage restoraton algorthm, IEEE Trans. Image Process., vol. 12, no. 9, pp , [20] W. T. Freeman, T. R. Jones, and E. C. Pasztor, Example-based superresoluton, IEEE Computer Graphcs and Applcatons, [21] W. T. Freeman, E. Pasztor, and O. Carmchael, Learnng low-level vson, IJCV, vol. 40, no. 1, pp , [22] C. Lu, H.-Y. Shum, and C.-S. Zhang, A two-step approach to hallucnatng faces: Global parametrc model and local nonparametrc model, n CVPR, [23] C. M. Bshop, A. Blake, and B. Marth, Super-resoluton enhancement of vdeo, n Proc. Artfcal Intellgence and Statstcs, [24] D. Kong, M. Han, W. Xu, H. Tao, and Y. Gong, Vdeo super-resoluton wth scene-specfc prors, n BMVC, [25] Q. Wang, X. Tang, and H. Shum, Patch based blnd mage super resoluton, n CVPR, [26] J. Sun, N. Zheng, H. Tao, and H. Shum, Image hallucnaton wth prmal sketch prors, n CVPR, [27] H. Chang, D. Yeung, and Y. Xong, Super-resoluton through neghbor embeddng, n CVPR, [28] S. Da, M. Han, W. Xu, Y. Wu, and Y. Gong, Soft edge smoothness pror for alpha channel super resoluton, n CVPR, [29] Y. Boykov and V. Kolmogorov, An expermental comparson of mncut/max-flow algorthms for energy mnmzaton n vson, IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 9, pp , [30] Y. Boykov, O. Veksler, and R. Zabh, Fast approxmate energy mnmzaton va graph cuts, IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 11, pp , [31] L. Rudn, S. Osher, and E. Fatem, Nonlnear total varaton based nose removal algorthms, Physca D, vol. 60, pp , [32] T. F. Chan, S. Osher, and J. Shen, The dgtal tv flter and nonlnear denosng, IEEE Trans. Image Process., vol. 10, no. 2, pp , [33] S. Babacan, R. Molna, and A. Katsaggelos, Parameter estmaton n tv mage restoraton usng varatonal dstrbuton approxmaton, IEEE Trans. Image Process., vol. 17, no. 3, pp , [34] A. Levn, D. Lschnsk, and Y. Wess, A closed form soluton to natural mage mattng, IEEE Trans. Pattern Anal. Mach. Intell., [35] J. Ja, Sngle mage moton deblurrng usng transparency, n CVPR, [36] Q. Shan, W. Xong, and J. Ja, Rotatonal moton deblurrng of a rgd object from a sngle mage, n ICCV, [37] S. Da and Y. Wu, Moton from blur, n CVPR, [38] E. P. Bennett, M. Uyttendaele, C. L. Ztnck, R. Szelsk, and S. B. Kang, Vdeo and mage bayesan demosacng wth a two color mage pror, n ECCV, [39] J. Wang, M. Agrawala, and M. F. Cohen, Soft scssors : An nteractve tool for realtme hgh qualty mattng, ACM Trans. on Graphcs (Proc. SIGGRAPH), [40] Irfanvew, [41] Vso mage reszer, [42] Genune fractals 5.0, [43] Photozoom pro 2, Shengyang Da (S 05) s currently a Ph.D. canddate n the Electrcal Engneerng and Computer Scence department of Northwestern Unversty. Hs research nterests nclude mage/vdeo processng, computer vson, and machne learnng. He dd summer nterns at NEC Laboratores Amerca (Cupertno, CA), Mcrosoft Research (Redmond, WA), and Google Research (Mountan Vew, CA) n 2006, 2007, and 2008, respectvely. He receved hs B.S. and M.S. degrees from the Electrcal Engneerng department of Tsnghua Unvresty, Bejng, Chna, n 2001 and 2004 respectvely. He receved the Outstandng Graduate Student Fellowshp at Tsnghua Unversty n 2004, and the Everly Fellowshp at Northwestern Unversty n He s a Student member of the IEEE.

13 13 Me Han receved her B.S. and Ph.D. n Computer Scence from Tsnghua Unversty, Chna, n 1992 and 1995, and Ph.D. n Robotcs from Carnege Mellon Unversty n From 2001 to 2007, she was a research staff member at NEC Laboratores Amerca. She has been a research scentst at Google snce August Her research nterests nclude computer vson, computer graphcs, mage and vdeo analyss, vdeo survellance, multmeda data mnng, and machne learnng. She served as a commttee member on ECCV 2002 workshop on Vson and Modellng of Dynamc Scenes and a sesson char on Internatonal Conference on Image Processng We Xu receved hs B.S. degree from Tsnghua Unversty, Bejng, Chna, n 1998, and M.S. degree from Carnege Mellon Unversty (CMU), Pttsburgh, USA, n From 1998 to 2001, he was a research assstant at the Language Technology Insttute at CMU. In 2001, he joned NEC Laboratores Amerca workng on ntellgent vdeo analyss. Hs research nterests nclude computer vson, mage and vdeo understandng, machne learnng and data mnng. Aggelos K. Katsaggelos (S 80-M 85-SM 92-F 98) receved the Dploma degree n electrcal and mechancal engneerng from the Arstotelan Unversty of Thessalonk, Thessalonk, Greece, n 1979, and the M.S. and Ph.D. degrees n electrcal engneerng from the Georga Insttute of Technology, Atlanta, n 1981 and 1985, respectvely. In 1985, he joned the Department of Electrcal and Computer Engneerng at Northwestern Unversty, Evanston, IL, where he s currently a Professor. He was the holder of the Amertech Char of Informaton Technology ( ). He s also the Drector of the Motorola Center for Communcatons and a member of the Academc Afflate Staff, Department of Medcne, Evanston Hosptal. He s the edtor of Dgtal Image Restoraton (Sprnger-Verlag, 1991), coauthor of Rate-Dstorton Based Vdeo Compresson (Kluwer, 1997), co-edtor of Recovery Technques for Image and Vdeo Compresson and Transmsson (Kluwer, 1998), and co-author of Super-Resoluton for Images and Vdeo (Claypool, 2007) and Jont Source-Channel Vdeo Transmsson (Claypool, 2007), and the co-nventor of 12 patents. Dr. Katsaggelos has served the IEEE and other Professonal Socetes n many capactes; he was, for example, Edtor-n-Chef of the IEEE Sgnal Processng Magazne ( ), a member of the Board of Governors of the IEEE Sgnal Processng Socety ( ), and a member of the Publcaton Board of the IEEE PROCEEDINGS ( ). He s the recpent of the IEEE Thrd Mllennum Medal (2000), the IEEE Sgnal Processng Socety Mertorous Servce Award (2001), an IEEE Sgnal Processng Socety Best Paper Award (2001), an IEEE Internatonal Conference on Multmeda and Expo Paper Award (2006), and an IEEE Internatonal Conference on Image Processng Paper Award (2007). He s a Dstngushed Lecturer of the IEEE Sgnal Processng Socety ( ). Yng Wu (SM 06) receved the B.S. from Huazhong Unversty of Scence and Technology, Wuhan, Chna, n 1994, the M.S. from Tsnghua Unversty, Bejng, Chna, n 1997, and the Ph.D. n electrcal and computer engneerng from the Unversty of Illnos at Urbana-Champagn (UIUC), Urbana, Illnos, n From 1997 to 2001, he was a research assstant at the Beckman Insttute for Advanced Scence and Technology at UIUC. Durng summer 1999 and 2000, he was a research ntern wth Mcrosoft Research, Redmond, Washngton. In 2001, he joned the Department of Electrcal and Computer Engneerng at Northwestern Unversty, Evanston, Illnos, as an assstant professor. He s currently an assocate professor of Electrcal Engneerng and Computer Scence at Northwestern Unversty. Hs current research nterests nclude computer vson, mage and vdeo analyss, pattern recognton, machne learnng, multmeda data mnng, and human-computer nteracton. He serves as assocate edtors for IEEE Transactons on Image Processng, SPIE Journal of Electronc Imagng, and IAPR Journal of Machne Vson and Applcatons. He receved the Robert T. Chen Award at UIUC n 2001, and the NSF CAREER award n He s a senor member of the IEEE. Yhong Gong receved hs B.S., M.S., and Ph.D. degrees n Electronc Engneerng from Unversty of Tokyo n 1987, 1989, and 1992, respectvely. He then joned the Nanyang Technologcal Unversty (NTU) of Sngapore, where he worked as an assstant professor n the School of Electrcal and Electronc Engneerng. Between June 1996 and December 1998, he worked for Robotcs Insttute, Carnege Mellon Unversty as a project scentst, and was a prncpal member of both the Informeda Dgtal Vdeo Lbrary project and the Experence- On-Demand project funded by NSF, DARPA and other governmental agences. In 1999, he joned NEC Laboratores Amerca, where he bult the multmeda content analyss team from scratch. In 2006, he became the head of the slcon valley branch of the labs. Hs research nterests nclude computer vson, multmeda content analyss, and machne learnng.