Multiple Image Thumbnailing

Transcription

1 Multple Image Thumbnalng Ganlug Coccaa, Ramondo Schettna a DISCo - Dpartmento d Informatca Sstemstca e Comuncazone, Unverstà degl stud d Mlano-Bcocca, Vale Sarca 336, Mlano, Italy ABSTRACT We have desgned a new self-adaptve mage croppng algorthm that s able to detect several relevant regons n the mage. These regons can then be sequentally proposed as thumbnals, to the user accordng to ther relevance order, thus allowng the vewer to vsualze the relevant mage content and eventually to dsplay or prnt only those regons n whch he s more nterested n. The algorthm explots both vsual and semantc nformaton. Vsual nformaton s obtaned by a vsual attenton model, whle semantc nformaton relates to the detecton and recognton of partcularly sgnfcant objects. In ths work we concentrate our attenton on the two common objects found n personal photos, such as face and skn regons. Examples are shown to llustrate the effectveness of the proposed method. Keywords: croppng, vsual attenton model, mage content 1. INTRODUCTION The capabltes of consumer devces and archtectures are undergong consderable progress. Now we all can easly take and share personal photos taken wth moble phones or prnt mages by pushng few buttons anywhere and anytme wthout the requrement of a computer or a full-sze vdeo dsplay. For example, the most common devces are moble phones, prnters and dgtal players. Many other devces are beng produced that permt storng dgtal photos n objects of everyday use, such as dgtal photo frames, key rngs, pen cups and other smlar objects. Accessng the mage content nformaton s thus a crucal challenge for these devces, whch usually come wth small form factor dsplays (Fgure 1). When vewng large mages on these knd of dsplays, mportant mage content can be mssed ether by the fact that the mage s shrnked to ft the dsplay and consequently detals are lost or by the fact that a specfc regon s zoomed n the mage, possbly dscardng other mportant regons. Moreover, photos may contan unwanted parts that dstract from the man subject (Fgure 2). Although the defnton of mportant regons s subjectve and context dependent, several approaches have been developed to cope wth the problem from dfferent perspectves usng computer vson approaches. a b d c e f g Fgure 1. Examples of devces wth dsplays of varous szes. a) photo kosk, b) moble phone, c) dgtal photo frame, d) dgtal key rng, e) multfuncton prnter, f) pen cup, g) dgtal player. a b c Fgure 2. Examples of photos that requre an ntellgent selecton of relevant regons. Dgtal Photography VI, edted by Francsco Ima, Ntn Sampat, Feng Xao, Proc. of SPIE-IS&T Electronc Imagng, SPIE Vol. 7537, 75370S 2010 SPIE-IS&T CCC code: X/10/$18 do: / SPIE-IS&T/ Vol S-1 d

2 The state-of-the-art methods can be dvded nto two broad categores: crop-based and retarget-based. The crop-based category comprses those methods that extract a smaller, rectangular regon from the orgnal mage wth the most salent nformaton. Two basc strateges for dentfyng the best croppng regon are found n the lterature: one based on the exstence of specfc objects and smple compostonal rules 1, 2, 3 and the other that consder the best croppng regon as the soluton of an optmzaton problem 4, 5, 6. In the retarget-based category we can fnd methods that try to systematcally remove or de-emphasze non sgnfcant parts of the mage, whle preservng the most relevant ones. Then the sgnfcant pxels are collected to reconstruct a whole mage. In ths case, the result of the algorthms s a modfed verson of the orgnal mage. We can group the dfferent approaches nto three broad categores: regon or object based 7, 8, 9 where the 10, 11, 12, 13, non sgnfcant regons or objects are de-emphaszed (removed, transformed, sampled, etc ) and pxel-based where an energy functon s used to determne the mportance (relevance) of each pxel n the mage. A mxed approach can be found n 14 where both objects and pxel relevance are used to determne an optmal browsng path to automatcally navgate through the mage content. In ths paper we descrbed a new self-adaptve mage croppng algorthm that by explotng both vsual and semantc nformaton s able to detect relevant regons n the mage. Instead of extractng only one regon or warpng the full-szed mage to ft a gven dsplay sze, we chose to select and extract several relevant regons. These regons, adapted to best ft the dsplay sze, can be sequentally shown to the user accordng to ther relevance order n the form of small thumbnal mages. Ths allows the user to browse the mage content by vsualzng the relevant regons contaned wthn t. The browsng s as smple as movng from the current thumbnal to the next or prevous one. If one or more regons are pleasant and nterestng enough for the user, they can be retaned, prnted or coped for further use. We ntroduce the algorthm framework n Secton 2, the buldng blocks of our algorthm are descrbed n Secton 3 and the results of the thumbnalng algorthm on several sgnfcant mages are reported and dscussed n Secton 4. Secton 5 concludes the paper wth the ongong works to further mprove the proposed algorthm. 2. MULTIPLE IMAGE THUMBNAILING The multple mage thumbnalng algorthm s based on both salency regon detecton and semantc regon detecton. The semantc regons consdered n the present work are lmted to the face and skn regons. However, the algorthm s framework can be easly extended n order to recognze and ntegrate a larger number of semantc concepts; for example, as extracted n 15. Fgure 3 shows the framework of our mage thumbnalng algorthm. Image Image Analyss Mult-Level Salency Map Face Map Skn Map Map Fuson Relevant Regons Detecton Envronment Constrants / Preferences Rankng Adaptaton Dsplay Fgure 3. The framework of the multple mage thumbnalng algorthm. In order to speed up the subsequent processng steps, whle mantanng an acceptable tradeoff between accuracy and speed, the mages are frstly pre-processed by scalng them to a default mage sze preservng the aspect rato. Ths SPIE-IS&T/ Vol S-2

3 allows us to speed up the mage processng, and more mportant, to lmt the range of the szes of the relevant semantc regons to be subsequently detected. After the pre-processng stage, the mage undergoes the analyss phase whch conssts of the detecton of the pctorally and semantcally pxel regons. Regons pctorally relevant are deleted by applyng a salency map detecton algorthm based on the dea of Yu-Fe Ma, and Hong-Jang Zhang 16. The algorthm s modfed to detect the salency regons at three levels of detal to take nto account the dfferent szes of the relevant areas. Semantc relevant regons are detected, explotng both a face detecton algorthm nspred by the Vola and Jones algorthm 17 and a skn regon detecton algorthm. The skn regons are used both for dentfyng relevant regons common n certan mage categores (.e. vacaton) where the user s also nterested n larger relevant regons and for valdatng the face regons ncreasng the precson of the face detecton algorthm. The results of the three detectors are maps ndcatng the relevance of each pxel n the mage. These maps are weghted, processed and then combned together nto a fnal relevant map. Dependng on applcaton constrants or preferences, the weghts can be set n order to enhance or dmnsh the mportance of certan features n the overall relevance map. Based on the relevance maps canddate relevant regons are detected and extracted. No assumptons are made on the sze or aspect rato of these regons. These regons are then ranked accordng to a relevance ndex and those regons exceedng a gven threshold (selected for example based on the user s preferences) are selected as the set of relevant regons to be presented to the user. Before the dsplay phase each relevant regon s processed n order to be pleasng when vewed on the current dsplay. Ths phase, called adaptaton, crops the regon correspondng to the relevant one from the orgnal full-szed mage adjustng ts sze, poston and aspect rato va a set of mage transformatons such as rotaton, resze, trm, and enlargement. The objectve of the adaptaton phase s to ft the cropped mage on the target dsplay whle preservng the contaned nformaton. Fnally, the adapted mages are sequentally shown to the user accordng to ther relevance ndex, startng from the most relevant one down to the least relevant one. 3. BUILDING BLOCKS 3.1 Mult-Level Salency Map Pctorally relevant regons are selected by applyng a salency map detecton algorthm based on the dea of Yu-Fe Ma, and Hong-Jang Zhang 16. The basc algorthm dvdes the mage nto small rectangular tles. At each tle, a contrast score s computed from the dfferences of average colors between the gven tle and ts neghbor s tles. The contrast score expresses the salency of the pxels n the tle. The contrast scores of all the tles defne the salency map of the mage. The sze of the tles, and the sze of the neghborhoods determne the dmensons of the salent areas that can be detected. The basc, sngle scale algorthm has been extended by computng three dfferent salency maps, each amed at a partcular level of detal. Snce the mage szes have been fxed n the mage preprocessng stage, only the sze of the neghborhood has to be modfed. We compute the three salency maps usng neghborhoods of ncreasng sze, amed at detectng small, medum and large salent areas. These salency maps are then fltered and combned together nto a sngle, normalzed map. An example of the mult-level salency map computaton s shown n Fgure 4. Low detals + Medum detals Fnal Map Hgh detals Fgure 4. Example of mult-level salency map computaton. SPIE-IS&T/ Vol S-3

4 The output of the salency detecton module s a salency map SM, of the same sze (W H) of the reszed mage, where the pxel values are proportonal to ther salency. where { S } W j H SM = 1 1 (1) S s the map value at poston (, j) wth 0 S Face Map We have adopted a varaton of the face detector proposed by Vola and Jones 17. The face detector algorthm uses a mult-scale, mult-stage classfer, whch operates on mage ntensty nformaton. It uses an over-complete set of Haarlke features. In order to compute these features very rapdly at many scales, an ntegral mage representaton s ntroduced. The ntegral mage can be computed from an mage usng a few operatons per pxel. Usng the ntegral mage, the Haar-lke features can be computed at any scale or locaton n constant tme. Wthn any mage sub-wndow, the total number of Haar-lke features s very large. In order to ensure fast classfcaton, the learnng process must exclude a large majorty of the avalable feature, and focus on a small set of crtcal ones. Feature selecton s acheved usng the AdaBoost learnng algorthm by constranng each weak classfer only to sngle features. The boostng algorthm s used to successvely tran more complex classfers n a cascade structure whch ncreases the speed of the detector by quckly dscardng background regons, whle spendng more computaton on promsng face-lke regons. Snce there may be several false postve face detectons, we explot the fact that the algorthm can detect the same face at dfferent scales to defne a confdence ndex (C). Thus, we keep track of how many tmes a face s detected and normalze t over the total number of face regons detected to obtan the confdence ndex. The ndex s used to flter out the false postve faces and to wegh the relevance of each face regon. Fgure 5 shows a typcal result of the face detecton algorthm. The boxes dentfy the face detected by the adopted algorthm. Ther wdth s proportonal to the number of tmes the algorthm detects a face n that locaton. The red boxes represent false face regons dscarded by also analyzng the percentage of skn pxels present n the regon (see below). The fnal true face regons are ndcated by the black boxes. A face map s then created from the face regons and passed to the map fuson module. Fgure 5. Example of face detecton algorthm. The black boxes dentfy the fnal face regons. The output of the face detecton module s a face map, FM, that can be descrbed by a set of f face regons, coupled wth ther confdence ndex: where FM FR s the th face regon; R s the boundng box of FR ; C s the confdence ndex of FR. { FR } = {( R C )} f =, 1 (2) SPIE-IS&T/ Vol S-4

5 3.3 Skn Map Many dfferent methods for dscrmnatng between skn pxels and non-skn pxels are avalable. The smplest and most often appled method s to buld an explct skn cluster classfer whch expressly defnes the boundares of the skn cluster n certan color spaces. The underlyng hypothess of methods based on explct skn clusterng s that skn pxels exhbt smlar color coordnates n a properly chosen color space. Ths type of bnary method s very popular snce t s easy to mplement and does not requre a tranng phase. For ths applcaton we have adopted a method based on the YCbCr color space, developed by Cha and Ngan 18. A skn color map s derved and used on the chromnance components of the nput mage to detect pxels that appear to be skn. The algorthm then employs a set of regularzaton processes to renforce those regons of skn-color pxels most lkely to belong to facal regons. We consder only ther color segmentaton step here. The choce of the skn cluster boundares s a crtcal matter and depends on the task at hand. In Gasparn et al. 19 several skn detecton algorthms are examned, and usng a genetc algorthm the optmal skn cluster boundares are determned under dfferent assumptons. We explot the skn cluster boundares n the work of Gasparn et al. resultng from the genetc algorthm optmzaton for the algorthm of Cha and Ngan. The authors show that the computed boundares may favor hgh recall, hgh precson, or a reasonable tradeoff between the two. In the present work we selected the tradeoff boundares as the default choce snce t can be used both as a helper n the valdaton of true face regons as well as the detecton of relevant skn regons. The output f the skn detecton module s a bnary map KM: where { K } W j H KM = 1 K s the map value at poston (, j) wth { 0,1} 1 (3) K. 3.4 Map Fuson, Regon Detecton and Rankng The three maps computed n the mage analyss phase are collected and combned together to form the fnal relevance map (RM). The relevance r of the mage pxels at poston (, j) can be calculated as follows: { } r RM = r = w S + w F + w K W 1 j H SM FM KM 1 (4) where w SM, w FM, and wkm are postve weghts that balance the contrbuton of each map n the fnal relevance map and ther sum equals to one. The weghts n Equaton 4 can be chosen accordng to the applcaton or user preferences, and thus the defnton of relevance can be shfted, for example, to take more nto account face and skn than the general salency. F s the value of the face map at poston (, j) and can be computed from the set of face boundng boxes and confdences of the face map FM. The F can be computed n dfferent ways. For example, we can weght each face regon accordng to the confdence ndex, and thus the relevant regons eventually contanng faces can be ranked dfferently usng Equaton 5. If we are nterested only n the presence of the faces whle dscardng a rankng among them so that a person may be detected n hs/her entrety, Equaton 6 can be used nstead. F C = 0 k f (, j) R otherwse k 1 k f (5) F 1 = 0 f (, j) R otherwse k 1 k f (6) Once the relevance map s computed t s bnarzed usng Equaton 7, wth an applcaton-dependent threshold T b obtanng an mage map RM. The parameter k s used to determne the degree of relevance to take nto account and can be user or task-defned. SPIE-IS&T/ Vol S-5

6 1 f r > k Tb RM '= { r' } r' = 1 W 1 j H (7) 0 otherwse The connected components of RM are then extracted, and those larger than a predefned sze correspond to the canddate relevant regons. To rank the canddate regons, we need to assocate a relevance ndex to each regon. The ndex can be computed takng nto account the salency values, the amount of skn pxels and the presence and/or confdences of the faces wthn the regon. If a regon possesses all three of these attrbutes t s ranked among the top postons. For certan tasks we may want to always nclude the faces n the fnal salent regons. These canddate regons wth the assocated relevance ndex can be passed drectly to the adaptaton module or further fltered by removng those regons havng a relevance ndex below a gven threshold. The output of the map fuson, regon detecton and rankng processng blocks s the set of ranked relevant regons to be dsplayed on the devce. 3.5 Adaptaton Before the selected regons can be presented to the user, they must undergo the adaptaton phase that modfes ther sze and poston n order to fully explot the dsplay sze. Thus, for each regon the adaptaton module must: Maxmze the nformaton dsplayed Correct the regon aspect rato Mnmze the overall unused dsplay space Frstly the coordnates of the regons dentfed at the prevous step are mapped to the correspondng coordnates n the full-szed mage. Then, for each regon to satsfy the above mentoned requstes, we use a set of processng rules based on smple mage transformatons. Dependng on the geometrc characterstcs of the regon to be processed, a set of these transformatons s adopted to adjust the fnal output. The transformatons are: where { T, T, T T } T =, RO EN TR T RO Rotaton maxmzes the dsplayed surface based on the aspect rato of the regon; T EN Enlarge adds rows (columns) of pxels to the mage ncreasng the nformaton dsplayed; T TR Trm removes rows (columns) of pxels from the regon to preserve the aspect rato; T Resze fts the regon to the dsplay area mantanng the aspect rato. RS The rotaton transformaton s logcally useful when vewng the mage on a moble dsplay n that we can take full advantage of the fact that the dsplay can be freely rotated to vew the mage up-sde. For fxed dsplays such as those for example found on a prnter, the rotaton can be removed from the transformatons set. RS 4. RESULTS AND DISCUSSION Fgure 6 shows several results of our multple thumbnalng algorthm before the adaptaton phase has taken place. These results are obtaned by also ncludng the face regons n the set of relevance regons to be adapted and presented to the user. It can be seen that the algorthm effectvely select dfferent knd of relevant regons dependng on the mage content and complexty. The results n 6a, 6d, 6e, 6g and, 6n show dfferent relevant regons selected on the bass of the salency map, whle n the results n 6b, 6h, 6m and 6n the face regons are also detected and ncluded (although not all the faces are successfully detected). Fgure 6m s an example where, despte the fact that the algorthm faled to detect the face n regon 1 and the salency of regon 2, the overall result s stll acceptable, demonstratng the need to nclude several salent object detectons n the croppng strategy. If the mage does not contan faces or spatally separated salent SPIE-IS&T/ Vol S-6

7 regons, a sngle relevant regon s detected as n 6c, 6o 6p and 6q. The sze of the regon depends on the complexty of the underlyng mage. If the mage s almost the same wthn the mage, the algorthm selects almost the whole mage as the relevant regon as n 6q. In 6r, 6s and 6t three examples are shown where the algorthm fals to detect the proper relevant regons. a b d c e f g h m n o p q r s t Fgure 6. Examples of results of the multple mage thumbnalng algorthm. Fgure 7 shows the advantages n usng the adaptaton module on some sample mages. In these examples only one relevant regon s consdered. The frst column shows the orgnal mage, scaled to ft the dsplay area, whle mantanng the mage proportons. It can be seen that not all the dsplay area s used and small detals are dffcult to read. The second column shows a relevant regon, scaled to ft the dsplay area. Agan, not all the area s used but the relevant detals are more readable. The thrd column shows the adapted relevant regon. It can be seen that all the dsplay area s used, and the mage transformatons are able to dsplay a more appealng mage. In the frst row, the advantages of the rotaton transformaton can be seen, when the mage to be dsplayed changes format from landscape to portrat. If the dsplay belongs to a moble phone that can be rotated, the resultng rotated mage can be dsplayed n a more appealng way. The second and thrd rows show two examples of the enlarged transformaton (vertcal and horzontal enlargement). The last row shows the effect of a trm operaton. The rows of pxels at the top and bottom of the mage are removed untl the correct aspect rato of the mage s obtaned. Usually all the adapted mages undergo the resze transformaton, ether n the form of downscalng or up-scalng. SPIE-IS&T/ Vol S-7

8 Orgnal Image Scaled Regon Adapted Regon Transformatons Rotaton Trm Resze Enlarge Resze Enlarge Resze Trm Resze Fgure 7. Examples of the capabltes of the adaptaton module. 5. CONCLUSIONS In ths paper we descrbed a dfferent self-adaptve mage croppng algorthm, whch by explotng both vsual and semantc nformaton s able to detect relevant regons n the mage. These regons, adapted to best ft the dsplay sze, are sequentally shown to the user accordng to ther relevance order n form of small thumbnals. By browsng through these thumbnals, the vewer can grasp the relevant mage content and eventually dsplay or prnt only those regons n whch he s most nterested n. Snce no objectve evaluaton exsts on a task, we demonstrate the effectveness of our algorthm by presentng and dscussng several results. We plan to further extend our multple mage thumbnalng algorthm by pre-classfyng the mages before the algorthm s appled as n 20. Instead of usng the mage classfcaton to select a dfferent processng ppelne, the pre classfcaton phase wll allow us to automatcally apply specfc parameters (weghts, thresholds, etc ), learned off-lne n a tranng phase on the bass of the semantc contents of the mages. For example, f an mage contans people the relevance weghts n Equaton 4 can be automatcally selected to shft the relevance toward ths semantc concept. We are also nvestgatng how to expand the mage analyss phase by recognzng more relevance objects, such as pets, monuments, text, graphc sgns, etc REFERENCES 1. Hou X., Zhang L., "Thumbnal Generaton Based on Global Salency," Proc. of the Internatonal Conference on Cogntve Neurodynamcs, Advances n Cogntve Neurodynamcs, ICCN 2007, , (2007). 2. Suh, B., Lng, H., Bederson, B. B., and Jacobs, D. W., "Automatc thumbnal croppng and ts effectveness," Proc. of the 16th Annual ACM Symposum on User nterface Software and Technology, , (2003). 3. Santella A., Agrawala M., DeCarlo D., Salesn D., Cohen M., "Gaze-based nteracton for sem-automatc photo croppng," Proc. of the SIGCHI conference on Human Factors n computng systems, , SPIE-IS&T/ Vol S-8

9 4. Stentford, F. W. M., "Attenton based Auto Image Croppng," Workshop on Computatonal Attenton and Applcatons, ICVS, Belefeld, (2007). 5. Zhang M., Zhang L., Sun Y., Feng L., Ma W., "Auto croppng for dgtal photographs", Proc. IEEE Conference on Multmeda and Expo, (2005). 6. Luo J., "Subject Content-Based Intellgent Croppng of Dgtal Photos," Proc. IEEE Internatonal Conference on Multmeda and Expo, , (2007). 7. Lu, F., and Glecher, M., "Automatc Image Retargetng wth Fsheye-Vew Warpng", Proc. of the 18th annual ACM symposum on User nterface software and technology, , (2005). 8. Setlur V., Tacag S., Raskar R., Glecher M., Gooch B., "Automatc mage retargetng," Proc. of the ACM 4th nternatonal conference on Moble and ubqutous multmeda, vol. 154, 59-68, (2005). 9. Ren T., Lu Y., Wu G., "Image retargetng based on global energy optmzaton," Proc. IEEE Internatonal Conference on Multmeda and Expo - ICME2009, , (2009). 10. Srvastava A., Bswas K.K., "Fast content aware mage retargetng," Proc. IEEE Sxth Indan Conference on Computer Vson, Graphcs & Image Processng, , (2008). 11. Sh L., Wang J., Xu L., Lu H., Xu C., "Context salency based mage summarzaton," Proc. IEEE Internatonal Conference on Multmeda and Expo - ICME2009, , (2009). 12. Avdan, S., Shamr, A., "Seam Carvng for Content-Aware Image Reszng," In ACM Transactons on Graphcs, 26(3), SIGGRAPH 2007, (2007) 13. Ren T., Guo Y., Wu G., Zhang F., "Constraned Samplng for Image Retargetng," Proc. Internatonal Conference on Multmeda and Expo ICME2008, , (2008). 14. Lu, H., Xe X., Ma W-Y., Zhang H-J., "Automatc Browsng of Large Pctures on Moble Devces," Proc. of the eleventh ACM nternatonal conference on Multmeda, , (2003). 15. Cusano C., Cocca G., Schettn R., "Image annotaton usng SVM", Proc. SPIE Internet Imagng V, 5304, , (2004). 16. Ma Y. and Zhang, H-J. "Contrast-based mage attenton analyss by usng fuzzy growng", Proc. of the Eleventh ACM nternatonal Conference on Multmeda, , (2003). 17. Vola P., Jones M.J., "Robust real-tme face detecton", Int. Journal of Computer Vson, 57(2), , (2004). 18. Cha D., and Ngan K.N., "Face segmentaton usng skn colour map n vdeophone applcatons," IEEE Transactons on Crcuts and Systems for Vdeo Technology, 9(4), , (1999). 19. Gasparn F., Corchs S., Schettn R., "Recall or precson orented strateges for bnary classfcaton of skn pxels", Journal of Electronc Imagng, vol. 17(2), , 1-15, (2008). 20. Cocca G., Cusano C., Gasparn F., Schettn R., "Self-Adaptve Image Croppng for Small Dsplays," IEEE Transacton on Consumer Electroncs, 53(4), , (2008). SPIE-IS&T/ Vol S-9