A 2D to 3D Conversion Scheme Based on Depth Cues Analysis for MPEG Videos

Transcription

1 A to 3 Converson Scheme ase on epth Cues Analss for PEG eos Guo-Shang Ln, Cheng-Yng Yeh, e-chh Chen, an en-ung Le ept. of Computer Scence an Informaton Engneerng, a-yeh Unverst awan epartment of Electrcal Engneerng, atonal Chung Cheng Unverst awan Abstract In ths artcle, we propose a to 3 veo converson scheme for PEG veos. he ffcult for /3 converson problem les on epth estmaton/ assgnment wth nsuffcent nformaton. Our epth assgnment s base on the analses of multple cues, e.g., moton parallax, atmospherc perspectve, texture graent, lnear perspectve, an relatve heght, separatel for the foregroun objects an the backgroun area. o ft more kns of veos, the propose epth assgnment scheme s content-aaptve b segmentng a veo nto shots an classfng each of them nto three categores for fferent converson schemes. Subjectve experments show that the 3 stereo veo generate b usng our epth assgnment scheme an the epth Image ase Renerng (IR technque presents lttle fference to that create base on the epth groun truths.. IROUCIO Recentl 3 (specfcall stereoscopc mages are attractng more attenton n varous applcatons, such as multmea sstems, games, 3- broacastng, an vrtual realt []. hs s because 3 mages prove hgher realsm than conventonal mages. th the technologes of 3 veo capturng an 3- spla gettng more mature, the mportance of the 3 veos wll ncrease. owever, t s stll ffcult to get 3 veos n the real worl snce a 3 veo capturng evce s heav to carr aroun an nstalle. Atonall current 3 veo technologes are onl evelope an prove n the professonal fels. hs motvates us to convert exstng veos nto 3 stereoscopc ones so that the populart of 3 contents further pushes the progress of 3 veo nustr. he technque concernng the converson of veos to stereoscopc ones s commonl calle /3 veo converson [5]-[9],[]-[5]. Up to now, there are two kns of /3 veo converson schemes. One s to analze mage contents an then rectl generate bnocular mages wthout epth estmaton. For example, [5] propose a metho calle (ofe me fference, b whch moton analss s performe for each frame an then a tme ela s etermne accorng to the magntue of the moton vector. wo bnocular mages are generate b selectng two frames wth the obtane tme ela. owever, s sutable onl for mage sequences wth horzontal object motons. he other s that a epth map s frst generate an then utlze to create bnocular mages. In [][8], epth maps were bult b usng onl moton vectors extracte from the compresse veo. In [7], epths were calculate b measurng contrast, sharpness, an chromnance of the nput mages. In [3], the shapes an epths of objects are frst manuall gven b a propose sem-automatc metho. hen the objects are tracke an the corresponng epths are assgne. owever, the semautomatc nature lmts ts practcal applcatons.. heoretcall computng exact epth nformaton from sources s an ll-pose problem. A guess of epth from nsuffcent nformaton s however unavoable. Our work s characterze of estmatng epths from multple cues of an PEG-ecoe veo, such as moton parallax, atmospherc perspectve, texture graent, lnear perspectve, an relatve heght. o eal wth PEG veos effcentl the moton cue s extracte rectl from the PEG bt stream an others from the ecoe frames. In our practcal mplementaton, an nput PEG veo s converte nto a +epth sequence or a stereoscopc mage sequence, whch can then be fe nto moern stereocopc splas (for nake ees or wth glasses. Our esgne sstem s also feature of content-aaptve schemes for robust converson, that s, a ecoe veo s frst segmente nto shots, each of whch s classfe an a proper converson scheme s then chosen accorngl.. PROPOSE COERSIO SCEES For human vsual sstem (S, epth s perceve b automatcall combnng all epth cues to estmate stances of objects or relatve splacements between them. epth cues can be categorze nto two classes: phsologcal an pschologcal. he fve epth cues aopte n ths work nclue moton parallax, atmospherc perspectve, texture graent, lnear perspectve, an relatve heght. he frst

2 three are use to estmate the epth of foregroun objects an the others are for creatng the epth of backgrouns. o eal wth fferent kns of veos, we propose an aaptve /3 veo converson scheme accorng to ther contents. After veo shot etecton an classfcaton, fferent schemes are propose accorngl to estmate an assgn the epths of foregrouns an backgrouns n each frame. ase on the IR (epth Image ase Renerng technque, a +epth sequence or a stereoscopc mage sequence can be constructe for stereo spla.. eo Shot etecton an Classfcaton o extract approprate epth cues for epth estmaton, shots of smlar moton actvt are frst etecte. All the processng escrbe afterwars s base on the shot etecton result []. o perform aaptve /3 veo converson, we classf each etecte shot nto: large frame fference an low backgroun complext low frame fference, an 3 large frame fference an hgh backgroun complext. o etermne the tpe of a veo shot, analses of frame fference an content complext are frst conucte. he frame fference for a shot s efne as X S t x U ( I I t where U ( enotes the unt step functon; an the heght an wth of a frame, respectvel;, ( are S represents the number of frames; I(x, s the pxel value (lumnance Y componen at a coornate (x, n the t-th frame, an s a threshol. As for the measure of backgroun complext texture energ s calculate b usng the Law s eght masks [3]: w ( k, l I( x + k, + l, Z, ( k l where w (k,l, ~8, enote the Law s masks. o smplf the processng spee, mask operaton s onl apple to the 8 8 pxel blocks n the upper half frame (manl compose of the backgroun area. he texture energ of each 8 8 pxel block s efne as 8 Z U Z, (3 t lock _ v where t s a preefne threshol, an (u,v s the block nex. For each veo shot, the average texture energ of the backgroun s calculate as ( s a threshol: t Z ( 4 / It s ntene that the larger t u, t U ( Z t. (4 Z, the hgher the complext. ase on X an Z, a veo shot can then be classfe. he ecson rule s escrbe as follows. If X s smaller than a preefne threshol, Algorthm (A for class- shot s aopte. Otherwse, If Z s less than a preefne threshol, Algorthm (A for class- shot s aopte. Otherwse, Algorthm 3 (A3 for class-3 shot s aopte. Fgure llustrates the block agram of epth estmaton an assgnment for algorthms,, an 3. he etals wll be gven below.,3 Slng-wnow-base object segmentaton 3 Intal epth Object-base epth estmaton, refnement Fuse wth ackgroun epth assgnment Fnal epth map Fg. he block agram of epth estmaton an assgnment for algorthms ~3.. Intal Foregroun epth Estmaton As mentone, we estmate epths from multple cues. o create a ense epth map, all estmatons are base on non-overlappe blocks of 8 8 pxels. (a he cue from moton parallax oton parallax s an mportant cue to epth nformaton for human bengs. Snce currentl we eal wth nput veos of PEG format, moton vectors (s retreve from the PEG bt stream can be use to reuce the processng tme. owever, the retreve s are macro-block-wse (a macro-block ( contans pxels, the are not ense enough for our applcatons. s of 8 8 pxel blocks can be nterpolate base on an affne moton moel: x x x [ p p p p p p ] 3 4 5, (5 where ( x, an ( x, represent the coornates of corresponng ponts n two consecutve frames, an p ~ p are transform parameters. Gven the avalable s of the current an ts 8-neghborng ones (for ntracoe s, there wll be no s, at least 3 pont-pars

3 (.e., corresponng reference ponts of s between two consecutve frames coul be use to establsh a sstem of equatons (5. Parameters p ~ p can then be solve b usng the least square error metho. ase on the solve parameters, new coornates of each 8 8 pxel block can be obtane b substtutng the coornates of ts reference pont (.e, ( x, nto the rght se of Eq.(5 an then the corresponng can be erve as ( x x,. ote that for skp-moe s, the of each 8 8 pxel block theren s set to zero. In case the avalable pont-pars for solvng p ~ p s nsuffcent, the of the current s cope to ts four escenants. For an ntra-coe whch can not accumulate enough s from neghborng s, a maxmum value of (usuall the value of search range for moton estmaton s assgne. Snce the s from PEG veo ma be ncorrect, t s necessar to appl a moton correcton/refnement process [] before the above-mentone nterpolaton. he cue f from moton parallax s efne as: f +, ( where an are the horzontal an vertcal moton, respectvel. (b he cue from atmospherc perspectve Emprcall the eges of a near object are obvousl sharper n contrast than those of a far object. o extract mage features for atmospherc perspectve, we calculate the varance an contrast for each 8 8 pxel block: ( I( x, I f, (7 3 lock _ v I mn + I v f I C max I,, (8 max where I, I max, an I mn represent the average, maxmum an mnmum pxel values wthn the (u,v-th block, respectvel. (c he cue from texture graent ormall the texture graent of a near object s larger than that of a far object. he computatonal proceure of texture graent s smlar to that n measurng backgroun complext (Eq.(3. hat s, we set f Z. After obtanng the above 4 features, the ntal epth value for each block s etermne as follows: fg C ω f + ω f + ω f + ω f C, (9 mn where ω, ω, ω, an ω C are weghtng parameters obtane base on a supervse tranng process [4]. For class-3 veo shots, the ntal epth estmaton from Eq. (9 s rectl use for stereoscopc veo generaton ue to the ffcult n segmentng foregroun objects from hgh-complext backgrouns..3 Object Segmentaton After ntal epth map generaton, we threshol the map an smooth the segmente object contours b usng morphologcal openng operatons. ase on the technque of connecte component labelng, object areas of small sze can be elmnate. o remove holes nse the objects, a hole-fllng algorthm s aopte. ote that all the above segmentaton processes are base on 8 8 pxel blocks. Regons other than the foregroun objects are consere as the backgrouns an wll be ncurre epth assgnment mentone n the sequel. For class- veo shots, the frame fferences are low an objects ma not be etecte base on the above proceure. o cope wth ths problem, a slng wnow s use to accumulate the frame fferences, nstea of calculatng nter-frame fference or moton onl. A proper threshol for the accumulate fferences s set to etect the slowl movng foregroun objects. Atonall the threshole result s use as the moton cue f n Eq. (9..4 ackgroun epth Assgnment ackgroun epth cues come from lnear perspectve an relatve heght. o bul the backgroun epth b usng lnear perspectve, the geometres of backgroun, e.g., the vanshng lnes an vanshng ponts, are aopte. anshng lnes can be obtane base on tratonal ough transformaton. ackgroun epth can then be generate base on the locatons of the assocate vanshng ponts. owever, the vanshng lnes an vanshng ponts ma not be foun n veo shots. In the absence of vanshng ponts, we evelop an algorthm to create the backgroun epth map. As we know, the relatve heght means that objects closer to the horzon s perceve to be farther awa an vce versa. herefore, we fn a horzon n a frame for backgroun epth generaton. Our epth assgnment proceures nclue:. Ientf the possbl farthest foregroun object whose bottom pont has the largest coornate, enote as. bottom. he vanshng pont s then locate at a certan stance above that bottom pont (enote as, enote as vp. 3. For the backgroun regon above the vanshng pont, assgn t wth a zero epth. For others, assgn them wth a epth of

4 g ( vp vp x, 55 ( /(. ( ote that backgroun epth assgnment can be pxelbase although foregroun epth assgnment s block-base..5 Stereoscopc eo Generaton ase on IR After foregroun an backgroun assgnments b usng Eq.(9 an Eq.(, respectvel the fnal epth map s obtane b combnng them base on a ax operator,.e., fg bg ax(, (note that nearer objects have larger epth values. Assume that the baselne length for stereoscopc spla s L, the epth map can be further converte to a spart map b [9] breakancer. he result emonstrates that breakancer an breakancer nearl cannot make fference to the testees an the propose /3 veo converson scheme functon well. In aton, the result also shows that the propose /3 veo converson scheme functons well for these fferent kns of veos. p L, ( 8 (a (b whch s capable of generatng both postve an negatve parallax for human percepton. Accorng to p, the left an rght veos can be generate base on the wellknown IR technque [4],[]. 3. EXPERIE RESULS o evaluate the performance of our propose algorthms, we select several mage sequences, e.g., reakancer, Flamenco, an ran for testng. o evaluate the mpact of frame sze on vsual qualt veos of three szes (7 88, 4 48, an 7 44 pxels are evaluate n our experments. he parameter L n Eq. ( s set to be 5. Fgure shows the snthesze results for reakancer. Fg. (a s one of the frames n the sequence an Fg. (b represents the estmate epth map (brghter epth values mean near stances. he result matches human s vsual percepton. Fgs. (c an ( show the snthesze left an rght mages, respectvel. As we can see, the vsual qualt of the snthesze mages s hgh, wthout percevable geometrcal stortons. In aton, Fg. 3 shows the snthesze results of Flamco. Smlarl the regons of the fve performers have hgher epths compare to other areas an hgh perceptual qualt of the snthesze mages are also obtane. A subjectve test s conucte to show the fference between the 3 veos converte from our estmate epth map an from the groun truths (prove b crosoft Co. who knl prove the mage sequences on the network. Fgure 4 shows the result of subjectve tests from testees. he hghest score,, means the best vsual qualt. he average score of our results s.95. ote that n the fgure, breakancer an breakancer represent the stereo veos generate wth the estmate epth an the groun truth, respectvel. he score s 7. for breakancer an 7.75 for (c ( Fg. reakancer : (a Input frame, (b estmate epth map, (c snthesze left mage, an ( snthesze rght mage. (a (c ( Fg. 3 Flamenco: (a nput frame (b estmate epth map (c snthesze left mage ( snthesze left mage. (b

5 8 4 est sequence Fg. 4 he result of subjectve test. 4. COCLUSIO reakancer reakancer Flamenco Foreman ran In ths work, a /3 veo converson scheme base on multple cue analss for PEG veos was propose. o eal wth varng contents, veo shots are frst etecte an classfe. In orer to generate the ntal epth map for each frame, 3 kns of cues such as moton parallax, atmospherc perspectve, an texture graent, are analze an fuse. e separate the epth assgnments for foregroun objects an backgroun areas wth fferent algorthms. he fnal epth map s then obtane b combnng the assgnments for foregrouns an backgrouns. o generate stereoscopc veos, spart map of each frame s frst calculate from the assocate epth map an use to create the left an rght mages base on the well-known IR technque. Subjectve tests show that our generate 3 veo presents lttle fference to that create base on the epth groun truths. [7]. urata an Y. or, A real-tme to 3 mage converson technque usng mage epth, SI, IGES, pp. 99-9, 998. [8]. Km,. n, an K. Sohn, A stereoscopc veo generaton metho usng stereoscopc spla characterzaton an moton analss, IEEE rans. on roacastng, ol.54, o., pp.88-97, 8. [9] Jaeseung Ko, anbae Km, an Changck Km, -o-3 Stereoscopc Converson: epth-ap Estmaton n a Sngle-ew Image, n Proc. of SPIE, ol. 9, 7. [] L. Zhang an. J. am, Stereoscopc mage generaton base on epth mages for 3, IEEE rans. roacastng, vol. 5, no., pp. 9 99, 5. [].. Poraza, P. asopoulos, an R. K. ar, An.4- base scheme for to 3 veo converson, n Proc. Internatonal Conference on Consumer Electroncs, pp., 9. [] Y. L. Chang, C. Y. Fang, L. F. ng, S. Y. Chen, an L. G. Chen, epth map generaton for -to-3 converson b short-term moton assste color segmentaton, n Proc. IEEE Int'l Conf. on ultmea an Expo, pp , 7. [3] C. u, G. Er, X. Xe,. L, X. Cao, an Q. a, A novel metho for sem-automatc to 3 veo converson, n Proc. 3 Conference: he rue son - Capture, ransmsson an spla of 3 eo, pp. 5 8, 8. [4] Y. J. Jung, A. ak, J. Km an. Park, A novel -to-3 converson technque base on relatve heght epth cue, n Proc. of SPIE, vol. 734, pp. 737U--737U -8, 9. [5] C. C. Cheng, C.. L, Y.. sa, an L. G. Chen, br epth cueng for -to-3 converson sstem, n Proc. of SPIE, vol. 734, pp , 9. REFERECES [] L.. J. eesters,. A. Ijsselstejn, an P. J.. Seuntens, A surve of perceptual evaluatons an requrements of three-mensonal, IEEE rans. Crcuts an Sstems for eo echnolog vol. 4, pp , 4. [].-. Le an C.-. La, ews veo summarzaton base on spatal an moton feature analss, n Proc. Pacfc-Rm Conference on ultmea, pp. 4 55, 4. [3].. Suzuk, Y. Yagnuma,. Yamaa, an Y. Shmzu, A Shape Feature Extracton etho ase on 3 Convoluton asks, n Proc. of Eghth IEEE Int l Smposum on ultmea, pp ,. [4] Cheng-Yng Yeh, Stereoscopc Converson echnque ase on onocular ew Source, aster thess, atonal Chung Cheng Unverst Jul 8. [5].-. Km,.-S. Song,.-K. Km, an K.-C. Cho, Stereoscopc converson of monoscopc veo b the transformaton of vertcal-to-horzontal spart n Proc. of SPIE, vol. 395, pp. 5-75, 998. [] I. Ieses, L. P. Yaroslavsk an. Fshban, Real-tme to 3 veo converson, J. Real-me Image Proc., vol., pp. 3-9, 7.