arxiv: v2 [cs.cv] 20 May 2018

Transcription

1 Sereoscopic Neural Syle Transfer Dongdong Chen 1 Lu Yuan 2, Jing Liao 2, Nenghai Yu 1, Gang Hua 2 1 Universiy of Science and Technology of China 2 Microsof Research cd722522@mail.usc.edu.cn, {jliao, luyuan, ganghua}@microsof.com, ynh@usc.edu.cn arxiv: v2 [cs.cv] 20 May 2018 Absrac This paper presens he firs aemp a sereoscopic neural syle ransfer, which responds o he emerging demand for 3D movies or AR/VR. We sar wih a careful examinaion of applying exising monocular syle ransfer mehods o lef and righ views of sereoscopic images separaely. This reveals ha he original dispariy consisency canno be well preserved in he final sylizaion resuls, which causes 3D faigue o he viewers. To address his issue, we incorporae a new dispariy loss ino he widely adoped syle loss funcion by enforcing he bidirecional dispariy consrain in non-occluded regions. For a pracical realime soluion, we propose he firs feed-forward nework by joinly raining a sylizaion sub-nework and a dispariy sub-nework, and inegrae hem in a feaure level middle domain. Our dispariy sub-nework is also he firs end-oend nework for simulaneous bidirecional dispariy and occlusion mask esimaion. Finally, our nework is effecively exended o sereoscopic videos, by considering boh emporal coherence and dispariy consisency. We will show ha he proposed mehod clearly ouperforms he baseline algorihms boh quaniaively and qualiaively. 1. Inroducion Sereoscopic 3D was on he cusp of becoming a mass consumer media such as 3D movies, TV and games. Nowadays, wih he developmen of head-mouned 3D display (e.g., AR/VR glasses) and dual-lens smar phones, sereoscopic 3D is aracing increasing aenion and spurring a lo of ineresing research works, such as sereoscopic inpaining [42, 32], video sabilizaion [19], and panorama [45]. Among hese sudies, creaing sereoscopic 3D conens is always inriguing. Recenly, syle ransfer echniques used o reproduce famous paining syles on naural images become a rending opic in conen creaion. For example, he recen film Loving Vincen is he firs animaed film made enirely of oil This work was done when Dongdong Chen is an inern a MSR Asia. painings by well-rained ariss. Inspired by he power of Convoluional Neural Nework (CNN), he pioneering work of Gays e al. [17] presened a general soluion o ransfer he syle of a given arwork o any images auomaically. Many follow-up works [25, 23, 40, 14, 13, 37, 29] have been proposed o eiher improve or exend i. These echniques are also applied o many successful indusrial applicaions (e.g., Prisma [1], Osagram [2], and Microsof Pix [3]). However, o he bes of our knowledge, here are no echniques ha apply syle ransfer o sereoscopic images or videos. In his paper, we address he need for his emerging 3D conen by proposing he firs sereoscopic neural syle ransfer algorihm. We sar wih a careful examinaion of naive applicaion of exising syle ransfer mehods o lef and righ views independenly. We found ha i ofen fails o produce geomeric consisen sylized exure across he wo views. As a resul, i induces problemaic deph percepion and leads o 3D faigue o he viewers as shown in Figure 1. Therefore, we need o enable he mehod o produce sylized exures ha are consisen across he wo views. Moreover, a fas soluion is required, especially for pracical real-ime 3D display (e.g., AR/VR glasses). Las bu no leas, syle ransfer in sereoscopic video as a furher exension should saisfy emporal coherence simulaneously. In his paper, we propose he firs feed-forward nework for fas sereoscopic syle ransfer. Besides he widely adoped syle loss funcion [17, 23], we inroduce an addiional dispariy consisency loss, which penalizes he deviaions of sylizaion resuls in non-occluded regions. Specifically, given he bidirecional dispariy and occlusion mask, we esablish correspondences beween he lef and righ view, and penalize he sylizaion inconsisencies of he overlapped regions which are visible in boh views. We firs validae his new loss erm in he opimizaionbased soluion [17]. As shown in Figure 1, by joinly considering sylizaion and dispariy consisency in he opimizaion procedure, our mehod can produce much more consisen sylizaion resuls for he wo views. We furher incorporae his new dispariy loss ino a feedforward deep nework ha we designed for sereoscopic sylizaion. 1

2 (a) (b) (c) (d) (e) Figure 1. (a) Given a sereoscopic image pair and a syle image, when he lef and righ view are sylized separaely (firs row), he lef sylizaion resul (b) will be inconsisen wih ha of he righ view (c) in spaially corresponding areas (d).this will lead o undesirable verical dispariies and incorrec horizonal dispariies, subsequenly causing 3D visual faigue in anaglyph images (e). In conras, by inroducing a new dispariy consisency consrain, our mehod (second row) can produce consisen sylizaion resuls for he wo views. Our nework consiss of wo sub-neworks. One is he sylizaion sub-nework SyleNe, which employs he same archiecure in [23]. The oher is he dispariy subnework DispOccNe, which can esimae bidirecional dispariy maps and occlusion masks direcly for an inpu sereo image pair. These wo sub-neworks are inegraed in a feaure level middle domain. They are firs rained on each ask separaely, and hen joinly rained as a whole. Our new dispariy sub-nework has wo advanages: 1) i enables real-ime processing, when compared agains some sae-of-he-ar sereo maching algorihms [39, 26] ha use slow global opimizaion echniques; 2) i is he firs endo-end nework which esimaes he bidirecional dispariies and occlusion masks simulaneously, while oher mehods [31, 44] only esimae a single direcional dispariy map in each forward and need pos-processing seps o obain he occlusion mask. In Sec. 5.2, we will show ha his bidirecional design is beer han he single direcional design. Our nework can also be easily exended o sereoscopic 3D videos by inegraing he sub-neworks used in [12]. In his way, he final sylizaion resuls can keep no only he horizonal spaial consisency a each ime sep, bu also he emporal coherence beween adjacen ime seps. This work may inspire film creaors o hink abou auomaically urning 3D movies or TVs ino famous arisic syles. In our experimens, we show ha our mehod ouperforms he baseline boh quaniaively and qualiaively. In summary, his paper consiss of four main conribuions: We propose he firs sereoscopic syle ransfer algorihm by incorporaing a new dispariy consisency consrain ino he original syle loss funcion. We propose he firs feed-forward nework for fas sereoscopic syle ransfer, which combines sylizaion, bidirecional dispariies and occlusion masks esimaion ino an end-o-end sysem. Our dispariy sub-nework is he firs end-o-end nework which simulaneously esimaes bidirecional dispariy maps and occlusion masks. We furher exend our mehod o sereoscopic videos by inegraing addiional sub-neworks o consider boh dispariy consisency and emporal coherency. In he remainder of his paper, we will firs summarize some relaed works. In our mehod, we validae our new dispariy consrain using a baseline opimizaion-based mehod, and hen inroduce our feed-forward nework for fas sereoscopic syle ransfer, and exend i o sereoscopic videos. Experimens will show he evaluaion and comparison wih oher ablaion analysis of our mehod. Finally, we conclude wih furher discussion. 2. Relaed Work Wih he increasing populariy and grea business poenial of 3D movies or AR/VR echniques, sereoscopic image/video processing echniques have drawn much aenion. Some ineresing sereoscopic opics include image inpaining [42], objec copy and pase [30], image reargeing [11, 7], image warping [33] and video sabilizaion [19]. In his work, we inroduce a new opic which urns sereoscopic images or videos ino synheic arworks. In he pas, re-drawing an image in a paricular syle required a well-rained aris o do los of ime-consuming manual work. This moivaed he developmen of a pleny of Non-phoorealisic Rendering (NPR) algorihms o make he process auomaic, such ha everyone can be an aris. However, hey are usually confined o he specific arisic

3 syles (e.g., oil painings and skeches). Gays e al. [18] were he firs o sudy how o use CNNs o reproduce famous syles on naural images. They leverage CNNs o characerize he visual conen and syle, and hen recombine boh for he ransferring syles, which is general o various arisic syles. To furher improve he qualiy, many domain priors or schemes are used, including face consrains [36], MRF priors [25], or user guidances [10]. To accelerae he rendering process, a feed-forward generaive nework [23, 40, 14, 13] can be direcly learn insead, which were successfully deployed popular apps (e.g., Prisma[1], Microsof Pix [3]). However, he algorihms described above are designed only for monocular images. When hey are independenly applied o sereoscopic views, hey ineviably inroduce spaial inconsisency, causing visual discomfor (3D faigue). Generally, sereoscopic 3D echniques consider he dispariy consisency in he objecive funcion. Sereoscopic syle ransfer is no an excepion. The firs sep is o esimae a high qualiy dispariy map from he wo inpu views, which is sill an acive research opic. Tradiional mehods ofen use sophisicaed global opimizaion echniques and MRF formulaions, such as [21, 43, 41]. Recenly, some CNN-based mehods have been explored. Zbonar e al. [44] adoped a Siamese nework for compuing maching disances beween image paches. Mayer e al. [31] synhesized a large daase and rained an end-o-end nework for dispariy esimaion. However, i can only obain a single direcional dispariy map in each forward and need posprocessing seps o obain he occlusion mask. Our dispariy sub-nework adops a similar nework archiecure bu wih a differen loss funcion. I can simulaneously esimae he bidirecional dispariy maps and occlusion masks, which is essenial o high-qualiy sereo image/video ediing. To he bes of our knowledge, i is he firs end-o-end nework for bidirecional dispariies and occlusion masks esimaion. The mos relaed work o ours is video syle ransfer. Previous mehods [4, 34, 12, 20, 22, 35] all incorporaed a new emporal consisency consrain in he loss funcion o avoid flickering arifacs. Analogous o he emporal consisency, sereoscopic syle ransfer requires spaial consisency beween he lef and righ view. Differen from sequenial processing in video syle ransfer, wo views should be processed symmerically in sereoscopic syle ransfer. I is crucial o avoid he srucure disconinuiy near he occlusion boundary as shown in Figure 2. Our syle ransfer nework is naurally exended o sereoscopic videos by considering boh spaial consisency and emporal consisency a he same ime. 3. Dispariy Loss for Spaial Consisency Since previous neural syle ransfer mehods only ackle monocular views, he naive exension fails o preserve spa- Figure 2. Comparison resuls of sylizing lef and righ view sequenially (op row) or joinly (middle row) wih he dispariy consisency consrain. The former mehod will ofen generae exure disconinuiy near occlusion mask boundary. Boom row is he righ view occlusion mask and enlarged sylizaion paches. ial consisency beween he lef and righ view images, as shown in in Figure 1. How o enforce spaial consisen consrain across wo views is worh exploring. Video syle ransfer is also confroned by similar consisency preservaion problem, bu in he emporal axis. To reduce inconsisency, some mehods [12, 34, 35] use he sylizaion resuls of previous frames o consrain ha of he curren frame. By analogy, we can firsly sylize he lef view, and hen use i o consrain he sylizaion of he righ view wih dispariy consisency, or vice versa. In his way, we can obain spaially consisen resuls in visible overlapping regions, bu he coninuiy of sylizaion paerns near he occlusion boundary is ofen damaged, as shown in Figure 2. The underlying reason is ha he lef sylizaion resul is fixed during he opimizaion procedure of he righ view. To avoid i, we should regard he lef and righ view in a symmeric way and joinly process hem. Therefore, we add a new erm by enforcing symmeric bidirecional dispariy consrain, o he sylizaion loss funcion, and joinly opimize lef and righ views. We firs validae he new loss in opimizaion-based syle ransfer framework [17], which is formulaed as an energy minimizaion problem. In he nex secion, we will furher incorporae he proposed loss o our feed-forward nework. Given a sereoscopic image pair I l, I r and a syle image S, he lef and righ view sylizaion resuls O l, O r are ieraively opimized via gradien descen. The objecive loss funcion L oal consiss of hree componens: conen loss L v con, syle loss L v sy and dispariy loss L v disp, where v {l, r} represens he lef or righ view, i.e.,

4 L oal = v {l,r} ( αl v con (O v, I v ) + βl v sy(o v, S) +γl v disp(o v, D v, M v ) ). Boh he conen loss and syle loss are similar o [23]: L v con(o v, I v ) = i {l c} F i (O v ) F i (I v ) 2, L v sy(o v, S) = i {l s} G(F i (O v )) G(F i (S)) 2, (2) where F i and G are feaure maps and Gram marix compued from he layer i of a pre-rained VGG-19 nework [38]. {l c }, {l s } are VGG-19 layers used for conen represenaion and syle represenaion, respecively. The new erm of dispariy loss enforces he sylizaion resul a one view o be as close as possible o he warped resul from he oher view in he visible and overlapping regions (i.e., non-occludded regions). I is defined as: L v disp(o v, D v, M v ) = (1 M v ) O v W(O v, D v ) 2 (3) where v is he opposie view of v (if v is he lef, hen v is he righ). W(O v, D v ) is he backward warping funcion ha warps O v using he dispariy map D v via bilinear inerpolaion, namely W(O v, D v )(p) = O v (p + D v (p)). M v is he occlusion mask, where M v (p) = 0 for pixel p visible in boh views and M v (p) = 1 for pixel p occluded in he opposie view. Given he lef and righ dispariy map, he occlusion mask M can be obained by a forward-backward consisency check, which is also used in [34]. Noe ha he loss L disp is symmeric for boh views, and relies on he bidirecional dispariies and occlusion masks. As shown in Figure 1, compared o he baseline (i.e., processing each view independenly), our mehod achieves more consisen resuls and can avoid 3D visual faigue in final anaglyph images. Joinly opimizing he lef and righ view in a symmeric way can furher avoid he disconinuiy near he occlusion boundary as shown in Figure 2,. 4. Sereroscopic Syle Transfer Nework In his secion, we propose a feed-forward nework for fas sereoscopic syle ransfer. The whole nework consiss of wo sub-neworks: he SyleNe which is similar o exising syle ransfer neworks [12, 13, 14, 20], and he DispOccNe which simulaneously esimaes bidirecional dispariy maps and occlusion masks. we inegrae hese wo sub-neworks in a feaure level middle domain, making he lef view and righ view compleely symmeric. SyleNe. We use he defaul syle nework srucure firsly proposed by [23] and used exensively in he oher (1) works [12, 13, 14, 20]. The archiecure basically follows an image auo-encoder, which consiss of several srided convoluion layers (encoding he image ino feaure space), five residual blocks, and fracionally srided convoluion layers (decoding feaure o he image). In our implemenaion, we follow he same designing as [12], where he layers before he hird residual block (inclusive) are regarded as he encoder, and he remaining layers are regarded as he decoder. DispOccNe. Recenly, Mayer e al. [31] inroduced an end-o-end convoluion nework called DispNe for dispariy esimaion. However, i can only predic single direcional dispariy map D l (l r) in each forward. Here, we use he similar nework srucure, bu add hree more branches in he expanding par for each resoluion (1/64,...,1/2). These hree branches are used o regress dispariy D r and bidirecional occlusion masks M l, M r. The loss funcion for each resoluion is: L = L d (Mv g, D v, Dv) g + λl o (W v, M v, Mv g ), v {l,r} L d (Mv g, D v, Dv) g = (1 Mv g ) D v Dv, g L o (W v, M v, Mv g ) = 1 W v (i)[mv g (i)log(m v (i)) n i + (1 Mv g (i))log(1 M v (i))], (4) where he superscrip g denoes he ground ruh. Differen from he original loss in [31], we remove he dispariy deviaion penaly in occluded regions in L d, where he dispariy values are undefined in real scenarios. W v is a pixel-wise class balance weigh map, where values in occluded regions are he raio of non-occluded and occlusion pixel number #non occ #occ, while values in non-occluded regions are 1. Noe ha he ground ruh of D v, M v a each resoluion are bilinearly inerpolaed from ha of he original image resoluion. The losses for each resoluion are summed. Please refer o he supplemenary maerial for deails of he sub-nework. Middle Domain Inegraion. Since he lef and righ view are compleely symmeric, we also consider he SyleNe and DispOccNe in a symmeric way. In fac, for a sereoscopic image pair, he overlapping regions visible in boh views can be defined in a inermediae symmeric middle domain [8, 27]. If we know he occlusion mask for each view, we can sylize he overlapping regions and occluded regions respecively, hen compose he image based on he occlusion mask. In his way, he final sylizaion resuls would naurally saisfy dispariy consisency. Image level composiion ofen suffers from flow or dispariy errors, and produces blurring and ghosing arifacs. As demonsraed in [12], feaure level composiion, followed by a decoder back o he image space, is more ol-

5 Encoder (SyleNe) warp : forward warp F l warp M l -1 warp F l fuse F l o Decoder (SyleNe) I l I r DispOccNe D l D r M l M r Combine warp D h M h F h warp F r fuse F r o O l O r F r M r Figure 3. The overall nework srucure for fas sereoscopic image syle ransfer. I consiss of wo sub-neworks: SyleNe and DispOccNe, which are inegraed in he feaure level middle domain H. eran o errors. Therefore, we inegrae he SyleNe and DispOccNe in a new feaure level middle domain H. The overall Nework archiecure is shown in Figure 3. Nework Overview. Specifically, We firs encode I l, I r ino feaure maps F l, F r wih he encoder of he SyleNe and predic he bidirecional dispariy maps and occlusion masks D l, D r, M l, M r wih he DispOccNe, which are bilinearly resized o mach he resoluion of F l, F r. Then for each view v, we warp F v and he halved D v o he middle domain using a forward warping funcion, which allows warping from each view o he middle domain wihou knowing he middle view dispariy. The warped wo views are combined, generaing he middle dispariy D h, feaure map F h, and hole mask M h. Similar o [27], he value D h (p) of a poin p is defined as he symmeric shif disance o he correspondence poin in he lef and righ view (lef: p D h (p), righ: p + D h (p)). The hole masks generaed in he lef and righ view forward warping are combined as M h, and he corresponding pixel values in F h are excluded in he following warping, i.e., D l := D l 2, D r := D r 2, D h = W(D l, D l, M l ) + W(D r, D r, M r ) 2 W(F l, D l, M l ) + W(F r, D r, M r ) F h =, 2 where W(x, y, m) is he forward warping funcion ha warps x using he dispariy map y guided by he occlusion mask m. Namely, if z = W(x, y), hen q z(p) = w q x(q + y(q)) q w, q : q + y(q) N 8 (p) q (6) where N 8 (p) denoes he eigh-neighborhood of p, w q is he bilinear inerpolaion weigh, making z boh differeniable o x and y. All he occluded pixels q in m are excluded in (5) he forward warping procedure, which avoids he many-oone mapping problem. Nex, we furher forward warp F h back o he original lef and righ view, and fuse hem wih F l, F h based on M l, M r respecively, i.e., F l = W(F h, D h, M h ), F r = W(F h, D h, M h ) F o v = M v F v + (1 M v ) F v, v {l, r} Finally, he fused feaure maps F o l, F o r are fed ino he decoder of he SyleNe o obain he final sylizaion resuls O l, O r Exension o Sereoscopic Videos To exend our nework for sereoscopic videos, similar o [12, 35], we incorporae one more emporal coherence erm for each view v ino our objecive funcion, i.e., L cohe = (1 Mv) Ov W 1(O v 1 ) 2 (8) v {l,r} where W 1(.) is he funcion o warp Ov 1 o ime sep using he ground ruh backward flow as defined in [12]. Inspired by [12], we furher add an addiional flow subnework and mask sub-nework (ogeher referred o as he Temporal nework ) ino he original nework. We show he basic working flow in he lef par of Figure 4. For a view v, wo adjacen frames Iv 1, Iv are fed ino he flow sub-nework o compue he feaure flow wv, which warps he inpu feaure map Fv o, 1 o Fv. Nex he difference Fv beween he new feaure map Fv compued from Iv and Fv is fed ino he mask sub-nework, generaing he composiion mask M. The new feaure map Fv u, is he linear combinaion of Fv and Fv weighed by M. We inegrae he above sereoscopic image syle ransfer nework (referred o as Sereo nework ) wih Temporal nework in a recurren formulaion. Specifically, for each view v, we recursively feed he previous dispariy consisen and emporal coheren feaure maps Fv o, 1, ogeher wih adjacen frames Iv 1, Iv, ino he Temporal nework, generaing he emporal coheren feaure map Fv u,. Then F u, (7) l

6 I v 1, I v Flow w o, 1 F v warp F v Temporal F v F v diff F v u, fuse M Mask F F v F v warp F l o, 1 F l u, I l 1, I l Temporal F l o, : backward warp Sereo I r 1, I r Temporal F r u, F r o, F r o, 1 Figure 4. The overall srucure for sereoscopic video syle ransfer. The lef par is he simplified working flow for Temporal nework. The righ par is he recurren formulaion for combining he above Sereo nework and he lef addiional Temporal nework. and Fr u, are furher fed ino he Sereo nework o guaranee dispariy consisency. A his poin, he oupu feaure maps F o, l, Fr o, are boh emporal coheren and dispariy consisen and fed o he decoder of SyleNe o ge he sylizaion resuls. Noe ha for = 1, F u,1 l, Fr u,1 are jus he oupu feaure maps of encoder of he SyleNe. 5. Experimens 5.1. Implemenaion Deails For fas sereoscopic image syle ransfer, he overall nework conains wo sub-neworks: he SyleNe and he DispOccNe. In our implemenaion, hese wo sub-neworks are firs prerained separaely, hen joinly rained. For he SyleNe, we adop he prerained models released by [23] direcly, which is rained on Microsof COCO daase[28]. To prerain DispOccNe, we adop he same raining sraegy in [31] wih he synheic daase FlyingThings3D, which conains raining and 4248 es sereo image pairs. During raining, we use he bidirecional consisency check o obain he ground ruh occlusion mask as in [34]. These wo sub-neworks are joinly rained wih a bach size of 1 (image pair) for 80k ieraions. The Adam opimizaion mehod [24] is adoped, which is widely used in image generaion asks [5, 6, 15, 16]. The iniial learning rae is and decayed by 0.1 a 60k ieraions. Because he syle loss is around 10 7 imes larger han he dispariy loss, he corresponding gradien of DispOccNe coming from SyleNe is scaled wih 10 7 o balance. By defaul, γ = 500 is used for all he syles, and α, β remain unchanged as he prerained syle models. For sereoscopic video syle ransfer, we add an addiional Temporal nework (one flow sub-nework and one mask sub-nework) o he above Sereo nework. In our defaul implemenaion, he addiional Temporal nework is rained using he same mehod as [12], and direcly inegraed wih a well-rained Sereo nework in a recurren for- Mehod MPI Sinel clean FlyingThings3D Time DispNe[31] s DispOccNe* s DispOccNe-SD s DispOccNe-OL s Table 1. Non-occluded endpoin errors for DispOccNe and is varians. The es ime is for a 960x540 image on GTX TianX. Mehod Dispariy loss Candy La muse Mosaic Udnie Time baseline [23] s fineuned [23] s our mehod s our mehod s Percepual loss baseline [23] our mehod our mehod Table 2. Comparison resuls of differen mehods of dispariy loss and percepual loss on FlyingThings3D es daase. The es ime is for 640x480 image pair on GTX TianX. mulaion as show in Figure Evaluaion and Analysis of he DispOccNe To evaluae he performance of our DispOccNe, we es our model on he MPI Sinel sereo daase [9] and FlyingThings3D es daase, respecively. To fully undersand he effecs of each modificaion, we furher rain wo varian neworks. The DispOccNe-SD only regresses single direcional dispariy and occlusion mask raher han bidirecional. The DispOccNe-OL is rained wih he original loss funcion in [31] wihou removing he penaly for occluded regions. Since we only care abou he dispariy precision in non-occluded regions, we use he endpoin error (EPE) in non-occluded regions as he error measure. As shown in Table 1, our DispOccNe is only 9.3% slower han he original DispNe while predics he bidirecional dispariy maps and occlusion masks simulaneously. Wih he modified loss funcion and nework srucure, i achieves even beer dispariy boh on he MPI Sinel daase and he FlyingThings3D daase. Compared o he varian nework DispOccNe-SD, which only rains he single direcional dispariy and occlusion mask, DispOccNe is also beer. We believe ha feeding bidirecional dispariies and occlusion masks helps he nework o learn he symmeric propery of he lef and righ dispariies, hus learn more meaningful inermediae feaure maps. Besides dispariy, occlusion mask is also very imporan for image or feaure composiion. In fac, here are wo differen ways o obain he occlusion mask. The firs is o make he nework learn he occlusion mask direcly (such as our mehod). The oher is o run pos bidirecional consisency check afer geing accurae bidirecional dispariy maps. However for he laer mehod, one needs 2 forward ime or rerains one nework for bidirecional dispari-

7 Figure 5. Comparison of occlusion masks: ground ruh (op lef), our mehod (op righ), and pos bidirecional consisency check(boom lef). The occlusion mask generaed by pos bidirecional consisency check conains more false posiives and noise. ies similar o ours. Moreover, when he dispariy map is no sufficienly good, he occlusion mask generaed by he laer mehod will conain more false alarms and noises, as shown in Figure 5. We also compare he F-score of he prediced occlusion masks by hese wo mehods on he FlyingThings3D es daase, our mehod is much beer (F-score: 0.887) han he laer mehod (F-score: 0.805) Evaluaion for Sereoscopic Syle Transfer Quaniaive Evaluaion. To validae he effeciveness of our mehod, we use wo differen quaniaive evaluaion merics: 1) he percepual loss αl con + βl sy o represen he faihfulness o he original syles, and 2) he dispariy loss L disp o evaluae he dispariy consisency. We compare hree differen mehods on he FlyingThings3D es daase for four differen syles: he baseline monocular mehod [23], and SyleNe [23] fineuned wih dispariy loss bu es wihou DispOccNe, our mehod (fineuned SyleNe + DispOccNe). As shown in Table 2, compared o he baseline mehod [23], our resuls are more dispariy consisen while keeping he original syle faihfulness (i.e., similar percepual loss). When esing fineuned SyleNe [23] wihou DispOccNe, he dispariy loss also decreases a lo. This shows ha he sabiliy of he original syle nework is acually improved a lo afer join raining wih he new dispariy loss. We furher conduc an user sudy o compare our mehod wih he baseline mehod [23]. Specifically, we randomly selec 5 sereoscopic image pairs and 2 videos from he MPI-Sinel and kiy daase for 4 differen syles hen ask 20 paricipans o answer which is more sereoscopic consisen?. Our mehod wins 94.8% of he ime while [23] only wins 5.2% of he ime. Qualiaive Evaluaion. In Figure 7, we show he comparison resuls wih our baseline (sylizing each view independenly) for a real sree view sereoscopic image pair. The op row wih red marked boxes is he baseline resuls, where he sylized exures in he corresponding regions beween he lef and righ views are ofen inconsisen. When Figure 6. Comparison resuls using a similar varian of [12](op row), which suffers from boh ghos arifacs and sylizaion inconsisency. The middle row is he resuls wih composiion mask replaced by our mehod, he ghos arifacs disappear bu inconsisencies sill exis. By conras, our resuls (boom row) do no have he above problems. waching hese resuls wih 3d devices, hese inconsisencies will make i more difficul for our eyes o focus, causing 3d faigue. In conras, our resuls are more consisen. In Figure 8, we show he comparison resuls for wo adjacen sereoscopic image pairs of a sereoscopic video. By incorporaing he addiional Temporal nework, our mehod can obain boh dispariy consisen and emporal coheren sylizaion resuls (More visual resuls can be found on youube 1 ) More Comparison Single Direcional vs. Bidirecional We have also designed an asymmeric single direcional sereoscopic image syle ransfer nework for feaure propagaion and composiion. Differen from he symmeric bidirecional nework srucure shown in Figure 3, we direcly warp he lef view feaure F l o he righ view using he righ dispariy map D r, hen conduc composiion wih F r based on he righ occlusion mask M r. As shown in Table 2, for he four es syles, he single direcional mehod (marked wih ) can obain similar sable errors, bu all higher percepual loss han our defaul bidirecional design. Furhermore, by experimen, we find i more difficul o joinly rain SyleNe and DispOccNe for his asymmeric design, because he gradien for he lef and righ view is very unbalanced, making DispOccNe diverge easily. Comparison wih Varian of [12] In he monocular video syle ransfer mehod [12], a flow sub-nework is uilized o guaranee emporal coherence, and he composiion mask is implicily rained wih a mask sub-nework. We 1 hps://

8 Figure 7. Comparison wih our baseline for a real sree view sereoscopic image pair. The op row wih red marked boxes is he baseline resuls, and he boom row wih corresponding green marked boxes is our resuls. Obviously, our resuls are more dispariy consisen. Figure 8. Comparison wih our baseline for wo adjacen sereoscopic image pairs. The op wo rows are he baseline resuls, and he boom wo rows are our resuls. Compared o our baseline, our resuls can guaranee boh dispariy consisency and emporal coherency. The op-righes is he inpu sereoscopic image pair of ime sep 1, and he boom-righes is ha of ime sep. have also designed a similar nework srucure wih flow sub-nework replaced by DispNe [31]. As shown Figure 6, i suffers from boh ghosing arifacs and sylizaion inconsisency. The ghosing arifacs are resuled from he undefined dispariy in occluded regions. When he occlusion mask is unknown, or he final composiion mask is no good enough, he incorrecly warped feaure will be used in he final composie feaure, causing ghos arifacs. We also visualize he implicily rained composiion mask M, which is clearly worse han he composiion mask M l, M r (occlusion mask) from our proposed DispOccNe. For furher validaion, we replace he final composiion mask wih he occlusion mask M l from our DispOccNe, he ghos arifacs disappear. Bu inconsisencies sill exis, because he original syle sub-nework is fixed in [12], which is sensiive o small perurbaions. By conras, our syle sub-nework is more sable afer join raining wih he dispariy consisency loss. 6. Conclusion In his paper, we presen he firs sereoscopic syle ransfer algorihm by inroducing a new dispariy consisency loss. For a pracical soluion, we also propose a feed-forward nework by joinly raining a sylizaion subnework and a dispariy sub-nework. To he bes of our knowledge, our dispariy sub-nework is he firs end-o-end nework ha enables simulaneous esimaion of he bidirecional dispariy maps and he occlusion masks, which can poenially be uilized by oher sereoscopic echniques.

9 To furher exend our mehod for sereoscopic videos, we incorporae an addiional Temporal nework [12] ino our Sereo nework. Along his direcion, here is much fuure work worh invesigaing. For example, moivaed by our DispOccNe, he flow sub-nework used for emporal coherence can poenially also be exended o simulaneously predic he bidirecional flow and occlusion masks if suiable daase exiss. Furhermore, how o unify he flow and dispariy ino one nework remains an open quesion, and worh exploring. References [1] Prisma labs. prisma: Turn memories ino ar using arificial inelligence, , 9. 1, 3 [2] hps://osagram.ru/ [3] Ai wih creaive eyes amplifies he arisic sense of everyone , 3 [4] A. G. Anderson, C. P. Berg, D. P. Mossing, and B. A. Olshausen. Deepmovie: Using opical flow and deep neural neworks o sylize movies. arxiv preprin arxiv: , [5] J. Bao, D. Chen, F. Wen, H. Li, and G. Hua. Cvae-gan: fine-grained image generaion hrough asymmeric raining. ICCV, [6] J. Bao, D. Chen, F. Wen, H. Li, and G. Hua. Towards open-se ideniy preserving face synhesis. arxiv preprin arxiv: , [7] T. Basha, Y. Moses, and S. Avidan. Geomerically consisen sereo seam carving. In Compuer Vision (ICCV), 2011 IEEE Inernaional Conference on, pages IEEE, [8] P. N. Belhumeur and D. Mumford. A bayesian reamen of he sereo correspondence problem using half-occluded regions. In Compuer Vision and Paern Recogniion, Proceedings CVPR 92., 1992 IEEE Compuer Sociey Conference on, pages IEEE, [9] D. J. Buler, J. Wulff, G. B. Sanley, and M. J. Black. A nauralisic open source movie for opical flow evaluaion. In Proc. ECCV, pages Springer, [10] A. J. Champandard. Semanic syle ransfer and urning wo-bi doodles ino fine arworks. arxiv preprin arxiv: , [11] C.-H. Chang, C.-K. Liang, and Y.-Y. Chuang. Conen-aware display adapaion and ineracive ediing for sereoscopic images. IEEE Transacions on Mulimedia, 13(4): , [12] D. Chen, J. Liao, L. Yuan, N. Yu, and G. Hua. Coheren online video syle ransfer. arxiv preprin arxiv: , , 3, 4, 5, 6, 7, 8, 9 [13] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua. Sylebank: An explici represenaion for neural image syle ransfer. In Proc. CVPR, , 3, 4 [14] V. Dumoulin, J. Shlens, and M. Kudlur. A learned represenaion for arisic syle. arxiv preprin arxiv: , , 3, 4 [15] Q. Fan, D. Wipf, G. Hua, and B. Chen. Revisiing deep image smoohing and inrinsic image decomposiion. arxiv preprin arxiv: , [16] Q. Fan, J. Yang, G. Hua, B. Chen, and D. Wipf. A generic deep archiecure for single image reflecion removal and image smoohing. In Proceedings of he 16h Inernaional Conference on Compuer Vision (ICCV), pages , [17] L. A. Gays, A. S. Ecker, and M. Behge. A neural algorihm of arisic syle. arxiv preprin arxiv: , , 3 [18] L. A. Gays, A. S. Ecker, M. Behge, A. Herzmann, and E. Shechman. Conrolling percepual facors in neural syle ransfer. arxiv preprin arxiv: , [19] H. Guo, S. Liu, S. Zhu, and B. Zeng. Join bundled camera pahs for sereoscopic video sabilizaion. In Image Processing (ICIP), 2016 IEEE Inernaional Conference on, pages IEEE, , 2 [20] A. Gupa, J. Johnson, A. Alahi, and L. Fei-Fei. Characerizing and improving sabiliy in neural syle ransfer. arxiv preprin arxiv: , , 4 [21] H. Hirschmuller. Sereo processing by semiglobal maching and muual informaion. IEEE Transacions on paern analysis and machine inelligence, 30(2): , [22] H. Huang, H. Wang, W. Luo, L. Ma, W. Jiang, X. Zhu, Z. Li, and W. Liu. Real-ime neural syle ransfer for videos. CVPR, [23] J. Johnson, A. Alahi, and L. Fei-Fei. Percepual losses for real-ime syle ransfer and super-resoluion. arxiv preprin arxiv: , , 2, 3, 4, 6, 7 [24] D. Kingma and J. Ba. Adam: A mehod for sochasic opimizaion. arxiv preprin arxiv: , [25] C. Li and M. Wand. Combining markov random fields and convoluional neural neworks for image synhesis. arxiv preprin arxiv: , , 3 [26] L. Li, S. Zhang, X. Yu, and L. Zhang. Pmsc: Pachmachbased superpixel cu for accurae sereo maching. IEEE Transacions on Circuis and Sysems for Video Technology, [27] J. Liao, R. S. Lima, D. Nehab, H. Hoppe, P. V. Sander, and J. Yu. Auomaing image morphing using srucural similariy on a halfway domain. ACM Transacions on Graphics (TOG), 33(5):168, , 5 [28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zinick. Microsof coco: Common objecs in conex. In Proc. ECCV, pages Springer, [29] Y. Liu, J. Ren, J. Liu, J. Zhang, and X. Chen. Learning selfiefriendly absracion from arisic syle images. arxiv preprin arxiv: , [30] W.-Y. Lo, J. Van Baar, C. Knaus, M. Zwicker, and M. Gross. Sereoscopic 3d copy & pase. ACM Transacions on Graphics (TOG), 29(6):147, [31] N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosoviskiy, and T. Brox. A large daase o rain convoluional neworks for dispariy, opical flow, and scene flow esimaion. In Proceedings of he IEEE Conference

10 on Compuer Vision and Paern Recogniion, pages , , 3, 4, 6, 8 [32] T.-J. Mu, J.-H. Wang, S.-P. Du, and S.-M. Hu. Sereoscopic image compleion and deph recovery. The Visual Compuer, 30(6-8): , [33] Y. Niu, W.-C. Feng, and F. Liu. Enabling warping on sereoscopic images. ACM Transacions on Graphics (TOG), 31(6):183, [34] M. Ruder, A. Dosoviskiy, and T. Brox. Arisic syle ransfer for videos. In Proc. GCPR, pages Springer, , 4, 6 [35] M. Ruder, A. Dosoviskiy, and T. Brox. Arisic syle ransfer for videos and spherical images. arxiv preprin arxiv: , , 5 [36] A. Selim, M. Elgharib, and L. Doyle. Paining syle ransfer for head porrais using convoluional neural neworks. ACM Trans. Graph. (Proc. of SIGGRAPH), 35(4):129, [37] L. Sheng, Z. Lin, J. Shao, and X. Wang. Avaar-ne: Muliscale zero-sho syle ransfer by feaure decoraion. arxiv preprin arxiv: , [38] K. Simonyan and A. Zisserman. Very deep convoluional neworks for large-scale image recogniion. arxiv preprin arxiv: , [39] T. Taniai, Y. Masushia, Y. Sao, and T. Naemura. Coninuous 3D Label Sereo Maching using Local Expansion Moves. Transacions on Paern Analysis and Machine Inelligence (TPAMI), (acceped). 2 [40] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempisky. Texure neworks: Feed-forward synhesis of exures and sylized images. arxiv preprin arxiv: , , 3 [41] L. Wang, H. Jin, and R. Yang. Search space reducion for mrf sereo. Compuer Vision ECCV 2008, pages , [42] L. Wang, H. Jin, R. Yang, and M. Gong. Sereoscopic inpaining: Join color and deph compleion from sereo images. In Compuer Vision and Paern Recogniion, CVPR IEEE Conference on, pages 1 8. IEEE, , 2 [43] K. Yamaguchi, T. Hazan, D. McAlleser, and R. Urasun. Coninuous markov random fields for robus sereo esimaion. Compuer Vision ECCV 2012, pages 45 58, [44] J. Zbonar and Y. LeCun. Sereo maching by raining a convoluional neural nework o compare image paches. Journal of Machine Learning Research, 17(1-32):2, , 3 [45] F. Zhang and F. Liu. Casual sereoscopic panorama siching. In Proceedings of he IEEE Conference on Compuer Vision and Paern Recogniion, pages ,