Hierarchical Recurrent Filtering for Fully Convolutional DenseNets

Transcription

1 Hierarchical Recurren Filering for Fully Convoluional DenseNes Jo rg Wagner1,2, Volker Fischer1, Michael Herman1 and Sven Behnke2 1- Bosch Cener for Arificial Inelligence Renningen - Germany 2- Universiy of Bonn - Compuer Science VI, Auonomous Inelligen Sysems - Friedrich-Eber-Allee 144, Bonn - Germany Absrac. Generaing a robus represenaion of he environmen is a crucial abiliy of learning agens. Deep learning based mehods have grealy improved percepion sysems bu sill fail in challenging siuaions. These failures are ofen no solvable on he basis of a single image. In his work, we presen a parameer-efficien emporal filering concep which exends an exising single-frame segmenaion model o work wih muliple frames. The resuling recurren archiecure emporally filers represenaions on all absracion levels in a hierarchical manner, while decoupling emporal dependencies from scene represenaion. Using a synheic daase, we show he abiliy of our model o cope wih daa perurbaions and highligh he imporance of recurren and hierarchical filering. 1 Inroducion A robus and reliable percepion and inerpreaion of he environmen is a crucial compeence of auonomous sysems. Deep learning based mehods grealy advanced he generaion of robus environmen represenaions and dominae he majoriy of percepion benchmarks. From a safey poin of view, a major drawback of popular daases is heir recording a dayime under good or normal environmen condiions. In order o deploy auonomous sysems in an unconsrained world wihou any supervision, one has o make sure ha hey sill work in challenging siuaions such as sensor ouages or heavy weaher. These siuaions induce failures of he percepion algorihm, which are no solvable by jus using a single image. We denoe hese failures in accordance o Kendall e al. [1] as aleaoric failures. To ackle such failures, one has o enhance he informaion provided o he percepion algorihm. This can be achieved by using a beer sensor, fusing informaion of muliple sensors, uilizing addiional conex knowledge, or by inegraing informaion over ime. In his paper, we focus on using emporal coherence o reduce aleaoric failures of a single-frame segmenaion model. We build upon he Fully Convoluional DenseNe (FC-DenseNe) [2] and propose a emporal filering concep, which exends i o work wih muliple frames. The emporal inegraion is achieved by recurrenly filering he represenaions on all absracion levels in a hierarchical manner. Due o he hierarchical naure of he filer concep, our model he Recurren Fully Convoluional DenseNe (RFC-DenseNe) can uilize emporal correlaions on all absracion levels. Addiionally, he RFCDenseNe decouples emporal dependencies from scene represenaion, which increases is ransparency and enables a direc ransformaion of any FC-DenseNe. 49

2 Despie is recurren naure, he proposed model is highly parameer efficien. Using simulaed daa, we show he abiliy of RFC-DenseNe o reduce aleaoric failures and highligh he imporance of recurren and hierarchical filering. 2 Relaed Work The majoriy of research, focused on using emporal informaion o reduce aleaoric failures of segmenaion models, applies a non-hierarchical filer approach. Valipour e al. [3] generae a represenaion for each image in a sequence and use a Recurren Nework o emporally filer hem. Jin e al. [4] uilize a sequence of previous images o predic a represenaion of he curren image. The prediced represenaion is fused wih he curren one and propagaed hrough a decoder model. Similar approaches exis, which apply pos-processing seps on op of frame segmenaions. Kundu e al. [5] use a Condiional Random Field operaing on an Euclidean feaure space, opimized o minimize he disance beween feaures associaed wih corresponding poins in he scene. Our approach differs from he above mehods due o is hierarchical naure. Tran e al. [6] build a semanic video segmenaion nework using spaioemporal feaures compued wih 3D convoluions. We differ from his approach due o he explici uilizaion of recurren filers. The Recurren Convoluional Neural Nework of Pavel e al. [7] is similar o our archiecure. This mehod uses layer-wise recurren self-connecions as well as op-down connecions o sabilize represenaions. The approach focuses on a fully recurren opology, while our approach decouples emporal filering and scene represenaion. Addiionally, our approach uses a dense connecion paern o ge an improved signal flow. 3 Recurren Fully Convoluional DenseNes 3.1 Revisiing he Fully Convoluional DenseNe (FC-DenseNe) The FC-DenseNe [2] is consruced by using a fully convoluional version of DenseNe as feaure exracor, uilizing a Dense Block () enhanced upsampling pah, and inerlinking boh pahs using skip connecions (Fig. 1). The DenseNe, used in he feaure exracor, is a convoluional nework, which ieraively adds feaures o a sack, he global feaure sae. Newly added feaures r are compued using all previous ones [r 1,..., r i,0 ] of maching spaial size: ([r 1,..., r i,0 ]; θ ); r = f i=0 is he funcion of he Dense Uni () wih parameers TD i=1 Feaure Exracor TD i=2 conv Skip Connecions oupu (1) i=4 Upsampling-pah TU i=3 θ. Dense Block () r i,0 r l=1 r l=2 r l=3 [r, r, r ] conv inpu where f TD: Transiion Down (BN - ReLU Conv Pool : Dense Uni (BN - ReLU Conv) FC-DenseNe TU: Transiion Up (3 3 Deconvoluion, sride of 2) TU Fig. 1: FC-DenseNe of deph wo wih hree layers per Dense Block. 50

3 3.2 Temporal Represenaion Filering Due o perurbaions inheren in he daa, he feaures r compued in each Dense Uni are only a crude approximaion of he rue represenaion r. To ge an improved esimae, we propose o filer hem using a Filer Module (): (f ([r 1,..., r i,0 ]; θ ), m 1 r = f ; θ ); (2) where f is he filer funcion wih parameers θ and hidden sae m 1. FC-DenseNe can be ransformed ino our proposed RFC-DenseNe by using a recurren version of he Dense Block (see Fig. 2), which employs a Filer Module afer each Dense Uni. To compue a robus segmenaion, RFCDenseNe uilizes he informaion of muliple images. These images are propagaed hrough he feaure exracor and he subsequen upsampling pah, while aking emporal correlaions via he Filer Modules ino accoun. The RFC-DenseNe adds filered feaures r o he global feaure sae of he model. Feaures r compued in each Dense Uni are hus derived from already filered ones, generaing a hierarchy of filered represenaions. Due o he hierarchical filer naure, RFC-DenseNe can uilize emporal correlaions on all absracion levels. In comparison, a non-hierarchical filer only has access o a sequence of high-level represenaions. The availabiliy of all feaures, required o solve aleaoric failures wihin he filer, is no guaraneed in such a seing. The RFC-DenseNe decouples emporal dependencies from scene represenaion by using dedicaed Filer Modules. This propery makes i easy o ransform any single-image FC-DenseNe ino a corresponding muli-image RFC-DenseNe. One could also use he weighs of a FC-DenseNe o iniialize he non-recurren par of he corresponding RFC-DenseNe. The decoupling addiionally increases he ransparency of he model, enabling a dedicaed allocaion of resources for emporal filering and hierarchical feaure generaion (scene represenaion). The proposed filer approach can also be employed in oher models, bu is especially suiable for he FC-DenseNe archiecure. The explici differeniaion of newly compued feaures r and feaures sored in he global feaure sae makes he emporal filering very parameer efficien. Each filer only has o process a small number of feaure maps resuling in a moderae increase in he oal number of model parameers. The disinc focus on a small feaure se in each Filer Module also reduces he compuaional complexiy of he filer ask. 3.3 Insances of he Filer Module m 2 i,0 l=1 r 1 l=1 m 1 m 2 l=2 r 1 l=2 m 1 m 2 l=3 r 1 l=3 1 1 [, r, r ] We invesigaed hree insances of he Filer Module wih increasing complexiy. All modules are based on Convoluional Long Shor Term Memory cells (Conv- m 1 Fig. 2: Recurren Dense Block using a Filer Module afer each Dense Uni. 51

4 1 [ 1,..., r i,0 ] m 2 r 1 BN ReLU e 1 c 1 h 1 Conv-LSTM (ed ) (ff ) (res ) m 1 Fig. 3: Three separae Filer Module insances: ff, res and ed. LSTMs), which have proven o produce sae-of-he-ar resuls on a muliude of spaio-emporal sequence modeling asks. The Conv-LSTM is defined by: k c h = ek hk k σ(w e W h 1 b ), k {i, f, o} ; (3) = f c 1 i o anh(c ); (4) = ec anh(w e hc W h 1 bc ); (5) where is he convoluional operaor, he Hadamard produc, and e he 1 1 inpu. The hidden sae m 1 of Equaion 2 is a summary of c and h. A propery of all Filer Modules are maching dimensions beween he unfilered e. r and filered r represenaion and a filer size of 3 3 for all kernels W The Filer Module ff (Fig. 3, green) uses a single Conv-LSTM following he wo pre-acivaion layers (Bach Normalizaion (BN) and Recified Linear Uni (ReLU)). The number of feaure maps sored in he cell sae c 1 maches he number of feaure maps of he unfilered represenaion. This propery resrics he filer capabiliies bu also limis he number of required parameers. The Filer Module res (Fig. 3, green and blue) uses he concep of residual learning [8] and applies i o ff. The inroduced skip connecion ensures a direc informaion and gradien flow, prevening signal degradaion. ed (Fig. 3, green and red) alleviaes he limiaion on he complexiy of he filer, inroduced by maching feaure dimensions of he unfilered represenaion and he cell sae. This insance employs an encoder-decoder srucure, consising of he Conv-LSTM and a Dense Uni. The number of feaure maps sored in he Conv-LSTM can be chosen o be a muliude αed of he number of unfilered maps. A drawback of ed is he increased parameer coun Experimenal Resuls Daase To evaluae he proposed models, we use a simulaed daase (see Fig. 4). The sequences emulae a 2D environmen of pixels, in which squares represen dynamic and saic objecs, recangles represen borders and walls, and circles represen moving foreground objecs. Each square is marked wih a random MNIST digi. The dynamic squares elasically collide wih all oher squares, borders and walls. The moving circles occlude each oher, as well as all oher objecs. The number of objecs, heir size and color, he color of MNIST digis as well as he velociy of all dynamic objecs is randomly sampled for each sequence. To simulae aleaoric failures, we perurb he daa wih noise, ambiguiies, missing informaion, and occlusions. Noise is simulaed by adding zero mean 52

5 Model deph Layers per Inpu sequence Ground ruh label of he las image Feaures per FCDb FCDs Feaures of he firs convoluion Fig. 4: Tes sequence of lengh 5 wih label. Table 1: FC-DenseNes. Gaussian noise o each pixel. Occlusions are inroduced by he foreground circles. To simulae missing informaion, we increase or decrease he inensiy of pixels by a random value and le his offse decay. This effec is added o whole images and o subregions. Ambiguiies are simulaed using differen classes for saic and dynamic squares. Our daase conains 25,000 independenly sampled sequences of lengh 5, which are spli ino 20,000 raining, 4,000 validaion and 1,000 es sequences. We addiionally generae a clean es se wih no aleaoric failures. For he segmenaion ask, we define 14 classes: background, borders and walls, saic squares, circles, and dynamic squares wih one class per MNIST digi. A label is only available for he las image in each sequence. 4.2 Models We perform a grid search o find he bes FC-DenseNe (FCDb ). The grid parameers are lised in Table 1. Our emporal models are buil based on a smaller version of he FC-DenseNe (FCDs ) o reduce raining ime. In oal, we rain seven emporal models: Four RFC-DenseNes using our filer concep (Table 2), a recurren, non-hierarchical model RMgf (cf. [3]), and wo non-recurren models: TM3D and TMs. RMgf globally filers he represenaion generaed in he las Dense Block of FCDs using ed wih αed = and a hidden-o-hidden filer size of 9. TM3D and TMs are FC-DenseNes: one using Filer Module 3D convoluions of size and one operaing RFCDff ff on sacked inpu sequences. The filer size of kernels RFCDres res h W in our RFC-DenseNes are se o [9, 5, 3, 5, 9]. RFCDed1 ed, αed = 1 RFCDed2 ed, αed = 2 All emporal models, excep for RFCDed2, have roughly he same number of parameers. Table 2: RFC-DenseNes. 4.3 Evaluaion We summarize he resuls in Table 3 by reporing he mean Inersecion over Union (mean IoU) on he es daase, as well as he clean es daase. All models which uilize emporal informaion significanly ouperform he single-image FC-DenseNes on he es daa showing he imporance of emporal filering. On he clean es daa, he single-image and muli-image models are roughly on par, suggesing ha emporal informaion especially benefis he reducion of aleaoric failures. The superior performance of FCDb on he clean Tes daase Clean es daase FCDb % % FCDs % % RFCDff % % RFCDres % % RFCDed % % RFCDed % % RMgf % % TM3D % % TMs % % Table 3: Mean IoU of he differen models on he es and clean es daase. 53

6 es daa compared o mos of he emporal models can mos likely be aribued o is increased non-emporal deph and widh. The mean IoUs of he differen RFC-DenseNes sugges a correlaion beween filer complexiy and performance. However, he difference is relaively small. The performance of RFCDed2 is unexpecedly poor in comparison o RFCDed1. Looking a class-wise IoUs did no provide any addiional insigh regarding possible sysemaic failures. We plan o furher invesigae oher recurren regularizaion echniques o improve he performance of RFCDed2. The hierarchical RFC-DenseNe models ouperform he non-hierarchical, recurren baseline RMgf by 2 % o 4 % on he es daa, showing he superioriy of our models in he reducion of aleaoric failures. We suspec ha our hierarchical filer concep beer uilizes emporal informaion, compared o a non-hierarchical approach. Taking only he diverse MNIST digis ino accoun, he performance difference increases furher. For hese classes, i is imporan o model low-level emporal dependencies. The non-hierarchical approach possibly suffers, because of he loss in scene deails from lower o upper layers. The performance of he non-recurren models, TM3D and TMs, is inferior o he recurren ones. This is mos likely due o he explici emporal srucure of recurren models, which benefis he deecion of emporal correlaions. 5 Conclusion In his work, we proposed a parameer-efficien approach o emporally filer he represenaions of he FC-DenseNe in a hierarchical fashion, while decoupling emporal dependencies from scene represenaion. Using a synheic daase, we showed he benefis of using emporal informaion in regards o aleaoric failures, as well as he advanages inroduced by our recurren and hierarchical filering concep. In he fuure, we plan o evaluae our approach on real-world daases. References [1] A. Kendall and Y. Gal. Wha Uncerainies Do We Need in Bayesian Deep Learning for Compuer Vision? In arxiv: , [2] S. Je gou, M. Drozdzal, D. Va zquez, A. Romero, and Y. Bengio. The one hundred layers iramisu: Fully convoluional DenseNes for semanic segmenaion. In CVPR, [3] S. Valipour, M. Siam, M. Jagersand, and N. Ray. Recurren fully convoluional neworks for video segmenaion. In WACV, pages 29 36, [4] X. Jin, X. Li, H. Xiao, X. Shen, Z. Lin, J. Yang, Y. Chen, J. Dong, L. Liu, e al. Video scene parsing wih predicive feaure learning. arxiv preprin arxiv: , [5] A. Kundu, V. Vinee, and V. Kolun. Feaure space opimizaion for semanic video segmenaion. In CVPR, pages , [6] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Deep end2end voxel2voxel predicion. In CVPR Workshops, pages 17 24, [7] M. S. Pavel, H. Schulz, and S. Behnke. Objec class segmenaion of RGB-D video using recurren convoluional neural neworks. Neural Neworks, 88: , [8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recogniion. In CVPR, pages , June