SENSING using 3D technologies, structured light cameras

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER Real-Time Enhancemen of Dynamic Deph Videos wih Non-Rigid Deformaions Kassem Al Ismaeil, Suden Member, IEEE, Djamila Aouada, Member, IEEE, Thomas Solignac, Bruno Mirbach, and Bj orn Oersen, Fellow, IEEE Absrac We propose a novel approach for enhancing deph videos conaining non-rigidly deforming objecs. Deph sensors are capable of capuring deph maps in real-ime bu suffer from high noise levels and low spaial resoluions. While soluions for reconsrucing 3D deails in saic scenes, or scenes wih rigid global moions have been recenly proposed, handling unconsrained non-rigid deformaions in relaive complex scenes remains a challenge. Our soluion consiss in a recursive dynamic muli-frame superresoluion algorihm where he relaive local 3D moions beween consecuive frames are direcly accouned for. We rely on he assumpion ha hese 3D moions can be decoupled ino laeral moions and radial displacemens. This allows o perform a simple local per-pixel racking where boh deph measuremens and deformaions are dynamically opimized. The geomeric smoohness is subsequenly added using a muli-level L 1 minimizaion wih a bilaeral oal variaion regularizaion. The performance of his mehod is horoughly evaluaed on boh real and synheic daa. As compared o alernaive approaches, he resuls show a clear improvemen in reconsrucion accuracy and in robusness o noise, o relaive large non-rigid deformaions, and o opological changes. Moreover, he proposed approach, implemened on a CPU, is shown o be compuaionally efficien and working in real-ime. Index Terms Deph enhancemen, super-resoluion, non-rigid deformaions, regisraion, Kalman filering, bilaeral oal variaion Ç 1 INTRODUCTION SENSING using 3D echnologies, srucured ligh cameras or ime-of-fligh (ToF) cameras, has seen a revoluion in he pas years where sensors such as he Microsof Kinec version 1 and 2 are oday par of accessible consumer elecronics [1]. The abiliy of hese sensors in direcly capuring deph videos in real-ime has opened remendous possibiliies for applicaions in gaming, roboics, surveillance, healh care, ec. These sensors, unforunaely, have major shorcomings due o heir high noise conaminaion, including missing and jagged measuremens, and heir low spaial resoluions. This makes i impossible o capure deailed 3D feaures indispensable for many 3D compuer vision algorihms. The face daa in Fig. 1a is an example of such challenging raw deph measuremens. Running a radiional face recogniion algorihm on his ype of daa would resul in a very low recogniion rae [2], [3], [4]. Some soluions have been proposed in he lieraure for recovering hese deails bu mosly in he conex of saic 3D scene scanning, wih LidarBoos [5] and is exension [6] and KinecFusion [7] being he mos known mehods. The curren major challenge is when he objec or objecs in he scene are K. Al Ismaeil, D. Aouada, and B. Oersen are wih Inerdisciplinary Cenre for Securiy, Reliabiliy, and Trus, Universiy of Luxembourg, Eschsur-Alzee L-2721, Luxembourg. {kassem.alismaeil, djamila.aouada, bjorn.oersen}@uni.lu. T. Solignac and B. Mirbach are wih he Advanced Engineering Deparmen, IEE S.A, Conern L-5326, Luxembourg. {homas.solignac, bruno.mirbach}@iee.lu. Manuscrip received 6 Aug. 2015; revised 21 Aug. 2016; acceped 30 Sep Dae of publicaion 26 Oc. 2016; dae of curren version 12 Sep Recommended for accepance by T. Brox. For informaion on obaining reprins of his aricle, please send o: reprins@ieee.org, and reference he Digial Objec Idenifier below. Digial Objec Idenifier no /TPAMI subjec o non-rigid deformaions. Indeed, LidarBoos and KinecFusion are rigid deph fusion approaches, and hey immediaely fail in providing any reasonable resul on nonrigidly deforming scenes, he focus of his paper. In [8], [9], [10], we proposed he UP-SR algorihm, which sands for Upsampling for Precise Super-Resoluion, as he firs dynamic muli-frame deph video super-resoluion (SR) algorihm ha can enhance deph videos conaining non-rigidly deforming scenes wihou any prior assumpion on he number of moving objecs hey conain or on he opology of hese objecs. These advanages were possible hanks o a direc processing on deph maps wihou using conneciviy informaion inheren o meshing as used in subsequen mehods, namely, KinecDeform [11] and DynamicFusion [12]. The UP- SR algorihm is, however, limied o laeral moions as i only compues 2D dense opical flow bu does no accoun for he full moion in 3D, known as scene flow, or he 2.5D moion, known as range flow. I consequenly fails in he case of radial deformaions. Moreover, i is no pracical because of a heavy cumulaive moion esimaion process applied o a number of frames buffered in he memory. This paper presens a soluion ha improves over he UP-SR algorihm by keeping is advanages and solving is wo limiaions no considering 3D moions and using an inefficien cumulaive moion esimaion. The proposed soluion is based on he assumpion ha he 3D moion of a poin can be approximaed by decoupling he radial componen from he laeral ones. This approximaion allows he handling of non-rigid deformaions while reducing he compuaional complexiy associaed wih an explici full 3D moion esimaion a each poin. Moreover, a recursive deph muli-frame SR is formulaed by replacing UP- SR s cumulaive moion esimaion wih a poin-racking ß 2016 IEEE. Personal use is permied, bu republicaion/redisribuion requires IEEE permission. See hp:// for more informaion.

2 2046 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER 2017 Fig. 1. Resuls of differen SR mehods for a scale facor of r ¼ 4 applied o a low resoluion dynamic deph video capured wih a ToF camera a a rae of 50 frames per ms. (a) Raw frame, (b) Bicubic inerpolaion, (c) SISR [31], (d) UP-SR [8], (e) Proposed recup-sr. Unis are in mm. operaion locally a each pixel. Similarly o earlier approaches for a recursive SR [13], [14], [15], [16], we use a Kalman filer for racking excep ha we rea each pixel separaely as opposed o considering he full image. As a resul, he proposed soluion efficienly runs muliple Kalman filers in parallel on local deph values and on heir radial displacemens. A subsequen processing is required in order o recover he smoohness propery of a deph map and correc he arifacs caused by his per-pixel filering. To ha end, we propose a muli-level version of he L 1 minimizaion wih a bilaeral oal variaion (BTV) regularizaion originally given in [17]. The proposed algorihm leads o a new approach for esimaing range flow by pixel racking wih a Kalman filer. I is imporan o noe ha while his flow conribues in handling non-rigid deformaions in 3D, he effeciveness of he proposed SR algorihm comes from how his flow is employed in he UP-SR reconsrucion framework. Indeed, merely applying he esimaed range flow in anoher deph SR mehod does no give saisfacory resuls. An overview of he proposed algorihm, named recup-sr, is given in Fig. 2. A firs visual illusraion of he performance of he proposed algorihm on a low qualiy deph video of a highly non-rigidly deforming face is given in Fig. 1e. In summary, he conribuion of his paper is a new muli-frame deph SR algorihm which has he following properies: 1) Accuracy in deph video reconsrucion. 2) Robusness o non-rigid deformaions and o noise. 3) Robusness o opological changes. 4) Independence of he number of moving objecs in he scene. 5) Real-ime implemenaion on a CPU. This paper is an exended version of [18] wih addiional heoreical deails and clarificaions explaining he ransiion from he earlier UP-SR algorihm o he recup-sr algorihm proposed herein, and an exended explanaion of he proposed muli-level ieraive deblurring. A significanly exended experimenal par is also provided conaining original analyses based on new resuls. The remainder of he paper is organized as follows: Secion 2 gives an overview of he differen caegories of deph enhancemen approaches and a review of range flow algorihms. Secion 3 formulaes he problem of deph video SR and describes he necessary background on UP-SR. The proposed recursive deph video SR algorihm is presened in Secion 4. Experimenal evaluaions are presened and discussed in Secion 5. Finally, he conclusion is given in Secion 6. 2 RELATED WORK Three caegories of approaches for he enhancemen of deph videos conaining non-rigid deformaions may be disinguished as deailed below. 2.1 Muli-Modal Fusion Approaches This caegory of mehods is based on he assumpion ha here is a correspondence beween he edges of a deph map and he edges of anoher modaliy of a beer qualiy, ofen chosen o be a 2D inensiy image of he same scene [19], [20], [21], [22], [23], [24], [26]. Such an image is considered o be a guidance image whose properies, in erms of srucure, signal o noise raio (SNR) and resoluion, can be Fig. 2. Flow char of recup-sr: A new muli-frame deph super-resoluion algorihm for dynamic deph videos of non-rigidly deforming objecs.

3 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2047 ransferred o is corresponding deph map in order o obain an enhanced version. Diebel and Thrun proposed he firs work in his caegory [19]. Their approach consiss in a muliresoluional Markov Random Field (MRF) defined o inegrae boh modaliies, deph and 2D inensiy, and used o esimae a deph map of he same resoluion as he resoluion of he inensiy image. In 2007, Kopf e al. [20] and Yang e al. [21] proposed he concep of join bilaeral upsampling (JBU), which is an adaped version of he bilaeral filer [27] for fusing a low resoluion (LR) deph map wih a high resoluion (HR) 2D image. In [22], [26], he JBU muli-modal filer was exended o considering no only a 2D image as a guidance image bu also he deph map iself. These weighing-based filers may be viewed as implici guided regularizaion approaches. In an explici guided regularizaion approach, a regularizaion funcion ha ranslaes he prior properies of a guidance image is added o a daa fideliy erm o form a cos funcion o be opimized as in [28], [30]. These more sophisicaed approaches are as of oday he mos effecive ones in he caegory of muli-modal fusion. Anoher varian in his caegory are mehods using passive sereo imaging in combinaion wih low qualiy acive deph sensors [23]. However, besides he fac ha all approaches in his caegory require an addiional camera, hey are also highly dependen on he assumpion of correspondence beween 2D inensiy and deph images. This requires a perfec calibraion and synchronizaion of a muli-camera sysem and a perfec daa mapping. 2.2 Learning-Based Approaches The firs work in his caegory is by Mac Aodha e al. known as pach-based single image SR (SISR) [31]. This mehod, as implied by is name, follows a pach-based approach where an LR pach is reconsruced from a large dicionary of synheic noise-free HR deph paches. The boundaries of overlapping paches follow a special reamen o keep smoohness properies. An imporan conribuion of [31] is he large daabase of HR deph maps used o creae he dicionary. Hornacek e al. proposed in [32] an adapaion of [33] o deph maps. The idea here is o run a search for similar paches only using he repeiive informaion already available in he deph map o be enhanced. To compue he similariy beween deph paches, he PachMach algorihm by Barnes e al. [34] was redefined in 3D under invariance o 3D rigid moions. In [35], a pachwork assembly algorihm for deph single image SR was proposed merging he concep of using a raining daabase [31] and he concep of self-similariy [32]. The problem was formulaed as an MRF opimizaion which could opionally add informaion from an HR 2D inensiy image following he same principle of muli-modal fusion approaches. Anoher work a he fronier of he wo caegories, learning-based and muli-modal fusion approaches, was proposed earlier by Li e al. in [36]. The auhors proposed o learn a mapping funcion beween HR 2D paches and heir corresponding LR deph paches using a raining daabase. A sparse coding algorihm was hen used for he final enhancemen. The performance of he mehods in his caegory rely on he qualiy of he raining daabase and/or on he repeiive srucure of he inpu deph map. In general, hese condiions are no always available or verified. 2.3 Dynamic Muli-Frame Approaches Using muliple frames o recover deph deails has been successful in he case of saic scenes or scenes wih global rigid moion [5], [6], [7]. Since hese mehods and heir immediae derivaives, he real challenge ha he research communiy has been facing is exending he muli-frame deph enhancemen concep o scenes wih non-rigid deformaions. There have been few aemps o handle single objec scanning under relaive small non-rigidiies by replacing a global rigid regisraion wih a non-rigid alignmen [37], [38], [39]. These echniques, however, canno handle large deformaions, and are no very pracical for real-ime applicaions. Real-ime non-rigid reconsrucion approaches have been achieved wih he help of a emplae which is firs acquired hen used for racking of non-rigidiies wih a good flexibiliy [40], [41]. Recenly, we have proposed KinecDeform [11], he firs nonrigid version of KinecFusion. I does no require any emplae, and similarly o KinecFusion, provides an enhanced smooher reconsrucion over ime wih he addiion of handling nonrigid deformaions in he scene. KinecDeform has been successfully esed on an Asus Xion Pro Live camera [42], equivalen o Microsof Kinec srucured ligh version 1. I canno, however, perform well on lower resoluion, noisier ToF cameras such as he PMD camboard nano [43]. Indeed, is regisraion module requires denser raw acquisiions. DynamicFusion is anoher recen non-rigid version of KinecFusion. Thankso a GPU implemenaion, i has been esed on a Kinec camera in real-ime. However, is reconsrucion accuracy has no been evaluaed, and i has only been validaed visually. Moreover, i builds on he assumpion of having only one moving objec in he scene. In addiion, is repored limiaions are is sensiiviy o complex scenes and scenes wih changes in opology. Also, similarly o KinecDeform, one may suspec DynamicFusion no o be able o perform well on a lower resoluion noisier ToF camera. The UP-SR algorihm falls under his caegory of dynamic muli-frame approaches. Indeed, i explois he deformaions colleced over ime in an inverse SR reconsrucion framework. UP-SR was shown in [8] and [10] o have a higher reconsrucion accuracy as compared o represenaive mehods from he firs and second caegories of deph enhancemen mehods. Is major drawbacks, however, are is limiaion o laeral moions and is compuaional complexiy. Using range flow in place of opical flow is a naural firs sep owards reconsrucing non-laeral, i.e., radial, local deformaions. In wha follows, we review relaed lieraure on range flow esimaion. 2.4 Range Flow Esimaion In order o handle 3D non-rigid deformaions, i is imporan o consider he full 3D moion per pixel or he range flow. The range flow consrain has been firs proposed in [45], and laer appeared in oher references [46], [47], [48], [49]. This consrain is usually used in a variaional framework o esimae he range flow. However, esimaing a dense range flow, i.e., a hree dimensional vecor for each poin is sill compuaionally complex and no achievable in real-ime, a leas, no wih a sub-pixel accuracy [50]. Using RGB-D deph cameras has allowed a muli-modal approach for range flow esimaion by defining a global energy funcional combining he range flow and he 2D opical flow

4 2048 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER 2017 consrains. In addiion o he challenge of reducing he compuaional complexiy, hese algorihms have o handle erroneous measuremens of deph sensors such as flying pixels, and missing and invalid values. A common approach is o define a smoohness condiion o be used as a regularizaion erm in he global opimizaion [29], [49], [51]. In [52] and [53], a probabilisic approach is followed by using a paricle filer. This concep is he closes o our range flow esimaion where informaion is recursively propagaed o he nex frame. Recenly, he atgv-sf algorihm has been proposed where he flow is direcly calculaed in 3D by back-projecion, resuling in a join esimaion of he laeral and radial moions [29]. To our knoweledge, his is repored o be currenly he bes performing range flow algorihm in erms of accuracy and runime. We will use i as a reference in our evaluaion. As explained in Secion 1, he curren paper improves UP-SR. In Secion 3, he necessary background on UP-SR is given afer formalizing he problem of muli-frame SR. 3 BACKGROUND AND PROBLEM FORMULATION The following noaion will be adoped: marices are denoed by boldface uppercase leers A, and vecors or column images are denoed by boldface lowercase leers a. Scalars are denoed by ialic leers, a; A. ^a: esimae of a. ~a: measured a. a i : elemen i of a. a : a a ime. a 0 1 : regisered a from 1 o 0. a ": upsampled a. 3.1 Muli-Frame Super-Resoluion Le us consider an LR video fg g acquired wih a deph sensor. The capured scene is assumed o be dynamically and non-rigidly deforming wihou any assumpion on he number of moving objecs. Each LR observaion g is represened by a column vecor of lengh m corresponding o he lexicographic 1 ordering of frame pixels. The objecive of deph SR is o reconsruc an HR deph video ff g using fg g, where each frame f is of lengh n wih n ¼ r 2 m such ha r 2 N is he SR scale facor. In he classical muli-frame deph SR problem, in order o reconsruc a given frame f 0 2ff g, also known as he reference frame, he N preceding observed LR frames are used. An LR observaion g is relaed o he reference frame hrough he following daa model g ¼ DHM 0 f 0 þ n ; 0 ; (1) where D is a known consan downsampling marix of dimension ðm nþ. The sysem blur is represened by he ime and space invarian marix H. The ðn nþ marices M 0 correspond o he moion beween f 0 and g before downsampling. The vecor n is an addiive whie noise a ime insan. Wihou loss of generaliy, boh H and M 0 are assumed o be block circulan commuaive marices. As a resul, he esimaion of f 0 may be decomposed ino wo seps; esimaion of a blurred HR image z 0 ¼ Hf 0, followed by a deblurring sep o recover ^f Lexicographic ordering: Concaenaion of he columns of he image. The above framework has been firs proposed in he case of saic 2D scenes in [17] and for saic deph scenes in [44]. In [8] and in [10] i has been exended o dynamic deph scenes defining he UP-SR algorihm. 3.2 Upsampling for Precise Super-Resoluion (UP-SR) The UP-SR algorihm sars by a dense upsampling of all he LR observaions. This is shown o ensure a more accurae regisraion of frames. The resuling r 2 -imes upsampled image is defined as g "¼ U g, where U is he ðn mþ dense upsampling marix. I is chosen o be he ranspose of he downsampling marix D. As a resul, he produc UD ¼ A gives a block circulan marix A ha defines a new blurring marix B ¼ AH. Therefore, he esimaion of f 0 goes hrough he esimaion of is new blurred version z 0 ¼ Bf 0. In order o esimae z 0, he N frames preceding g 0 are required. Every wo consecuive frames are relaed by he following dynamic model g þ1 "¼ M þ1 g "þd þ1 ; (2) where M þ1 is he moion beween hem. The vecor d þ1 is referred o as he innovaion image. I conains novel measuremens appearing, or disappearing due o occlusions or large moions. Noe ha, in he UP-SR framework, he innovaion is assumed o be negligible, and he marix M þ1 is assumed o be an inverible permuaion. I can be esimaed using classical dense 2D opical flow. The elemens of he esimaed ^M þ1 marix are 1 s and 0 s, basically indicaing he address of he source pixels in g " and he address of he desinaion pixel in g þ1 ". This informaion is equivalen o finding for each pixel posiion p i ¼ðxi ;yi Þ; i ¼ 1;...; n, he horizonal and verical displacemens in pixels u i and vi, respecively. In he coninuous case, hese displacemens correspond o he laeral moions ðu i ;vi Þ where u i ¼ dxi d and v i ¼ dyi d. The small moions beween consecuive frames are hen cumulaed, and a frame g " is regisered o he reference frame g 0 " wih he following cumulaive moion compensaion approach g 0 "¼ ^M 0 g "¼ ^M ^M þ1 g " : (3) fflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflffl} ð 0 Þ imes As a resul, he original daa model in (1) is simplified o define he following UP-SR daa model g 0 "¼ z 0 þ n ; 0 ; (4) where n ¼ ^M 0 U n is an addiive noise vecor of lengh n. I is assumed o be independen and idenically disribued. Using an L 1 -norm, he blurred esimae ^z 0 is found by pixel-wise emporal median filering of he upsampled regisered LR observaions fg 0 "g. As a second and final sep, follows an image deblurring o esimae ^f 0 from ^z 0. The only considered moions in he UP-SR algorihm are laeral ones using 2D dense opical flow. Radial

5 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2049 displacemens in he deph direcion, ofen encounered in deph sequences, are herefore no handled. In order o address his problem, we propose o consider range flow in he UP-SR framework. 3.3 Range Flow Approximaion A ime-varying deph surface Z may be viewed as a mapping of a pixel posiion p i ¼ðxi ;yi Þ on he sensor image plane, a a ime insan, suchhap i 7!Zðxi ;yi Þ. The value Zðxi ;yi Þ corresponds o he ih elemen of he deph image z wrien in lexicographic vecor form, ha we will denoe in wha follows as z i. The deformaion of he surface Z from ð 1Þ o akes he poin p i 1 o a new posiion pi. I may be expressed hrough he derivaive of Z following he direcion of he 3D displacemen resuling in a range flow ðu i ;vi ;wi Þ where he radial displacemen in he deph direcion w i ¼ dzi d is added as he hird componen o he laeral displacemen. In his work, we propose o decouple he esimaion of he laeral moions ðu i ;vi Þ from he esimaion of he radial displacemen w i. Indeed, deph cameras provide an inensiy image a (2D image in he case of RGB-D sensors, or an ampliude image in he case of ToF sensors), which makes i possible o esimae he laeral moions direcly using he 2D opical flow consrain beween wo consecuive inensiy images a 1 and a. This decoupling approach enables o reduce he complexiy, bu also o inroduce a probabilisic framework ha allows o recursively esimae w i and he correced deph value a he same poin. Once ðu i ;vi Þ is esimaed, we proceed wih esimaing w i under a probabilisic framework where we accoun for radial moion uncerainies. 4 PROPOSED APPROACH The proposed deph video enhancemen approach is based on an exension of he UP-SR algorihm. As our goal is a real-ime processing, he major difference resides in replacing he cumulaion of N frames in UP-SR for processing a reference frame a ime 0, by a recursive processing ha only considers wo consecuive frames a ð 1Þ and where he curren frame is o be enhanced each ime. The measuremen model for each curren frame may be defined by seing 0 ¼ in (4), resuling in ~z i :¼½g "Ši ¼ z i þ½n "Š i 8; (5) where ½n "Š i is assumed o be zero mean Gaussian wih he variance s 2 n, i.e., ½n "Š i Nð0; s 2 nþ. The problem a hand is hen o esimae z i given a noisy measuremen ~z i and an enhanced noise-free deph value z i 1 esimaed a he preceding ieraion. The ime-deforming deph scene is viewed as a dynamic sysem where he sae of each pixel is defined by is deph value and radial displacemen. These saes are esimaed dynamically over ime using a Kalman filer. The UP-SR dynamic model in (2) is direcly used o characerize he dynamic sysem and inroduce he uncerainies of deph measuremens and radial deformaions in one probabilisic framework. The proposed recursive approach, ha we refer o as recup- SR, is summarized in he flow char of Fig. 2. The main seps are described in wha follows. Fig. 3. Correcing ampliude images using a sandardizaion sep [54]. (a) and (b) show he original LR ampliude images for a dynamic scene conaining a hand moving owards he camera where he inensiy (ampliude) values differ significanly depending on he objec disance from he camera. The correced ampliude images for he same scene are presened in (c) and (d), where he inensiy consisency is preserved. 4.1 Laeral Regisraion In order o be able o carry a per-pixel processing, essenial for handling non-rigid deformaions, one needs o properly align hese pixels beween consecuive frames. This is achieved by regisraion hrough 2D dense opical flow ha esimaes he laeral moion beween he inensiy images a 1 and a. In he case of RGB-D cameras, hese images are provided direcly. Mapping and synchronizaion have o be ensured, hough, as in [49] and [51]. In he case of ToF cameras, he provided inensiy images, known as ampliude images, can no be used direcly. Their inensiy values differ significanly depending on he camera inegraion ime and on he disance of he scene from he camera; hence, no verifying he opical flow assumpion of brighness consisency. Thus, in order o guaranee an accurae regisraion, i is necessary o apply a sandardizaion sep similar o he one proposed in [54] prior o moion esimaion, see Fig. 3. If inensiy images are no available, for example when using synheic daa, he 2D opical flow can be direcly esimaed using LR raw deph images, bu afer a denoising sep (e.g., using a bilaeral filer). We noe ha his denoising should only be used in he preprocessing sep. The original raw deph daa is he one o be mapped in he regisraion sep. In all cases, as for UP-SR, we regiser he upsampled versions of he LR images afer upscaling he moion vecors esimaed from he LR images. We define he regisered deph image from ð 1Þ o as z 1. Consequenly, he radial displacemen w i may be iniialized by he emporal difference beween deph measuremens, i.e., w i ~zi ½z 1 Ši : (6) This firs approximaion of w i is an iniial value ha requires furher refinemen direcly accouning for he sysem noise. We propose o do ha using a per-pixel racking wih a Kalman filer as deailed in Secion 4.2.

6 2050 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER Refinemen by Per-Pixel Tracking According o he definiion of image pixel regisraion, we have z i 1 :¼½z 1 Ši. The dynamic model follows from (2) as z i ¼ zi 1 þ mi 8; (7) where m is a noisy version of he innovaion d.whereas in UP-SR his innovaion is negleced, in recup-sr i is assimilaed o he uncerainy considered in he dynamic model. In his work, we assume a consan velociy modelwihanacceleraiong i following a Gaussian disribuion g i Nð0; s2 aþ. As a resul, he noisy innovaion maybe expressed as m i ¼ wi 1 D þ 1 2 gi D2 : (8) The dynamic model in (7) can hen be rewrien as: ( z i ¼ zi 1 þ wi 1 D þ 1 2 gi D2 w i ¼ wi 1 þ gi D : (9) Considering he following sae vecor s i ¼ zi w i ; (10) where boh he deph measuremen and he radial displacemen are o be filered, (9) becomes s i ¼ Ksi 1 þ gi ; (11) wih K ¼ð 1 D 0 1 Þ, and gi ¼ gi ð 1 2 D2 Þ is he process noise D which is whie Gaussian wih he covariance Q ¼ s 2 D 2 =4 D=2 a D2 : (12) D=2 1 Using sandard Kalman equaions, he predicion is achieved as ( ^s i j 1 ¼ Ksi 1j 1 ; ^P i j 1 ¼ (13) KPi 1j 1 KT þ Q; where P i is he error covariance marix. The error in he predicion of ^s i j 1 is correced using he observed measuremen ~z i. This error is considered as he difference beween he predicion and he observaion, and weighed using he Kalman gain marix G i j which is calculaed as follows: 1; G i j ¼ ^P i j 1 bt b^p i j 1 bt þ s 2 n (14) such ha he observaion! vecor is b ¼ð1; 0Þ T. The correced sae vecor s i j ¼ zi j and correced error covariance w i j marix P i j are compued as follows: 8 < s i j ¼ ^si j 1 þ Gi j ~z i b^si j 1 ; : P i j ¼ ^P i j 1 Gi j b^p i j 1 : (15) This per-pixel filering is exended o all he deph frame resuling in n Kalman filers run in parallel. Each filer racks he rajecory of one pixel. A his level, pixel rajecories are assumed o be independen. The advanage of he processing per pixel is o reduce all he required marix inversions o simple scalar inversions. The burden of radiional Kalman filer-based SR as in [13] will consequenly be avoided. Moreover, for a recursive muli-frame SR algorihm, insead of using a video sequence of lengh N o recover one frame, we use he preceding recovered frame ^f 1 o esimae f from he curren upsampled observaion g ". Furhermore, in order o separae background from foreground deph pixels, and ackle he problem of flying pixels, especially around edges, we define a condiion for rack re-iniializaion. This condiion is based on a fixed hreshold such ha ( Coninue he rack if j~z i ^zi j 1 j < ; New rack & spaial median if j~z i ^zi j 1 j5: The choice of he hreshold value is relaed o he ype of he used deph sensor and he level of he sensor-specific noise. This sep is very imporan for he overall performance of recup-sr. Wihou i, he emporal filering would coninue even if he regisered pixels from ime ð 1Þ o are no maching (e. g. a background pixel wrongly regisered o a foreground pixel) and hence he oucome of filering will be compleely wrong for he pixel under consideraion a ime. This is an ad hoc soluion ha provides saisfacory resuls for he considered examples. The assumpion of independen rajecories leads o blurring arifacs, and requires a correcive sep o bring back he correlaion beween neighbouring pixels from he original deph surface Z. To ha end, we use an L 1 minimizaion where we propose a muli-level ieraive BTV regularizaion as deailed in Secion Muli-Level Ieraive Bilaeral TV Deblurring Similarly o he UP-SR algorihm, f is esimaed in wo seps; firs, finding a blurred version ^z, which is he resul of Secion 4.2. Then he deblurring akes place o recover ^f from ^z. To ha end, we apply he following deblurring framework ^f ¼ argmin kbf ^z k 1 þ Gðf Þ ; (16) f where is a regularizaion parameer ha conrols he amoun of regularizaion needed o recover he original blur and noise-free frame. The marix B is he blur marix inroduced in Secion 3.2. We choose o use a BTV regularizer [17] in order o enforce he properies of bilaeral filering on he final soluion [27], [55], [56]. I is a filer ha has been shown o perform well on deph daa [20], [21], [22]. Indeed, i is a filer ha smoohes an image while preserving is sharp edges based on pixel similariies in boh he spaial and in he inensiy domains. The BTV regularizer is defined as Gðf Þ¼ Xp¼P X q¼p p¼ P q¼0 a jpjþjqj k f X p Y q f k 1 : (17)

7 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2051 TABLE 1 3D RMSE in mm for he Super-Resolved Dancing Girl Sequence Using Differen SR Mehods s ¼ 25 mm s ¼ 50 mm Arm Torso Leg Full body Arm Torso Leg Full body Bicubic SISR [31] UP-SR [8], [10] Proposed recup-sr The SR scale facor is r ¼ 4. The marices X p and Y q are shifing operaors which shif f by p and q pixels in he horizonal and verical direcions, respecively. The parameers P defines he size of he exponenial kernel and he scalar a 2Š0; 1Š is is base which conrols he speed of decay. Minimizing he cos funcion in (16) has shown o give good resuls in UP-SR [9], [10]; however, unless all he parameers are perfecly chosen, which is a challenge in iself, he final resul can end up being a denoised and deblurred version of f, which is also over-smoohed. This issue has been addressed by ieraive regularizaion in he case of denoising [57], [58], [59], [60], and in he more general case of deblurring [61]. In he same spiri, we use an ieraive regularizaion where we propose o focus on he choice of he regularizaion parameer. Specifically, our deblurring mehod consiss in running he minimizaion (16) muliple imes where he regularizaion srengh is progressively reduced in a dyadic way. We define, hus, a muli-level ieraive deblurring wih a BTV regularizaion such ha he soluion a level l is ^fðlþ ¼ argmin f ðlþ kbf ðlþ f ðl 1Þ k 1 þ 2 l GðfðlÞ Þ ; wih f ð0þ ¼ ^z : (18) Combined wih a seepes descen numerical solver, he proposed soluion is described by he following pseudocode: for l ¼ 1;...;L for k ¼ 1;...;K ¼ ^fðl;kþ P q¼p ^f ðl;k 1Þ n b B T sign B^f ðl;k 1Þ q¼0 ajpjþjqj ði Y q X p Þ sign end for z ^f ðl;kþ end for ^f ðl;k 1Þ z X p¼p þ 2 l p¼ P X p Y q^fðl;k 1Þ The parameer b is an empirically chosen scalar which represens he sep size in he direcion of he gradien, I is he ideniy marix, and signðþ is he sign funcion. The parameer L is he number of levels considered, and K is he number of ieraions for one level. We noe ha he correc formulaion of he problem a he beginning of Secion 4 is o use he final deblurred deph value f i 1 obained as a soluion of (18) insead of z i 1. 5 EXPERIMENTAL RESULTS We evaluae he performance of recup-sr agains sae-ofhe-ar mehods and evaluae he impac of each sep in he o algorihm. Finally, we give addiional examples illusraing he feaures of recup-sr on real daa from simple scenes wih one moving objec o more complex cluered scenes conaining muliple moving objecs wih non-rigid deformaions. Deph videos of dynamic scenes wih non-rigid deformaions were capured wih a ToF camera, PMD camboard nano [43] or he 3D MLI [65]. 5.1 Comparison wih Sae-of-Ar Mehods In order o compare our algorihm wih sae-of-ar mehods, we use a scene wih a highly non-rigidly moving objec. We use he publicly available Samba [62] daa. This daa corresponds o a real sequence of a dancing lady scene in full 3D. This sequence conains boh non-rigid radial moions and self-occlusions, represened by arms and leg movemens, respecively. We use he publicly available oolbox V-REP [63] o creae from he Samba daa a synheic deph sequence wih fully known ground ruh. We choose o fix a deph camera a a disance of 2 meers from he 3D scene. Is resoluion is 1;024 2 pixels. The camera is used o capure he deph sequence. Then we downsample he obained deph sequence wih r ¼ 4 and furher degrade i wih addiive Gaussian noise wih sandard deviaion s varying from 0 o 50 mm. The creaed LR noisy deph sequence is hen super-resolved using sae-of-ar mehods: he convenional bicubic inerpolaion, UP-SR [8], SISR [31], and he proposed recup-sr. Table 1 repors he 3D reconsrucion error of each mehod a differen noise levels. Then, we compare he accuracy of he reconsruced 3D super-resolved scene wih sae-of-ar resuls. The comparison is done by back-projecing he reconsruced HR deph images o he 3D world using he camera marix and calculaing he 3D Roo Mean Squared Error (RMSE) of each back-projeced 3D poin cloud wih respec o he ground ruh 3D poin cloud. The comparison is done a wo levels: (i) Differen pars of he reconsruced 3D body, namely, arm, orso, and leg, and (ii) full reconsruced 3D body. As expeced, by applying he convenional bicubic inerpolaion mehod direcly on deph images, a large error is obained. This error is mainly due o he flying pixels around objec boundaries. Thus, we run anoher round of experimens using a modified bicubic inerpolaion, where we remove all flying pixels by defining a fixed hreshold. Ye, he 3D reconsrucion error is sill relaively high across all noise levels, see Table 1. This is due o he fac ha bicubic inerpolaion does no profi from he emporal informaion provided by he sequence. We observe in Table 1 ha he proposed mehod provides, mos of he ime, beer resuls as compared o sae-of-ar algorihms. In order o visually evaluae he performance of he proposed recup-

8 2052 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER 2017 Fig. 5. Comparison wih muli-modal fusion mehods. (a) Raw LR deph image. (b) HR 2D image. (c) PWAS [25], (d) UML [26], (e) atgv [28], (f) Proposed recup-sr and corresponding full video is available hrough his link. Color bar in mm. mehod of umpsampling using anisoropic Toal Generalized Variaion (atgv) which is based on a global opimizaion. The resuls for one frame are given in Fig. 5. One can see ha PWAS and UML suffer from exure copying from he 2D image while atgv gives large errors on boundaries. The resul of recup-sr is cleaner wih smooh clear edges and no exure copied from 2D. I is imporan o noe ha recup-sr falls under he caegory of muli-frame SR (Secion 2). As such, he HR 2D images were no used; insead a sequence of 30 frames was used o obain he repored resul. The laeral flow was compued from ampliude images direcly capured by he 3D MLI camera. Fig. 4. 3D Ploing of one super-resolved deph frame wih r ¼ 4 using: (b) bicubic inerpolaion, (c) UP-SR [8], (d) SISR [31], (e) proposed recup-sr wih L ¼ 3;K ¼ 7;¼ 2:5. (a) LR noisy deph frame. (f) 3D ground ruh. Color bar in mm. SR algorihm, we plo he super-resolved resuls for one frame of he dancing girl sequence in 3D. We show he resuls for he sequence corruped wih s ¼ 30 mm. We noe ha recup-sr ouperforms sae-of-ar mehods by keeping he fine deails (e.g., he deails of he face). Noe ha he UP-SR algorihm fails in he presence of radial movemens and self-occlusions, see black boxes in Fig. 4c. In conras, he SISR algorihm can handle hese cases, bu canno keep he fine deails due o is pach-based naure, see Fig. 4d. In addiion, a heavy raining phase is required for SISR. We nex compare recup-sr wih muli-modal fusion approaches. To ha end, we use he same moving chairs daa presened in [8] and in [10]. The used seup o capure his daa was an LR ToF camera, he 3D MLI of resoluion (56 61) [65], mouned in he ceiling a a heigh of 2.5 m, and coupled wih an HR 2D camera, he Dragonfly2 CCD camera of resoluion ( ) from Poin Grey. Boh cameras looking a he scene of wo persons siing on chairs sliding in wo differen direcions. In muli-modal fusion approaches, he HR 2D image is a guidance image used o enhance he resoluion of he LR deph image. We consider hree represenaive algorihms; he Pixel Weighed Average Sraegy (PWAS) filer [25], and is improved version called Unified Muli-Laeral (UML) filer [26] which are wo fusion filers based on he concep of bilaeral filering, and he 5.2 Evaluaion of Differen Seps We evaluae he performance of he recup-sr algorihm a differen levels. Firs, we show how i is efficien in filering boh he deph value as well as he radial displacemen and hence he corresponding velociy. Then we evaluae he range flow esimaion, and finally, we show he imporance of deblurring Filering of Deph and Radial Displacemen We sar wih a simple and fully conrolled scene conaining one 3D objec moving radially wih respec o he camera. The considered objec is a synheic hand. A sequence of 20 deph frames is capured a a radial disance of 5 cm beween each wo successive frames, and wih D ¼ 0:1 s. The generaed sequence is downsampled wih a scale facor of r ¼ 2, andr ¼ 4, and furher degraded wih addiive Gaussian noise wih a sandard deviaion s varying from 10 o 80 mm. We hen super-resolve he obained LR noisy deph sequences by applying he proposed algorihm wih a scale facor of r ¼ 1, r ¼ 2, andr ¼ 4. Obainedresuls show ha by increasing he scale facor r, a higher 3D error is inroduced as seen in Fig. 6. In he simple case where r ¼ 1, he SR problem is merely a denoising one, and hence here is no blur due o upsampling. In conras, by increasing he SR facor r, more blurring effecs occur leading o a higher 3D error. Furhermore, in order o evaluae he qualiy of he filered deph daa and he filered velociy, we randomly choose one pixel p i from each superresolved sequence wih r ¼ 1, r ¼ 2, andr ¼ 4, andafixed noise level for s=50 mm. For each one of hese pixels, we rack he corresponding enhanced deph value f i and he

9 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2053 Fig. 9. Resuls of SR on he hand deforming sequence of Secion 5.2.2: (a) atgv-sf (flow+sr) [29] (b) recup-sr (flow) + atgv-sf (SR) (c) atgv-sf (flow) + recup-sr (SR), (d) proposed recup-sr (flow+sr). Fig. 6. 3D RMSE in mm of he super-resolved hand sequence in Secion using recup-sr. Fig. 10. Resuls of SR on he hand deforming sequence of Secion 5.2.2: (a) Raw LR, (b) atgv-sf (flow)+recup-sr (SR), (c) proposed recup- SR (flow+sr). Fig. 7. Tracking resuls for differen deph values randomly chosen from he super-resolved hand sequence in Secion wih differen SR scale facors r ¼ 1;r¼ 2; and r ¼ 4, are ploed in (a), (b), and (c), respecively. The corresponding filered velociies are shown in (d), (e), and (f), respecively. Fig. 11. Effecs of applying differen seps separaely and combined on a sequence of 35 LR noisy deph frames wih s ¼ 10 mm. The combinaion of he Kalman filer wih he spaial muli-level deblurring provides he bes performance in reducing he 3D RMSE. Fig. 8. Esimaed range flow on he hand deforming sequence of Secion 5.2.2: (a) atgv-sf [29], (b) proposed recup-sr. Arrows represen he laeral moion. The color represens he radial moion in mm. Full video available hrough his link. corresponding enhanced velociy Dfi D hrough he superresolved sequence. In Figs. 7a, 7b, and 7c, we can see how he deph values are filered (blue lines) as compared o he noisy deph measuremens (red lines) for all scale facors. Similar behaviour is observed for he corresponding filered velociies Esimaed Range Flow One of he main conribuions of his paper is he esimaion of range flow by poin racking wih a Kalman filer. We visually evaluae he accuracy of his flow on a known sequence of a hand non-rigidly deforming and capured wih he PMD CamBoard Nano camera. We qualiaively compare his resul wih he sae-of-ar range flow algorihm atgv- SF [29]. The resul for one frame is given in Fig. 8. One can see ha he proposed approach provides a smooher and more homogeneous flow, which is more accurae. Moreover, his flow is compued a a frame rae of 19 frames per second on a CPU and wihou parallelizaion considering r ¼ 2, which is faser han atgv-sf s repored runime of 1 frame per second using an opimized code. We have used he publically available atgv-sf Malab code which ook around 200 seconds for one frame. We have kep he same parameers used wih he

10 2054 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER 2017 Fig D ploing of (saring from lef column): 1) LR noisy deph frames, 2) super-resolved deph frames wih r ¼ 4 using Kalman filer, 3) superresolved deph frame wih r ¼ 4 using he proposed mehod wih one-level deblurring sep wih L ¼ 1;K ¼ 25 4) super-resolved deph frame wih r ¼ 4 using he proposed mehod wih he proposed muli-level deblurring sep wih L ¼ 5;K ¼ 25, 5) error map of comparing he obained resuls in forh column wih he 3D ground ruh. PMD CamBoard Nano camera as in [29]. The atgv-sf flow was also used o super-resolve a deph scene conaining a non-rigidly deforming objec. The proposed SR consiss in median filering all regisered frames using he esimaed flow afer back-projecing hem o he 3D world. We give he resul of atgv-sf (flow+sr) as well as our resul recup-sr (flow+sr) in Figs. 9a and 9d, respecively. This resul shows ha he proposed algorihm ouperforms atgv-sf a he level of he flow and also a he SR level. Fig. 9b and 9c give he resul of using he flow of recup-sr (flow) in he SR of atgv- SF (SR), and vice versa. This shows ha while he flow esimaed hrough recup-sr is good and ouperforms sae-of-ar mehods, i is no sufficien o direcly use i in a muli-frame SR algorihm. Using he atgv-sf (flow) in he proposed SR algorihm, however, provides a more accepable resul alhough sill inferior o he proposed recup-sr (flow+sr). This is confirmed by he corresponding 3D rendering given in Fig. 10, and can be explained by he fac ha he proposed recup-sr algorihm is a simulaneous filering of he flow and also he deph values. This ensures an effecive reconsrucion of non-rigid deph scenes Deblurring In order o beer undersand he conribuion of deblurring, we consider he Facecap daa [64] which is a simple scene of a real 3D face sequence wih non-rigid deformaions. We use a similar seup o he one used wih he Samba daase by fixing a camera a a disance

11 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2055 Fig. 14. Radial deph displacemen filering. (a) 2D opical flow calculaed from he normalized LR ampliude images. (b) Raw noisy deph radial deph displacemen. (c) Filered radial deph displacemen using Kalman filer. (d) Filered radial deph displacemen using he proposed mehod. Color bar in mm. Fig. 13. Resuls of applying he proposed algorihm on a real sequence capured by a LR ToF camera ( pixels) of a non-rigidly moving face. Firs and second rows conain a 3D ploing of seleced LR capured frames. Third and fourh rows conain he 3D ploing of he super-resolved deph frames wih r ¼ 4. Disance unis on he coloured bar are in mm. Full video available hps://dropi.uni.lu/inviaions? share=b6ed393cddde b&dl=0 of 0.7 m from he 3D face. We creae a new synheic deph sequence of he moving face. Then, we downsample he obained deph sequence wih r ¼ 4 and furher degrade i wih addiive Gaussian noise wih sandard deviaion s varying from 0 o 20 mm. The obained LR noisy deph sequence is hen super-resolved wih r ¼ 4 using: 1) Kalman filer, 2) spaial deblurring, and 3) he proposed recup-sr algorihm. In he deblurring process, wo differen echniques are considered, one-level deblurring and he proposed muli-level deblurring. The accuracy of he reconsruced 3D face sequences is measured by calculaing he 3D RMSE. In Fig. 11, we repor he obained resuls for he superresolved LR noisy deph sequence wih s ¼ 10 mm. We see how he Kalman filer aenuaes he noise gradually and hence decreasing he 3D RMSE for an increased number of frames (black solid line). We noice ha, in he presence of a non-smooh moions, he consan velociy filering model needs few number of ieraions (frames) before converging which affecs he reconsrucion qualiy of he super-resolved deph frame. For example, due o he up and down nonsmooh and fas moions of he eye brows beween frame number 20 o 25, he per-pixel emporal filering is no converged ye, and hence he 3D error increases for a few number of frames before decreasing again Fig. 11 (Black solid line). By considering he deblurring sep alone wihou engaging in he per-pixel emporal filering process, we can see ha he 3D RMSE is almos consan hroughou he sequence as shown in Fig. 11 (solid blue and green lines). This can be explained by he fac ha here is no engagemen of emporal informaion. Insead only a spaial filering is applied a each frame independenly of each oher. Finally, by looking a he obained resuls in Fig. 11, we find ha he bes performance is achieved by combining he spaial and he emporal filers (blue and green dashed lines), wih an advanage of using he proposed muli-level deblurring approach over he one-level convenional deblurring approach. Noe ha an inensive search is applied o find he bes deblurring parameers which lead o he smalles 3D RMSE error. In Fig. 12, we show he physical effecs of he previously discussed cases by ploing he corresponding 3D superresolved resuls of he las HR deph frame in he sequence. Saring from he firs column, we show he LR noisy faces for differen noise levels. The filered resuls using a perpixel Kalman filering are shown in he second column where we see how he noise has been aenuaed. The resuls of he proposed algorihm using he one-level deblurring sep, wih L ¼ 1 and K ¼ 25, and he muli-level deblurring sep, wih L ¼ 5 and K ¼ 5, are ploed in he hird and fourh columns, respecively. By visually comparing he obained resuls, we find ha he proposed algorihm wih he muli-level deblurring process provides he bes resuls and hence confirms he quaniaive evaluaion of Fig. 11 (blue dashed line) where i provides he lowes 3D RMSE. 5.3 Addiional Examples We run he recup-sr on differen LR real deph sequences capured wih he PMD CamBoard Nano of resoluion ( ) [43]. Firs, we sar wih a simple scene wih one non-rigidly moving face. Then, we show he robusness of recup-sr o opology changes by esing i on a more complex and cluered scene conaining muliple moving objecs. The algorihm s runime on all hese sequences for an SR scale facor of r ¼ 1 is 38 frames per second, r ¼ 2 is 20 frames per second, and r ¼ 3 is 9 frames per second. This is using a 2.2 GHz i7 processor wih 4 Gigabye RAM and an unopimized code.

12 2056 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER 2017 Fig. 15. Resuls of applying he proposed algorihm on a real sequence capured by a LR ToF camera ( pixels) of a non-rigidly moving face. Firs and second rows conain a 3D ploing of seleced LR capured frames and he 3D ploing of he super-resolved deph frames wih r ¼ 6, respecively. Third row shows he racking life for each pixel hrough he sequence. Unis of he coloured bar represens he racking life (ieraions) One Non-Rigidly Moving Objec We es he proposed algorihm on a real LR deph sequence of a non-rigidly deforming face wih large moions and local non-rigid deformaions. We super-resolve his sequence using he proposed algorihm wih an SR scale facor of r ¼ 4. Obained resuls are given in 3D in Fig. 13. They visually show he effeciveness of he proposed algorihm in reducing noise, and furher increasing he resoluion of he reconsruced 3D face under large non-rigid deformaions. Full video of resuls is available hrough his link. To visually appreciae hese resuls as compared o sae-of-ar mehods, we esed he bicubic, UP-SR, and SISR mehods on he same LR real deph sequence. Obained resuls show he superioriy of he recup-sr as compared o oher mehods, see Fig. 1. We show in Fig. 14b how he raw radial deph displacemen is noisy and ranges from 50 o 10 mm while in fac he real displacemen of he face in his frame has o be smooh and homogeneous. By applying he proposed algorihm, he noisy displacemen is refined o mach he real homogeneous displacemen of an approximae value of 20 mm, see Fig. 14d. We run anoher experimen on a second real sequence composed of 120 deph frames of a face moving wih long hair causing srong self-occlusions. The goal of his experimen is o show how he racking process is reiniialized in he self-occlusion case for all pixels represening he selfoccluded area. We super-resolve he acquired sequence wih a scale facor of r ¼ 6. Obained resuls are shown in Fig. 15. I is ineresing o see in he hird row how he racking life for each pixel is evolving hrough he ime wih sronger occlusions causing shorer racking, hence illusraing he impac of he hreshoulding parameer on he performance of he proposed algorihm. For example, all pixels wih he dark red colors in Fig. 15d have been appeared hrough he full sequence and no self-occlusion happened and hence he rack coninues. In conras, for mos of he boundary pixels he racking process has been reiniialized (blue dark) and hus a spaial median filer is applied for hese pixels Cluered Scene Finally, we esed recup-sr on a cluered scene of moving hands ransferring a ball from one hand o anoher. This scene is quie complex where i conains muliple objecs moving wih non-rigid deformaions, and self-occlusions wih one hand passing over he second one. Moreover, he scene conains a challenging case of opology changes represened by hands ouching each oher and hen separaing. We noe ha a srong emporal filering leads o a longer ime for convergence in he case of self-occlusions or nonsmooh moions. Similarly, a srong spaial filering leads o undesired over-smoohing effecs and hence removing he fine deails from he final reconsruced HR deph sequence. Thus, in order o handle such a scene, a rade-off beween

13 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2057 Fig. 16. Resuls of applying he proposed algorihm on a real sequence capured by a LR ToF camera ( ) of a cluered scene. Firs row conains a 3D ploing of seleced LR capured frames. Second row conains a 3D ploing of he corresponding super-resolved deph frames wih r ¼ 3. Full video available hrough his link. he emporal and spaial filering has o be achieved. Obained resuls in Fig. 16 show he robusness of he proposed algorihm in handling his kind of scenes. Full video of resuls is available hrough his link. 6 DISCUSSION AND CONCLUSIONS We have proposed a new algorihm o enhance he qualiy of low resoluion noisy deph videos acquired wih coseffecive deph sensors. This algorihm improves upon he UP-SR algorihm [8], [10] which was able o handle nonrigid deformaions bu limied o laeral moions. The newly proposed algorihm, recup-sr, is designed o handle nonrigid deformaions in 3D hanks o a per-pixel filering ha direcly accouns for radial displacemens in addiion o laeral ones. This algorihm is formulaed in a dynamic recursive way ha allowed a compuaionally efficien real-ime implemenaion on CPU. Moreover, as compared o saeof-he-ar mehods, he processing on deph maps while approximaing local 3D moions has allowed o mainain a good robusness agains opological changes and independence of he number of moving objecs in he scene. This propery is a clear advanage over mos recen mehods ha explicily compue a flow in 3D and apply a processing on meshed poin clouds [11], [12]. In order o keep smoohness properies wihou losing deails, each filered deph frame is furher refined using a muli-level ieraive bilaeral oal variaion regularizaion afer filering and before proceeding o he nex frame in he sequence. This pos-processing is shown experimenally o give he bes final resuls in erms of 3D error wihou having access o full informaion abou he scene and he sensor. Suppored by he experimenal resuls on boh synheic and real daa, we believe ha recup-sr opens new possibiliies for compuer vision applicaions using cos-effecive deph sensors in dynamic scenarios wih non-rigid moions. Furher developmens in he case of srong self-occlusions are sill required. As shown, he proposed recup-sr algorihm needs a number of deph measuremens before converging, which is no suiable for some applicaions. Moreover, a realisic deph sensor noise model is more complex han he Gaussian model considered in his work. An exended work would be o adap he proposed soluion o a more elaborae noise model on deph measuremens. ACKNOWLEDGMENTS This work was suppored by he Naional Research Fund, Luxembourg, under C11/BM/ /FAVE/Oersen. The Facecap daa were provided couresy of he research group Graphics, Vision & Video of he Max-Planck-Insiue for Informaics. The auhors would like o hank David Fersl e al. for making he codes for [28] and [29] publically available, and Konsaninos Papadopoulos and Michel Anunes for heir help wih he experimens. They also hank he Associae Edior and he anonymous reviewers for heir suggesions and commens. REFERENCES [1] [Online]. Available: hps:// kinecforwindows/, Accessed on: Nov. 04, [2] S. Berrei, A. Del Bimbo, and P. Pala, Superfaces: A superresoluion model for 3D faces, in Proc. 5h Workshop Non-Rigid Shape Anal. Deformable Image Alignmen, 2012, pp [3] D. Aouada, K. Al Ismaeil, K. K. Idris, and B. Oersen, Surface UP-SR for an improved face recogniion using low resoluion deph cameras, in Proc. 11h IEEE In. Conf. Adv. Video Signal- Based Surveillance, 2014, pp [4] S. Berrei, P. Pala, and A. Del Bimbo, Face recogniion by superresolved 3D models from consumer deph cameras, IEEE Trans. Inf. Forensics Secur., vol. 9, no. 9, pp , Sep [5] S. Schuon, C. Theobal, J. Davis, and S. Thrun, LidarBoos: Deph superresoluion for ToF 3D shape scanning, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2009, pp [6] Y. Cui, S. Schuon, S. Thrun, D. Sricker, and C. Theobal, Algorihms for 3D shape scanning wih a deph camera, IEEE Trans. Paern Anal. Mach. Inell., vol. 35, no. 5, pp , May [7] S. Izadi, e al., KinecFusion: Real-ime 3D reconsrucion and ineracion using a moving deph camera, in Proc. ACM Symp. User Inerface Sofw. Technol., 2011, pp [8] K. Al Ismaeil, D. Aouada, B. Mirbach, and B. Oersen, Dynamic super resoluion of deph sequences wih non-rigid moions, in Proc. 20h IEEE In. Conf. Image Process., 2013, pp

14 2058 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 39, NO. 10, OCTOBER 2017 [9] D. Aouada, K. Al Ismaeil, and B. Oersen, Pach-based saisical performance analysis of upsampling for precise super-resoluion, in Proc. 10h In. Conf. Compu. Vis. Theory Appl., 2015, pp [10] K. Al Ismaeil, D. Aouada, B. Mirbach, and B. Oersen, Enhancemen of dynamic deph scenes by upsamplig for precise super-resoluion (UP-SR), Compu. Vis. Image Undersanding, vol. 147, pp , [11] H. Afzal, K. Al Ismaeil, D. Aouada, F. Deselle, B. Mirbach, and B. Oersen, KinecDeform: Enhanced 3D reconsrucion of nonrigidly deforming objecs, in Proc. 3DV Workshop Dynamic Shape Meas. Anal., 2014, vol. 2, pp [12] R. Newcombe, D. Fox, and S. Seiz, DynamicFusion: Reconsrucion and racking of non-rigid scenes in real-ime, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2015, pp [13] M. Elad and A. Feuer, Super-resoluion resoraion of an image sequence: Adapive filering approach, IEEE Trans. Image Process., vol. 8, no. 3, pp , Mar [14] B. C. Newland, A. D. Gray, and D. Gibbins, Modified Kalman filering for image super-resoluion: Experimenal convergence resuls, in Proc. 9h In. Conf. Signal Image Process., 2007, pp [15] S. Farsiu, M. Elad, and P. Milanfar, Video-o-video dynamic super-resoluion for grayscale and color sequences, EURASIP J. Appl. Signal Process., vol. 2006, pp , [16] J. Tian and K.-K. Ma, A new sae-space approach for super-resoluion image sequence reconsrucion, in Proc. IEEE In. Conf. Image Process., 2005, vol. 1, pp [17] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Fas and robus muliframe super resoluion, IEEE Trans. Image Process., vol. 13, no. 10, pp , Oc [18] K. Al Ismaeil, D. Aouada, T. Solignac, B. Mirbach, and B. Oersen, Real-ime non-rigid muli-frame deph video super-resoluion, in Proc. IEEE Compu. Vis. Paern Recogni. Workshop, 2015, pp [19] J. Diebel and S. Thrun, An applicaion of Markov random fields o range sensing, in Proc. Conf. Neural Inf. Process. Sys., 2005, pp [20] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyendaele, Join bilaeral upsampling, ACM Trans. Graph., vol. 26, 2007, Ar. no. 96. [21] Q. Yang, R. Yang, J. Davis, and D. Niser, Spaial-deph super resoluion for range images, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2007, pp [22] F. Garcia, D. Aouada, B. Mirbach, T. Solignac, and B. Oersen, A new muli-laeral filer for real-ime deph enhancemen, in Proc. 8h IEEE In. Conf. Adv. Video Signal-Based Surveillance, 2011, pp [23] J. Zhu, L. Wang, R. Yang, J. E. Davis, and Z. Pan, Reliabiliy fusion of ime-of-fligh deph and sereo geomery for high qualiy deph maps, IEEE Trans. Paern Anal. Mach. Inell., vol. 33, no. 7, pp , Jul [24] R. Or-El, G. Rosman, A. Wezler, R. Kimmel, and A. M. Brucksein, RGBD-fusion: Real-ime high precision deph recovery, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2015, pp [25] F. Garcia, B. Mirbach, B. Oersen, F. Grandidier, and A. Cuesa, Pixel weighed average sraegy for deph sensor daa fusion, in Proc. IEEE Conf. Image Process., 2010, pp [26] F. Garcia, D. Aouada, B. Mirbach, T. Solignac, and B. Oersen, Unified muli-laeral filer for real-ime deph map enhancemen, Image Vis. Compu., vol. 41, pp , [27] C. Tomasi and R. Manduchi, Bilaeral filering for gray and color images, in Proc. IEEE In. Conf. Compu. Vis., 1998, pp [28] D. Fersl, C. Reinbacher, R. Ranfl, M. Ruher, and H. Bischof, Image guided deph upsampling using anisoropic oal generalized variaion, in Proc. IEEE In. Conf. Compu. Vis., 2013, pp [29] D. Fersl, C. Reinbacher, G. Riegler, M. R uher, and H. Bischof, atgv-sf: Dense variaional scene flow hrough projecive warping and higher order regularizaion, in Proc. 2nd In. Conf. 3D Vis., 2014, vol. 1, pp [30] B. Ham, M. Cho, and J. Ponce, Robus image filering using join saic and dynamic guidance, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2015, pp [31] O. Mac Aodha, N. Campbell, A. Nair, and G. Brosow, Pach based synhesis for single deph image super-resoluion, in Proc. 12h Eur. Conf. Compu. Vis., 2012, pp [32] M. Hornacek, C. Rhemann, M. Gelauz, and C. Roher, Deph super resoluion by rigid body self-similariy in 3D, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2013, pp [33] D. Glasner, S. Bagon, and M. Irani, Super-resoluion from a single image, in Proc. IEEE 12h In. Conf. Compu. Vis., 2009, pp [34] C. Barnes, E. Shechman, A. Finkelsein, and D. Goldman, PachMach: A randomized correspondence algorihm for srucural image ediing, ACM Trans. Graph., vol. 28, pp. 24:1 24:11, [35] J. Li, Z. Lu, G. Zeng, R. Gan, and H. Zha, Similariy-aware pachwork assembly for deph image super-resoluion, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2014, pp [36] Y. Li, T. Xue, L. Sun, and J. Liu, Join example-based deph map super-resoluion, in Proc. IEEE In. Conf. Mulimedia Expo, 2012, pp [37] J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, Scanning 3D full human bodies using Kinecs, IEEE Trans. Vis. Compu. Graph., vol. 18, no. 4, pp , Apr [38] H. Li, E. Vouga, A. Gudym, L. Luo, J. T. Barron, and G. Gusev, 3D self-porrais, ACM Trans. Graph., vol. 32, pp. 187:1 187:9, [39] M. Dou, J. Taylor, H. Fuchs, A. Fizgibbon, and S. Izadi, 3D scanning deformable objecs wih a single RGBD sensor, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., 2015, pp [40] H. Li, J. Yu, Y. Ye, and C. Bregler, Realime facial animaion wih on-he-fly correcives, ACM Trans. Graph., vol. 32, pp. 42:1 42:10, [41] M. Zollhofer, e al., Real-ime non-rigid reconsrucion using an RGB-D camera, ACM Trans. Graph., vol. 33, 2014, Ar. no [42] [Online]. Available: hp:// Accessed on: [43] PMD Technologies. Siegen, Germany. Camboard Nano. [Online]. Available: hp:// Accessed on: Nov. 04, [44] K. Al Ismaeil, D. Aouada, B. Mirbach, and B. Oersen, Deph super-resoluion by enhanced shif and add, in Proc. 15h In. Conf. Compu. Anal. Images Paerns, 2013, pp [45] M. Yamamoo, P. Boulanger, J. A. Beraldin, and M. Rioux, Direc esimaion of range flow on deformable shape from a video rae range camera, IEEE Trans. Paern Anal. Mach. Inell., vol. 15, no. 1, pp , Jan [46] H. Spies, B. Jahne, and J. Barron, Regularised range flow, in Proc. Eur. Conf. Compu. Vis., 2000, pp [47] H. Spies, B. Jahne, and J. L. Barron, Range flow esimaion, Compu. Vis. Image Undersanding, vol. 85, pp , [48] H. Spies and J. L. Barron, Evaluaing he range flow moion consrain, in Proc. 16h In. Conf. Paern Recogni., 2002, vol. 3, pp [49] J.-M. Gofried, J. Fehr, and C. S. Garbe, Compuing range flow from muli-modal kinec daa, in Proc. 7h In. Symp. Advances Visual Compu., 2011, pp [50] A. Wedel, T. Brox, T. Vaudrey, C. Rabe, U. Franke, and D. Cremers, Sereoscopic scene flow compuaion for 3D moion undersanding, In. J. Compu. Vis., vol. 95, pp , [51] E. Herbs, X. Ren, and D. Fox, RGB-D flow: Dense 3-D moion esimaion using color and deph, in Proc. IEEE In. Conf. Robo. Auom., 2013, pp [52] S. Hadfield and R. Bowden, Kinecing he dos: Paricle based scene flow from deph sensors, in Proc. IEEE In. Conf. Compu. Vis., 2011, pp [53] S. Hadfield and R. Bowden, Scene paricles: Unregularized paricle-based scene flow esimaion, IEEE Trans. Paern Anal. Mach. Inell., vol. 36, no. 3, pp , Mar [54] M. Surmer, J. Penne, and J. Hornegger, Sandardizaion of inensiy-values acquired by ime-of-fligh-cameras, in Proc. IEEE Workshop Compu. Vis. Paern Recogni., 2008, pp [55] K. Al Ismaeil, D. Aouada, B. Mirbach, and B. Oersen, Bilaeral filer evaluaion based on exponenial kernels, in Proc. 20h IEEE In. Conf. Paern Recogni., 2012, pp [56] M. Elad, On he origin of he bilaeral filer and ways o improve i, IEEE Trans. Image Process., vol. 11, no. 10, pp , Oc [57] W. Li, C. Zhao, Q. Liu, Q. Shi, and S. Xu, A parameer-adapive ieraive regularizaion model for image denoising, EURASIP J. Advances Signal Process., vol. 2012, 2012, Ar. no [58] M. R. Chares, M. Elad, and P. Milanfar, A general ieraive regularizaion framework for image denoising, in Proc. 40h Annu. Conf. Inf. Sci. Sys., 2006, pp

15 AL ISMAEIL ET AL.: REAL-TIME ENHANCEMENT OF DYNAMIC DEPTH VIDEOS WITH NON-RIGID DEFORMATIONS 2059 [59] S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, An ieraive regularizaion mehod for oal variaion-based image resoraion, Muli-Scale Model Simulaion, vol. 4, pp , [60] P. Milanfar, A our of modern image filering: New insighs and mehods, boh pracical and heoreical, IEEE Signal Process. Mag., vol. 30, no. 1, pp , Jan [61] A. Kheradmand and P. Milanfar, A general framework for regularized, similariy-based image resoraion, IEEE Trans. Image Process., vol. 23, no. 12, pp , Dec [62] [Online]. Available: hp://people.csail.mi.edu/drdaniel/ mesh_animaion/, Accessed on: Nov. 04, [63] [Online]. Available: hp:// Accessed on: Nov. 04, [64] L. Valgaers, C. Wu, A. Bruhn, H. P. Seidel, and C. Theobal, Lighweigh binocular facial performance capure under unconrolled lighing, ACM Trans. Graph., vol. 31, pp. 187:1 187:11, [65] [Online]. Available: hp:// Accessed on: Nov. 04, Kassem Al Ismaeil received he BSc degree in elecronic engineering and he MSc degree in compuer science boh from he Universiy of Aleppo, Aleppo, Syria, in 2006 and 2008, respecively. He received he Erasmus Mundus MSc degree in compuer vision and roboics from he Universiy of Burgundy, Le Creuso, France, in 2011, and he PhD degree in compuer science from Inerdisciplinary Cenre for Securiy, Reliabiliy, and Trus (SnT), Universiy of Luxembourg, Luxembourg, in He received he Bes Paper Award a he IEEE 2nd CVPR Workshop on Muli-Sensor Fusion and Dynamic Scene Undersanding (MSF 15). His research ineress include 3D compuer vision, image and video processing, wih a focus on deph super-resoluion, 3D reconsrucion, and srucure from moion. He is a suden member of he IEEE. Djamila Aouada (S 05-M 09) received he sae engineering degree in elecronics from Ecole Naionale Polyechnique (ENP), Algiers, Algeria, in 2005, and he PhD degree in elecrical engineering from Norh Carolina Sae Universiy (NCSU), Raleigh, Norh Carolina, in She is a research scienis wih Inerdisciplinary Cenre for Securiy, Reliabiliy, and Trus (SnT), Universiy of Luxembourg. She has been leading he compuer vision aciviies wih SnT since She has worked as a consulan for muliple renowned laboraories (Los Alamos Naional Laboraory, Alcael Lucen Bell Labs., and Misubishi Elecric Research Labs.). Her research ineress span he areas of signal and image processing, compuer vision, paern recogniion and daa modelling. She is he co-auhor of wo IEEE Bes Paper Awards. She is a member of he IEEE. Thomas Solignac received he engineering degree in elecommunicaions from he Insiu Superieur d Elecronique du Nord, Lille, France, in 1995, and he DEA degree in elecronics from he Insiu d Elecronique e de Micro-elecronique du Nord, Valenciennes, France, in He worked as developmen engineer and projec leader in indusry auomaion wih main focus on fully auomaed opical inspecion sysems for he wood indusry. Since 2005, he has been a senior compuer vision engineer wih IEE S.A., Luxembourg working in he research and developen of innovaive 2D- and 3D-camera based machine vision applicaions wih main focus on real-ime objec deecion and racking. The applicaion fields are auomoive safey sysems, advanced driver assisan sysems, as well as advanced building securiy and auomaion. Bruno Mirbach received he PhD degree in heoreical physics from he Universiy of Kaiserslauern, Kaiserslauern, Germany, in He was a posdocoral researcher wih Cener for Nonlinear and Complex Sysems, Como, Ialy, Max-Planck- Insiue of he Physics of Complex Sysems, Dresden, Germany, and he Universiy of Ulm, Germany. His research ineress include focussing on non-linear dynamics and quanum chaos. Afer ha he joined auomoive indusry, working on he research and developmen of inelligen opical sysems for safey applicaions. He is currenly senior algorihm group leader wih IEE S.A., Luxembourg, responsible for he developmen of compuer vision algorihms for advanced driver assisan sysems, as well as for sysems in advanced building securiy and auomaion. His curren research ineress span he areas of 3D-vision, 2D/3D sensor fusion, image processing and machine learning, wih he main focus on real-ime objec deecion and racking. Bj orn Oersen (S 87-M 89-SM 99-F 04) received he MS degree in elecrical engineering and applied physics from Link oping Universiy, Link oping, Sweden, in In 1989, he received he PhD degree in elecrical engineering from Sanford Universiy, Sanford, California. He has held research posiions in he Deparmen of Elecrical Engineering, Link oping Universiy, Informaion Sysems Laboraory, Sanford Universiy, Kaholieke Universiei Leuven, Leuven, and Universiy of Luxembourg. During 96/97, he was direcor of research wih ArrayComm Inc, a sar-up, San Jose, California, based on Oersen s paened echnology. He has co-auhored journal papers ha received he IEEE Signal Processing Sociey Bes Paper Award in 1993, 2001, 2006, and 2013 and hree IEEE conference papers receiving Bes Paper Awards. In 1991 he was appoined a professor of signal processing wih he Royal Insiue of Technology (KTH), Sockholm. From 1992 o 2004 he was head of he deparmen for signals, sensors, and sysems wih KTH and from 2004 o 2008 he was dean of he School of Elecrical Engineering, KTH. Currenly, he is a direcor for he Inerdisciplinary Cenre for Securiy, Reliabiliy and Trus, Universiy of Luxembourg. His research ineress include securiy and rus, reliable wireless communicaions, and saisical signal processing. He is a fellow of he IEEE. " For more informaion on his or any oher compuing opic, please visi our Digial Library a