Cognitive Algorithms for 3D Video Acquisition and Transmission

Transcription

1 Department of Information Engineering, University of Padova Cognitive Algorithms for 3D Video Acquisition and Transmission Simone Milani Politecnico di Milano / University of Padova milani@elet.polimi.it, simone.milani@dei.unipd.it Giancarlo Calvagno University of Padova calvagno@dei.unipd.it Streaming Day 2011 Torino, Italy - September 30, 2011

2 Outline Current and future video transmission devices Enabling 3D video communications Acquisition of dynamic 3D scenes Denoising of depth information Optimizing the coding and transmission of 3D data Cognitive Source Coding A low complexity CSC architecture Experimental results and conclusions 2

3 Current architectures for video transmission Digital video transmission nowadays involves several heterogeneous devices and networks. Video coding and transmission New problems and challenges require smart video coding and transmission systems that adapt to different realities. 3D video communications enhances all these and brings forward new ones. 3

4 Needs of modern video systems Limited power requirements to enhance mobility and autonomy Robustness to losses, errors, delays, congestions Capability of interoperating Possibility to adapt to video services and applications Ability to adapt to the transmitted signal (video, depth) and grant different levels of Quality-of-Service (QoS) Capability of understanding environment and situation Possibility to integrate information from other sensors (positioning, movement, etc...) 4

5 Characteristics of 3D Video 3D video signals = texture + geometry information In Depth Image Based Rendering, 3D video signal is made of Texture stream (=strandard video signal) Depth stream Depth Texture Depth is important since it permits synthesizing different views and adapt visualization to different devices. Stereoscopy with active glasses VR caskets Standard screens with navigation Autostereoscopic displays System designers must take into account the difference among different signals and different devices. 5

6 Emerging needs of 3D applications Previous requirements are even more urgent for 3D video communications. Several needs are added: high quality, high-resolution depth maps; enabling accurate synthesis of different views; handling, compressing, and transmitting different types of signals; synchronizing different streams; adapting the information to the displaying devices. Practical implementation of 3D video communication systems must take care of these requirements. 6

7 Practical implementations at UniPD LTTM lab setting 2 Scout cameras + ToF MESA sensor In collaboration with LTTM lab of prof. G. Cortelazzo Left cam ToF cam Right cam MS Xbox Kinect sensor Netbook Asus EeePC 1201-T 7

8 Denoising of depth sensors Depth sensors permits implementing real-time 3D video communication systems and permits avoiding the computational complexity of stereo depth estimation. oreover, they works well in presence of smooth surfaces with limited texture. The biggest problem with depth sensors is noise. EXTERNAL NOISE Scheme for MS Xbox Kinect sensor 8

9 Denoising System for MS Kinect The denoising system: first corrects the depth values discarding wrong measurements; then interpolates the data. The approach was tested on different light environments. 9

10 Experimental results (1/3) Controlled light. The system was tested on stereo acquisition system where Kinect is coupled with a standard camera (test set available on-line). Din: D : Dout: Dout : input depth output correction output correction + interpolation output interpolation, no correction The obtained depth information is used to generate the warped view. PSNR measures the accuracy of warping. (Average over 10 acquisition) 10

11 Experimental results (2/3) Uncontrolled light, outdoor. Side view. Original Denoised Depth map 11

12 Experimental results (3/3) Uncontrolled light. Multiple reflections and winows. 12

13 Cross-layer Solutions and Cognitive Source Coding Cross-layer solution Several cross-layer solutions have been proposed to solve the problems of modern video communication systems However, CL solutions usually require complex optimization strategies to be effective; CL schemes most of the times simply tune the transmission parameters. Cognitive Source Schemes are CL solutions where the very architecture of the source coder changes implementing different coding strategies. 13

14 Cognitive Source Coding Cognitive Coding (CSC) system Cognitive Source Radio (CR) system... coding... is is an an intelligent intelligent source wireless communication system and transmission system that that is is aware aware of of scene its surrounding environment environment (i.e., (i.e., outside outside world), world), and and uses uses the the methodology methodology of of understanding-by-building understanding-by-building to to learn from outside world and adapt its internal learnthe from the environment and adapt its internal states. states. [Haykin 05] CR Different modulation schemes Sensing the channel and adapting CSC Different source coding schemes Estimating channel state and signal characteristics, and changing the coding scheme Advantages: Flexibility, Reconfigurability 14

15 The adopted scheme for 3D video transmission The signal is partitioned into different subsignals by a segmentation unit followed by a classification unit The adopted CSC scheme combines Single Description Coder (i.e. H.264/AVC) Multiple Description Coder Distributed Source Coder FEC cross-packet Coder LEGO-like video source coder 15

16 Classification At he beginning, the signal is splitted into subsignals that include Front elements, Moving objects The input depth signal is oversegmented into superpixels Background, Static objects Other classes... Superpixels are classified according to depth and texture information into segments belonging tomoving and static objects Subsignals Front Back Texture Vf Vb Depth Df Db 16

17 Basic options At the beginning the input subsequences can be split into two video streams (odd and even lines) whenever MD option is enabled. Then, signals are predicted and residuals can be coded using traditional Displaced Frame Difference (DFD) or Distributed Video Coding (DVC). Additional FEC packets can be added processing the coded video packet stream (RFC 2733). 17

18 Preliminary experimental results Horse test sequence (video + depth) Performance of different configurations for the test sequence Horse. _._. SD + DFD + FEC MD +DFD SD + DVC + FEC MD + DVC 18

19 Improving results and complexity 4 configurations (C) SD + DFD + FEC SD + DVC + FEC MD + DFD MD + DVC Performance can be improved by classifying subsignals feature via a Support Vector Machine (SVM) partitioning. avg. vertical gradient (Sobel) avg. temporal gradient Different kernels were tested: gaussian polynomial logarithmic 19

20 Computational complexity setting Computational complexity can be reduced by changing Motion search window Macroblock partitioning modes according to the characteristics of the subsignal Small search window Wider search window All partitioning modes No block partitioning FME tuning Test all the coding modes Avoid some coding modes Different complexities tailored to different objects 20

21 Experimental results (1/3) Ballet Sequence 21

22 Experimental results (2/3) Ballet sequence (frame 10) H FEC CSC with SVM Ballet sequence (frame 4) 22

23 Experimental results (3/3) H FEC CSC with SVM 23

24 A CSC low-complexity encoder an MS Xbox Kinect sensor a netbook Asus EeePC 1201-T Modes: SD or MD DFD or DVC FEC protection Cognitive Source Coding Features include (at least so far): GOP IPPP CABAC entropy coding Intra coding Fixed transform block size (4x4) Luma coding only 24

25 Conclusions and future work 3D video signals require effective strategies to enhance the quality of the acquired signal and transmit it; adapting the source coding scheme to the channel characteristics and recorded objects can significantly improve the quality of the reconstructed sequence; an effective denoising system has been designed for MS Kinect sensor; a CSC scheme has been defined based on an SVM classification of the elements in the sequence; a low complexity architecture has been designed and implemented based on MS Kinect. Ongoing work optimization of the coded bit rate for the different subsignals; optimization of denoising by distinguishing the sources of errors; enabling cognitive denoising by estimating the type of illumination in the environment. 25

26 Thank you for the attention!! Thank you for your attention More results and documentation can be found at Simone Milani, University of Padova EUSIPCO 2011, Barcelona, Spain 26