Challenges and solutions for real-time immersive video communication

Transcription

1 Challenges and solutions for real-time immersive video communication Part III - 15 th of April 2005 Dr. Oliver Schreer Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institut, Berlin, Germany Dr. Oliver Schreer 1

2 Structure of the Short Course Part I (13 th of April 2005) Introduction Scope of immersive video communication Quick tour into projective geometry Part II (14 th of April 2005) Quick tour into camera model and epipolar geometry Analysis of real-time video streams Part III (15 th of April 2005) The concept of real-time immersive video conferencing Quick tour into stereo image processing Hybrid-recursive disparity estimation Dr. Oliver Schreer 2

3 Outline The concept of real-time immersive videoconferencing A quick tour into stereo image processing Hybrid-recursive disparity estimation Dr. Oliver Schreer 3

4 Eye Contact Problem in Video Conferencing Dr. Oliver Schreer 4

5 Application for a Virtual Camera virtual camera set-up stereo camera B virtual camera A Dr. Oliver Schreer 5

6 The 3D Video Processing Chain Head Tracker Foreground/ Background Segmentation Tracking of Hands Rectification MPEG-4 Compositor Depth Estimation 3D View Combining MPEG-4 Encode Transmission 3D Warping of Video Objects MPEG-4 Decode Analysis Synthesis Dr. Oliver Schreer 6

7 Rectification Rectification Rectification Dr. Oliver Schreer 7

8 The 3D Video Processing Chain Head Tracker Foreground/ Background Segmentation Tracking of Hands Rectification MPEG-4 Compositor Depth Estimation 3D View Combining MPEG-4 Encode Transmission 3D Warping of Video Objects MPEG-4 Decode Analysis Synthesis Dr. Oliver Schreer 8

9 Outline The concept of real-time immersive videoconferencing A quick tour into stereo image processing Hybrid-recursive disparity estimation Dr. Oliver Schreer 9

10 General Task Correct assignement of features in both views, which are images of the same 3D point Dr. Oliver Schreer 10

11 The Correspondence Problem different field of view of two cameras low textured regions occlusions periodical patterns Dr. Oliver Schreer 11

12 Similarity Constraints in Stereo epipolar constraint M 1 T T T m2 Fm1 = 0 m1 l1 = 0, m2l2 = 0 M 2 l 1 m 1 e 1 e 2 m 22 m 21 C 1 C 2 search directions I 1 I 2 uniqueness constraint smootheness constraint Dr. Oliver Schreer 12

13 Similarity Constraints in Stereo ordering constraint I 1 I 2 limited search range M max δ=u 1, da u 2 =0 M min m max m 2 m min C 1 u 1,min u 1,max C 2 u 2 =0 Dr. Oliver Schreer 13

14 Pixel-based Block-Matching v I 1 I 2 δ(u,v) u 1 u 2 u 1 Dr. Oliver Schreer 14

15 Block-Matching with Resolution Pyramid left right Dr. Oliver Schreer 15

16 Block-Matching Fine-to-Fine left right Dr. Oliver Schreer 16

17 Two Stage Block-Matching global stage L R local stage L R Dr. Oliver Schreer 17

18 Outline The concept of real-time immersive videoconferencing A quick tour into stereo processing Hybrid-recursive disparity estimation Dr. Oliver Schreer 18

19 General Concept of Depth Estimator (DE) Original Images Disparity Maps L R Fast Depth Estimation Post- Processing L R R L 1st Step: Fast, Simple and Robust Approach hybrid block- and pixel recursive matching algorithm accept that it produces some mismatches Dr. Oliver Schreer 19

20 General Concept of Depth Estimator (DE) Original Images Disparity Maps L R Fast Depth Estimation Post- Processing L R R L 2nd Step: Detection and Masking of Mismatches efficient consistency check between L R and R L match substitute inconsistent disparities by inter/extrapolation Dr. Oliver Schreer 20

21 Hybride Recursive Disparity Estimation (HRM) Objectives Spatial and temporal consistent disparities Real-time processing on full TV resolution CCIR 601 ( 720 x 576 pel) candidates Dr. Oliver Schreer 21

22 Hybride Recursive Disparity Estimation (HRM) Objectives Spatial and temporal consistent disparities Real-time processing on full TV resolution CCIR 601 ( 720 x 576 pel) pixel recursion Dr. Oliver Schreer 22

23 Thank you for your attention in the first part! Coffee break Dr. Oliver Schreer 23

24 Structure of Hybride-Recursive Matching pixelrecursion update vector choice of best disparity start vektor disparity vector left image right image blockrecursion 3 candidates disparity memory Dr. Oliver Schreer 24

25 Block-Recursion test of 3 candidate vectors vertical temporal horizontal previous image current Image no search area around candidates choice of best candidate via DBD (displaced block difference) Dr. Oliver Schreer 25

26 Interleaved Meander Scan even frame odd frame Dr. Oliver Schreer 26

27 Structure of Hybride-Recursive Matching pixelrecursion update vector choice of best disparity start vektor disparity vector left image right image blockrecursion 3 candidates disparity memory Dr. Oliver Schreer 27

28 Pixel-Recursion Initialisation of two pixel-recursive processes Computation of update-vectors by using optical flow principle Choice of best update-vector by DPD (displaced pixel difference) initial vector d incremental update update vector d start vector Dr. Oliver Schreer 28

29 Structure of Hybride-Recursive Matching pixelrecursion update vektor choice of best disparity start vector disparity vector left image right image blockrecursion 3 candidates disparity memory Dr. Oliver Schreer 29

30 General Concept of Depth Estimator (DE) Original Images Disparity Maps L R Fast Disparity Estimation Post- Processing L R R L Dr. Oliver Schreer 30

31 Consistency Check Right-to-left vector Left-to-right vector L->R R->L occluded region Dr. Oliver Schreer 31

32 Simple Post-Processing L->R R->L L->R R->L horizontal linear interpolation Dr. Oliver Schreer 32

33 Adaptive Post-Processing Homogenous areas: linear interpolation Discontinuities: segment-based hole-filling homogenous areas L->R R->L discontinuities Dr. Oliver Schreer 33

34 Adaptive Post-Processing L->R R->L L->R R->L segment-based hole filling Dr. Oliver Schreer 34

35 Segment-based Interpolation homogenous areas discontinuities Homogenous regions Discontinuities bilinear interpolation segment-based approach Dr. Oliver Schreer 35

36 Experimental Results without consistency check & post-processing L->R simple post-processing L->R segment-based post-processing L->R R->L R->L R->L Dr. Oliver Schreer 36

37 Experimental Results without consistency check & post-processing Dr. Oliver Schreer 37

38 Experimental Results simple post-processing Dr. Oliver Schreer 38

39 Experimental Results segment-based post-processing Dr. Oliver Schreer 39

40 Experimental Results segment-based post-processing with perfect mask Dr. Oliver Schreer 40

41 The 3D Video Processing Chain Head Tracker Foreground/ Background Segmentation Tracking of Hands Rectification MPEG-4 Compositor Depth Estimation 3D View Combining MPEG-4 Encode Transmission 3D Warping of Video Objects MPEG-4 Decode Analysis Synthesis Dr. Oliver Schreer 41

42 3D View Combining Texture & Depth at Middle View Position Dr. Oliver Schreer 42

43 The 3D Video Processing Chain Head Tracker Foreground/ Background Segmentation Tracking of Hands Rectification MPEG-4 Compositor Depth Estimation 3D View Combining MPEG-4 Encode Transmission 3D Warping of Video Objects MPEG-4 Decode Analysis Synthesis Dr. Oliver Schreer 43

44 Final 3D Warp m M w m view ψ virtual view ψ m view ψ Dr. Oliver Schreer 44

45 Experimental Results of View Synthesis without post-processing simple post-processing segment-based post-processing Dr. Oliver Schreer 45

46 Results of View Synthesis Fixed view - motion Dynamic view still image Dr. Oliver Schreer 46

47 Comparison to Other Approaches Visual quality Pyramid Two-stage BM HRM Processing speed at TV-resolution ( P IV 1,7 GHz ) 40 4x4 Grid in ms Dr. Oliver Schreer 47

48 Simple Extension Towards Tele-Collaboration Application at Screen Integrated Tools In cooperation with France Telecom Dr. Oliver Schreer 48

49 Extension Towards Scalable System Architecture CAVE-like Terminal CAVE-like Terminal Basic Module Corner Terminal Multi-User Terminal Corner Terminal Scene & Conference Server Content Management Dr. Oliver Schreer 49

50 Conclusion Real-time capable algorithms available in the whole 3D video processing chain segmentation tracking disparity analysis on full TV resolution MPEG-4/H.26x encoding/decoding view synthesis Architecture concept without dedicated hardware Dr. Oliver Schreer 50

51 Acknowledgement I like to thank... Peter Kauff (head of my group) Khalid Elazouzi (segmentation & tracking of hands) Nicole Atzpadin (disparity analysis) Christoph Fehn (immersive TV) Ingo Feldmann (shadow detection) Ralf Tanger (foreground/background segmentation) Contact Oliver.Schreer@hhi.fhg.de web: ip.hhi.de Dr. Oliver Schreer 51

52 ... thank you for your attention END Dr. Oliver Schreer 52