Real-Time 2D to 2D+Depth Video Conversion. OzViz David McKinnon QUT

Transcription

1 Real-Time 2D to 2D+Depth Video Conversion OzViz David McKinnon QUT

2 Why? The goal is to by able to convert video sequences where the camera undergoes motion but the scene is stationary, fast. This is one piece of the puzzle for the conversion of old video footage to 3D to be compatible with 3D TV s, also useful for robotics and VFX. Would like to convert this type of algorithm into a plugin for an Autodesk match mover or Nuke.

3 System Overview Matching moving to recover the camera path There a number of GPU accelerated stages within this stage, such as feature extraction and matching Stereo Local Stereo (Inaccurate, Fast, Per-Pixel Reasoning, WTA Unreliable, GPU Implementations Easy) Global Stereo (Accurate, Slow, Memory Intensive, Whole Image Reasoning, Reliable, GPU Implementations Hard) Consistency Checking OpenGL is designed to accelerate 3D graphics. This can be exploited for consistency between depths maps.

4 Matchmoving/Camera Tracking Tracks some points Work out where the photos were taken

5 Cross-Based Local Method Real-Time Stereo Matching Cross-Based Local Approach, Jiangbo Lu et. al. Calculate Arm Lengths Metric Aggregation : Vertical + Horizontal WTA Depth Selection

6 Cross-Based Local Method Trade-off between window size and reliability of the window metric on areas with less texture. Bilateral filtering is the top performing window metric, but is very slow! Other methods over-segment the image into super-pixels and work on regions. The cross-based approach straddles the two concepts but it is more efficient.

7 Cross-Based Local Method Multi-view plane sweeping method are used to compile a photo-consistency cost volume.

8 Cross-Based Local Method Metric Aggregation : Vertical + Horizontal Passes 1. // turn on blending (set to accumulate uniformly) 2. glenable(gl_blend); 3. glblendfunc(gl_one, GL_ONE); 4. // render to the FBO 5. for(int z = 0; z < cost->depth(); z++) { 6. // attach and draw 7. cost->attach_fbo(z); 8. gldrawbuffer(gl_color_attachment0_ext); 9. if(vwgl_fbo::is_fbo_ready()) { 10. // setup the viewport and matrices 11. cost->height()); cost->fit_viewport(0, 0, cost->width(), 12. >height()); cost->fit_matrices(0, 0, cost->width(), cost- 13. // draw the quad 14. cost->draw_quad(z); 15. } else 16. vcl_cout << "FBO not ready" << vcl_endl; 17. }

9 Cross-Based Local Method The Results

10 Temporal Consistency For stereo through time, the results should temporally consistent. Cut down on frame-flicker. Solution : smooth each frame s depth-image through time using robust means. Real-Time Visibility-Based Fusion of Depth Maps, Paul Merrell et. al.

11 Temporal Consistency Example of re-projection of depth fields Original Reprojection Comparison

12 Temporal Consistency Vertex Code char *vertex_code = "varying vec4 vpos; varying float conf; uniform float depth_min; uniform float depth_max; uniform float width; uniform float height; uniform float spacing; uniform float ds; uniform mat3 ih; uniform vec3 C; uniform sampler2drect tex; void main(void){ vec3 pix = vec3(gl_color.r*spacing*width*ds, gl_color.g*spacing*height*ds, 1.0); vec2 samp = texture2drect(tex, pix.st/(spacing*ds)).ra; vpos = gl_modelviewmatrix*vec4((samp.r*(depth_max-depth_min)+depth_min)*ih*pix+c, 1.0); conf = samp.g; if(abs(conf)!= 1.0) { gl_position = gl_projectionmatrix*vpos; gl_pointsize = 1.0; } else { gl_pointsize = 0.0; gl_position = vec4(0.0, 0.0, 0.0, 1.0); } }\0"; Fragment Code char *fragment_code = "uniform float qdepth_min; uniform float qdepth_max; varying vec4 vpos; varying float conf; void main(void){ float gray = (vpos.z/vpos.w - qdepth_min)/(qdepth_max - qdepth_min); gl_fragcolor.r = gray; gl_fragcolor.a = conf; }\0"; OpenGL Code glenable(gl_vertex_program_point_size); gldrawbuffer(gl_color_attachment0_ext); glenableclientstate(gl_vertex_array); glenableclientstate(gl_color_array); glbindbufferarb(gl_array_buffer_arb, cbo_); glcolorpointer(3, GL_FLOAT, 0, 0); glbindbufferarb(gl_array_buffer_arb, vbo_); glvertexpointer(3, GL_FLOAT, 0, 0); gldrawarrays(gl_points, 0, width*height*spacing*spacing); gldisableclientstate(gl_vertex_array); gldisableclientstate(gl_color_array);

13 Putting it together Stereo module can run at ~60Hz Consistency checking at > 100Hz Bottleneck is the ego-motion/matchmoving determination running at ~10Hz There are many avenues for parallelisation yet to fully exploited Timings approximate values on a GTX 480

14 That s all Thankyou any questions?