Forward interpolation on the GPU

Transcription

1 Forward interpolation on the GPU GPU Computing Course 2010 Erik Ringaby

2 Questions Why is inverse interpolation not used here? Name at least one reason why forward interpolation is difficult to implement efficiently with CUDA? Why may my implementation run slower on <G80, <HD2000 models?

3 Agenda Background Main algorithm Forward vs. Inverse interpolation CUDA-implementation idea My implementation Coordinate calculation Results Conclusion

4 Computer Vision Laboratory Background Working on a project for rectifying rolling shutter images Original Corrected iphone 3GS

5 Global shutter With a global shutter, all lines have their integration time simultaneously A global reset is done just before readout Frame 1 readout Frame 2 integration Line 1 Line 2 Line 3 Line 4 Line n Readout Integration time

6 Rolling shutter All rows are exposed at different time, but the frame is stored as a single still image The readout resets the pixel content Frame 1 readout Frame 2 readout Line 1 Line 2 Line 3 Line 4 Line n Readout Integration time

7 Virtual global shutter

8 Computer Vision Laboratory Rectified RS-frame Original Corrected

9 Current implementation Most of the code written in Matlab KLT from OpenCV Forward interpolation in MEX Takes about 1-4s per frame

10 Main algorithm overview Find point correspondences between frames Estimate camera motion (rotation and/or translation) Transform rows/coordinates to one common coordinate system Interpolate image with new coordinates

11 Camera motion estimation Unfortunate the most time-consuming part is hard to parallelise Camera motion is estimated by minimising non-linear least squares Input is inter-frame point correspondences spread over whole frame (global) which should all be in the same cost function

12 Camera motion estimation Using Matlabs lsqnonlin as a black box and receives the motion parameters Big matrix with residuals Happy for parallelisation suggestions

13 Forward vs. Inverse interpolation When to use forward interpolation? Sometimes inverse mapping not available Here only an approximate inverse mapping exists

14 Forward vs. Inverse interpolation Neighbouring pixels within a row in the rectified image do not necessarily have the same transformation parameters Reconstruction comparison later

15 Forward interpolation At a first glance potentially easy to implement on a GPU due to a parallel nature Previous implemented via a MEX-function which is fed with the forward mapping

16 For each pixel in the input image, paint a blob in the output image The closest 3x3 grid are updated with:, 2 2 w( u) = exp( 0.5 u x ) / σ wb where CPU-implementation wr wg w u is grid location and x is sub-pixel location of pixel When all pixels have been written, divide with w to get RGB-image

17 CUDA implementation idea Local parallel problem, the input pixels does not need information from other pixels Even though CUDA supports scattered writes, gather operations tend to be much faster Since the output pixels don t know which input pixels that are going to contribute, it s difficult to write it as a gather operation

18 CUDA implementation idea Can be parallelised per row since we know how pixels are written within a row Different rows can however overlap and we have a race condition Atomic functions not supported on my GPU (G80) Even if it would support it, performance would decrease High-level function to determine problems

19 My implementation Geometry based What the GPU originally was designed for Move vertices with GLSL

20 First approach Set vertices at each rows end-points Send the forward mapping to the GPU and move the vertices to the correct position

21 Theoretical mistake Modification Not constant distance between pixels in the same row Need denser mesh for (row = 0; row <= height/rowres;row++){ for (col = 0; col <= width/colres; col++){ vertices[2*(nwidth+1)*row+2*col] = col*colres; vertices[2*(nwidth+1)*row+2*col+1] = row*rowres; } } texcoords[2*(nwidth+1)*row+2*col] = col*colres/width; texcoords[2*(nwidth+1)*row+2*col+1] = row*rowres/height; if ((col < nwidth) && (row < height/rowres)){ indices[4*nwidth*row+4*col] = (row * (nwidth+1)) + col; indices[4*nwidth*row+4*col+1] = (row * (nwidth+1)) + col + 1; indices[4*nwidth*row+4*col+2] = ((row+1) * (nwidth+1)) + col + 1; indices[4*nwidth*row+4*col+3] = ((row+1) * (nwidth+1)) + col; }

22 Final approach GPUs want to calculate data Instead of calculate coordinates on the CPU and sending them to the GPU, Calculate them directly on the GPU

23 How to calculate coordinates In the current main algorithm, rotations are assumed to be smooth across a frame From point correspondences the rotation for the bottom row is estimated Rotation is presented as a three element vector, n, with magnitude as rotation angle and direction as rotation axis 2 R = expm(n) = I + [ˆ] n x sinφ + [ˆ] n x(1 cosφ) r32 r23 n = logm( R) = φnˆ ~ / ~, ~, nˆ = n n n = r13 r31, φ = sin r21 r 12 1 ( n ~ / 2)

24 How to calculate coordinates R = expm(n) n = logm( R) I + [ˆ] n = φnˆ, sinφ + [ˆ] n 2 = x x (1 cosφ) r32 r23 ˆ ~ / ~, ~ n = n n n = r13 r31, φ = r21 r 12 sin 1 ( n ~ / 2) SLERP (Spherical Linear interpolation) Matrix exponential R n interp diff = expm( n = logm(expm( n 1 )expm( wn 1 diff ) )expm( n 2 w [ 0, 1] ))

25 How to calculate coordinates Each row gets its own rotation R through SLERP (w = currentrow / totalrows ) Transform all of them to a reference row with rotation R 0 (e.g. first or middle row) T 1 Forward mapping: x' = KR 0 R( x2) K x K intrinsic camera matrix Only need to send K and three values for last row rotation to the shader

26 GLSL GLSL is a perfect language for this Built-in data types like vec3, vec4 and mat3 Built-in functions length(), transpose(), cross(), matrix and vector multiplication

27 Forward vs. Inverse interpolation T Forward: x' = KR R( x Uses only vertex shader 0 2) K 1 x T 1 1 Inverse: x = K( R0 R( x2)) K x' Uses only fragment shader May be much slower on older cards (<G80, <HD2000) due to dedicated vertex/fragment processors instead of unified

28 Results Extreme case with ground-truth rotations to show differences Rolling shutter Forward Inverse No holes within mask

29 Results

30 Results Forward Mex ms Calculate coords in Matlab ~410ms Griddata usually ~19000ms (up to 1min) Inverse interpolation Matlab ~ ms Inverse interpolation GPU ~2ms Forward interpolation GPU ~15ms Just display image mesh ~15ms

31 Results The geometry is limiting the performance Downsample the grid Every 2 nd pixel per row 9-10ms Every 4 th pixel per row 5ms Every 8 th pixel per every 2 nd row 1-2ms, as fast as inverse interpolation

32 Results

33 GLSL still useful! Conclusion Forward interpolation on the GPU possible and fast (have been some discussion about this)

34 Questions?

35 Thank you!