Studying Dynamic Scenes with Time of Flight Cameras

Transcription

1 Studying Dynamic Scenes with Time of Flight Cameras Norbert Pfeifer, Wilfried Karel Sajid Ghuffar, Camillo Ressl Research Group Photogrammetry Department for Geodesy and Geoinformation Vienna University of Technology geo.tuwien.ac.at Images: Diploma thesis Stefan Niedermayer

2 Time of Flight Cameras Outline Calibration and Scattering still and dynamic imagery Optical Flow and Range Flow Range video analysis dynamic imagery General question: does high (temporal) sampling compensate for large random noise? 2

3 ToF cameras: combining advantages... of photography and laser scanning Area wise, simultaneous, robust determination of 3D object coordinates by range measurement of an emitted signal Suited for low contrast surfaces and in darkness No homologous points necessary No moving parts Low power consumption Small, compact, mobil Cheap Bundle of vectors Simplifies indirect orientation on moving platforms Simultaneous capturing of a dynamic object space 3

4 Distance measurement d = ½ c t Common for all methods: nir Amplitude modulation Pulsed modulation Avalange photo diodes (APD) for detection APD in Geiger mode: single photon counting (SPAD) Multiple Double Short time Integration (MDSI) Continuous wave modulation (Photo mix detectors, PMD) Intensity λ mod φ A P H Amplitude and Phase (distance) measured Time 4

5 PMD: typical technical data Swissranger Image matrix 144x176px² Modulation 5-30MHz Field of view Max. framerate 39.6 x47.5 Uniqueness range 30-5m 25 fps Dimensions 50x67x42.5mm³ Illumination 1W Mass 162g Carrier WL 850nm Focal distance 8mm Precision: cm Accuracy: dm 5

6 PMD: random range errors Empirically determined σ 2 obsdist vs. average amplitude σ 2 obsdist 1/A 2 6

7 Local errors Distance itself (not linear) Position at sensor Amplitude Reflectivity / material (?) Integration time Internal / external temperature Incidence angle (?) Temporal drift Not local errors Internal: scattering External: multipath Relations unclear PMD: systematic range errors Scattering cmp. lens flare Multipath 7

8 Distance calibration approach Self calibration: model selection and parameter estimation as integrated process on one data set Amplitudes: lower relative noise in δρ comparison to range and low local deformation Lateral resolution very low (sensor matrix) expoit image space entirely circular, non-coded targets Planar test field: no multipath, little scattering Bundle block adjustment IOR & EOR EOR mask test field area and use only areas outside targets distance residuals Check residuals against posssible factors of influence: model selection, parameter estimation by LS adjustment 8

9 Calibration with range videos Low lateral resolution vs. high temporal resolution : 25kPixel vs. 25fps Image sequences Handheld camera guidance permanente, random movements, dense sampling of parameter space Automatic target tracking ~ 6000 frames Only distances < 2.5m : A 1/d2 Motion blur (espcially rotations) 9

10 Calirbation with still images Averaging of thousands of frames with EOR=const to still image suited for entire distance range Automatic target detection and orientation Laborious search for correspondence between object space and image space (no target code) 850 stills : with difference in integration time, exterior orientation changes in amplitude, object distance, position in image space,... 2 test fields with different reflectivity separate between object distance and amplitude 10

11 Model selection and parameter estimation Der. minus obs. dist. corr. 4 all but obs. dist., corr. model offset,d1,d2s,d2c,d3s,d3c,a1,it1,row1,row2,col1,col2,rowcol2 / origobs x Der. minus obs. dist. corr. 4 all but amplitude, corr. model offset,d1,d2s,d2c,d3s,d3c,a1,it1,row1,row2,col1,col2,rowcol2 / origobs x Der. minus obs. dist. [m] count Der. minus obs. dist. [m] count Observed dist. [m] Amplitude [] Der. minus obs. dist. corr. 4 all but int. time, corr. model offset,d1,d2s,d2c,d3s,d3c,a1,it1,row1,row2,col1,col2,rowcol2 / origobs der. minus obs. dist. [m] int. time [] 11

12 Internal relections Scattering Observation: mixture of focussed and scattered light Emphasized with high image contrast active illumination Modell: addition in the complex plane Assumption: sinusoidal signal May produce strange distances (phase wraps) Investigation of scattering As few assumptions as possible Without prior calibration Influencing factors Integration time? Amplitude? Distance? Position at sensor? 12

13 Method a_0016: mean amp. diff [%]; itime: 30 Background subtraction Frames with / without foreground object Vary parameters to be investigated Background Black cardboard Foreground Circular white target Mounted on vertical staff Mask area of staff and half shadow (illumination not an ideal point source), investigation of remaining image area 2 types of differences: Separated difference of amplitude & range errors in the observation Difference of the complex signal error in the signal High noise level Static scene and fixed EOR Pixel wise averaging over up to 60 hours of continuous data acquisition

14 Variation of scattering with the position on the sensor Complex subtraction Not shift invariant Radially symmetric to principal point Range error 14

15 Findings of the experiment Scattering linear in the amplitude double target size ~ double amplitude difference Scattering additive in complex signal, independent of object space distance... Internal effect Varies with position on sensor Point spread function (using an unresolved target) 63% focussed light; 8 neighboring pixels : 1,4 ; 72% within disk (r=19px); side lobes (maxima) : 0,07 15

16 PSF: Application gg = ff h + ηη Observed image is a convolution of the real image with the PSF + noise Inversion: Richardson-Lucy: ff nn+1 = ff nn gg ff nn h htt Reduce RMSE distance spatially variant PSF: 73% spatially invariant PSF: 69% Amplitude and Range difference images with/without object in foreground before and after deconvolution 16

17 Analyzing dynamic scene content Considerations Video sequences: image of object points move in data stream vs. wide baseline: points jump Range and brightness are functions parameterized over image space r(x,y), i(x,y) Assumption: functions are continuous and differentiable Brightness of points remains constant Wrong assumption because of active illumination (1/r2, cos(incidencea)) Therefore Track range and brightness changes of points in image space for Computing camera trajectory (exterior orientation) Follow moving objects throughout the scene 17

18 Optical flow and range flow Comparable least squares matching / ICP but approx. values = 0, no iterations, no re-assignment of correspondence Apply to entire image area Range and brightness should be considered together (complementary?) Intensity Image Intensity Derivatives Depth derivatives Depth Image Use framework of optical flow and range flow 18

19 Optical and range flow I(x) u t 1 t 2 Z(x) t 1 t 2 II xx xx tt = x (U,W) X(x) II xx uu + II yy vv + II tt = 0 ZZ xx UU + ZZ yy VV WW + ZZ tt = 0 Equation for one pixel Two/three unknowns (optical/range flow) Image/surface gradients required, otherwise coefficients are zero Application in window local method, parameter estimation by least squares adjustment Applied to all pixels, e.g. independently 19

20 Fusion of Range and Intensity for object tracking Intensity Image Intensity Derivatives Depth derivatives Depth Image Optical Flow Constraint Equation Range Flow Constraint Equation II xx u + II yy v + II tt = 0 ZZ xx U+ ZZ yy V W + ZZ tt = 0 ff II xx ZZ ff ZZ xx ZZ ff II yy ZZ ff ZZ yy ZZ xx II xx ZZ II yy yy ZZ ZZ xx xx ZZ ZZ yy yy ZZ 1 UU VV WW = II tt ZZ tt 20

21 Long Term Trajectories Independently moving objects Long range trajectory generation Spatio-temporal segmentation into independently moving object 21

22 Video Note noise level in depth and intensity image range errors at dark trousers of second person: calibration not applied missing texture around feet of first person: neither in range nor in brightness hardly background texture movement within floor/wall plane pair depends on small sign at the wall: spatial regularization to determine full flow at all pixel 22

23 Note Illumination fall off Systematic range errors Full 3D estimation of flow, esp. lower train Successful suppresion of erroneous background motion Upper train shows consistent flow despite grey level differences and range errors 23

24 Optical and Range Flow for Camera Relative Orientation Motion fields generated by camera motion Estimating 6 parameters of relative orientation using dense intensity and range data 24

25 Video Add robust parameter estimation (reweighting in LS adjustment) to eliminate moving objects 25

26 Weights in LS adjustment

27 Conclusions and findings Calibration is possible and shows stability Systematic range errors in cm-dm order Stills provided better results in calibration than videos averaging of stills reduced random error and thus eased model selection motion blur as additional source of signal (error) in dynamic scenarios Internal scattering can become larger (dm-order) Internal scattering compensation possible but too time consuming hardware solution to reduce internal scattering of light (better absorption) Combined optical and range flow as natural combination of both channels can consider stochastic signal properties, as it is embedded in LSA High noise level propagates to flow vectors due to local analysis e.g. fluctuation of flow vectors, boundaries of segments Random noise can only be effectively removed with very rigid models ToF cameras rather trigger method than application development ( Quantitative evaluation, details on methods, etc.: in the publications ) 27

28 41

29 42

30 Ende 43