Omni Stereo Vision of Cooperative Mobile Robots

Transcription

1 Omni Stereo Vision of Cooperative Mobile Robots Zhigang Zhu*, Jizhong Xiao** *Department of Computer Science **Department of Electrical Engineering The City College of the City University of New York (CUNY) 138th Street and Convent Ave New York City, NY Acknowledgements Supported by NSF Environmental Monitoring DARPA/ITO Mobile Autonomous Robot S/W China NSF Scene Modeling Collaborators at UMass, Amherst Edward Riseman, UMass Allen Hanson, UMass Deepak Karuppiah, UMass Roderic Grupen, UMass

2 Outlines Omni Vision Fundamentals Depth Error Characterization of Omnidirectional Stereo Vision Virtual Stereo Vision of Cooperative Mobile Robots: A Example 3 Generalized Omni Vision Omnidirectional / panoramic vision Omnidirectional: 360 degrees FOV Panoramic: large FOV, Camera: rotation or translation Omni Vision: How to look Viewer-centered: outward looking Modeling scenes Object-centered: inward looking Modeling objects rather than scenes Many viewpoints over a large space plane Earth 4

3 Applications of Omni Vision Robotic Applications Indoor / Outdoor surveillance Mobile robot navigation Formation control Human detection and tracking Ground / Aerial robots 5 Important Issues of Omnistereo What is this talk about Omnistereo imaging principle Epipolar geometry for correspondence Depth error characterization in both direction and distance Calibration methods A example of virtual stereo vision system using two mobile robots Other important issues not in this talk Sensor designs Correspondence algorithms 3D Match Algorithm 6

4 Omni Vision Fundamentals 7 Omni Imaging & Representation Omnidirectional (panoramic) Imaging Catadioptric Camera (single effective viewpoint) PAL, Netvision360 by RemoteReality, and many Image Mosaicing Rotating camera, translating camera, arbitrary motion Omnidirectional Representation Cylindrical Representation Panoramic Annular Lens (PAL) By Pal Greguss Netvision-a by RemoteReality 8

5 Cylindrical Panoramic Unwarping Two Steps: (1) Center determination () Distortion rectification -order polynomial approximation Circular to cylindrical transformation after eliminating radial distortion 9 Panoramic Mosaics from a Rotating Camera 0 o 360o 10

6 1st frame Cylindrical Panorama connecting frame conic mosaic head-tail stitching panorama 11 Cylindrical Projection Image projection (φ, v) of a 3D point P (X,Y,Z) ( φ, v) = (tan 1 D = X + Y ( Y, X ), F Z O Z D ) Y v φ X Vertical axis D Distance Cylindrical image P (X, Y, Z) 1

7 Depth Error Characterization of Omnidirectional Stereo Vision 13 Omnidirectional stereo vision Omni Stereo Vision Get depth information from D image At least viewpoints Omni Stereo Vision: How many viewpoints Binocular/N-Ocular: a few ( or more) fixed Circular Projection: many inside a small area Dynamic Omnistereo: a few, but re-configurable 14

8 Binocular / N-Ocular Omnistereo A few fixed viewpoints Three configurations Horizontally-aligned binocular (H-Bi) omnistereo Vertically-aligned binocular (V-Bi) omnistereo N-ocular omnistereo trinocular case Issues Distance error in the direction of 360 degrees Distance error versus distance Epipolar geometry 15 H-Bi Omnistereo: Depth Error From Image pair { (φ 1, v 1 ), (φ, v ) } to a 3D point P (X,Y,Z) Triangulation sinφ sinφ D = B = B sin( φ φ ) sinφ -Fixed baseline B - Horizontal disparity (vergent angle) φ = φ φ1 Depth Error 1 D D φ B sinφ -Depth accuracy is non-isotropic; P (X, Y, Z) D φ Z Y v v 1 φ 1 φ O 1 X O B -Best distance estimation is achieved when φ =90 (max vergent angle) -Depth error proportional to Depth / Baseline -Singularity case: triangulation is invalid 16

9 H-Bi Omnistereo: Singularity Case Zero Vergent angle when φ 1 =φ =0 or 180 degree Distance Ratio Method v v D = B = B v v1 v Vertical disparity: v P (X,Y,Z) v 1 v O 1 epipoles O B D -Visible Epipoles: the images of the camera centers in the others could be visible! -In practice, mutual occluding region exists - Vertical disparity and vertical epipolar lines 17 H-Bi Omnistereo: Epipolar Geometry Given point (φ, v ) in the nd image, its corresponding point (φ 1, v 1 ) in the 1 st image lies in a sine curve v 1 singularity triangulation v 1 sinφ = 1 v sinφ sinφ depth-blind spots φ 1 -The epipolar curves are sine curves in the non-singularity cases and - The epipolar lines are along the v direction in the singularity cases 18

10 V-Bi Omnistereo From Image pair { (φ 1, v 1 ), (φ, v ) } to a 3D point P (X,Y,Z) Bv D = F v v 1 B = F v v Z Y O 1 X v 1 - Vertical baseline Bv B v P - Vertical disparity v - Same as perspective stereo O v - Depth accuracy isotropic in all directions - Depth error proportional to square of distance -Epipolar lines are simply vertical lines -No mutual occlusions -Cannot be used for stereo viewing by human eyes (horizontally aligned) 19 N-Ocular Omnistereo Why more viewpoints? Every point of the 360 FOV from the center of the sensor-triangle can be covered by at least two pairs of rays from different cameras with good triangulations R 1 R 3 R 13 O 3 O 1 O R 13 R 3 R 1 -Less mutual occlusion problem by using redundant cameras -one pair of stereo match can be verified using the second pair - depth accuracy is still not isotropic, but is more uniform in directions - However no gain in epipolar geometry 0

11 Circular Projection Omnistereo Many viewpoints on a viewing circle Omnivergent Stereo (Shum et al ICCV99) every point in the scene is imaged from two cameras that are vergent on that point with maximum vergence angle; and stereo recovery yields isotropic depth resolution in all directions. Solution: Circular Projection 1 Circular Projection: Principle Many viewpoints on a viewing circle Z Case 1: an omni sensor O viewing circle Case : two 1D sensors A virtual camera moving in a viewing circle captures two set of rays on a plane tangent to the viewing circle: the left-eye in clockwise direction, and the right-eye in counterclockwise direction

12 Circular Projection: Geometry Max vergent angles for left and right rays O 1 r B left-eye ray φ 1 D φ P right-eye ray baseline φ D = r / sin viewing circle O O φ disparity P: 3D space point r: radius of the viewing circle φ 1,φ : viewing directions of left and right rays φ: vergent angle (angular disparity) B: baseline length (< r); D: distance (OP) 3 Circular Projection: Properties Depth estimation is isotropic D D φ Same depth error in all directions, r vergent angles are same at the same distance Make full use of the 360 viewing Depth error proportional to depth /baseline Same as H-Bi Omnistereo limited baseline (B < r) Horizontal Epipolar lines Superior than H-Bi Omnistereo when a single viewing circle for left and right omni-images 4

13 Dynamic Omni-stereo a few viewpoints moving freely Requirements: Optimal configuration for any given point in the world Change both the vergent angle and the baseline freely Issues: Dynamic Calibration View Planning Image 1 Camera 1 Target Baseline Image Camera 5 Dynamic Omni-stereo: Depth Error Question 1: Vergent angle Max vergent angle (φ = 90 o ) Question : Baseline The larger the better? = D + D D B B B sinφ φ D D B sinφ φ rotation shift The error in estimating the baseline 6

14 Dynamic Omni-stereo: Mutual Calibration Sensors as calibration targets Make use of the visible epipoles PAL PAL 1 Known target geometry Cylindrical body of the moving platform α B = Rc / sin( ) B B α R c O α B cylinder body O 1 Rc 7 Dynamic Omni-stereo: Optimal View Baseline error proportional to B Larger baseline, even larger error B B α R c Overall distance error is min if Best baseline and max vergent angle B = RcD sin 1 B φ = ( ) D (sinφ =1) Distance error with optimal configuration proportional to D 1.5 D = D 1.5 R c φ 8

15 Dynamic Omni-stereo Optimal view application Track a single target by two robots One stationary, one moving Omnistereo head with reconfigurable vergent and baseline rotation T (1) O () shift T () O (1) O 1 9 Comparisons Four Cases Fixed viewpoint omnistereo One fixed, one circular projection Both circular projection Dynamic omnistereo Java Interactive Simulations 30

16 Java Interactive Simulations Path of the moving target Errors out of range 1 st camera nd camera 1 st camera nd camera moves on a circle 1 st camera nd camera Errors are smaller in case 4 1 st camera nd camera moves on a curve 31 Comparisons Viewpoints Configuration. Epipolar Geometry Error in direction Error in Distance Binocular, fixed Sine curve Nonisoptric D /B Circular projection Many, small circle Horizontal line isoptric D /r Dynamic, free Sine curve Optimal for target D 1.5 3

17 Experiment: Two-robot Scenario A stationary monitor robot with PAL 1 camera; to monitor the movements in the environment A moving explorer robot with PAL camera; to follow a moving object of interest and/or to find a better viewpoint for constructing the virtual stereo geometry with the camera in the monitor PAL PAL 1 33 Mutual Awareness & Dynamic Calibration Each robot as a cylinder with vivid color Easily seen and extracted in the image of the other robot s camera Baseline between the two omni camera is estimated using the occluding boundary of the cylinder O α B Why cylinder? cylinder body O 1 Rc - view-invariant - easy to detect α B = Rc / sin( ) 34

18 Pano 1: Mutual Calibration and Human Searching Image of the nd robot Images of a person Pano : Image of the 1st robot Results: B = 180 cm, (baseline between two camera) D 1 = 359 cm, (distance from camera 1 to the target) D = 08 cm, (distance from camera to the target) 35 Size-Ratio method in co-linearity Object O 1 O Camera 1 Width Camera D 1 D Sizes of an object in a pair of images tell the distances 36

19 Current and Future Work Multi-robots with omnidirectional vision Optimal view planning Integrate panoramic camera with pan/tilt/zoom camera Increase the capacity (viewing angle, image resolution, etc) Vision-based robot formation control 37 3D Match Algorithm Performed on objects extracted from images Four steps: (1) Moving object detection and tracking () Head detection and localization (3) Stereo match based on 3D features (4) Improving match by temporal tracking 38

20 Object Detection &Tracking Detection and tracking multiple moving people by motion analysis and region grouping Head detection and localization Head location is one of the reliable primitives for 3D match - usually visible - easy to detect -symmetric >>Further extension: appearance-based partial match 40

21 4.3. Stereo Match based on 3D features Why not D match? (1) large perspective distortion () low image resolution Matching primitives of an object blob : (1) Intensity of blob () bearing of head ->D (3) width of blob ->W (4) point at top of head -> H 41 (1) Measures for each assumed match: - Intensity similarity: r is Are image intensities consistent? - Ray convergence: r rc Do rays thru. two images converge in 3D space? - Width consistency: r wc Are image widths consistent with assumed geometry? - Height consistency: r hc Are image heights consistent with assumed geometry? (3). Match Selection 3D match algorithm Image 1 Image object O 1 (). Overall Match Goodness r( i, j) = ris( i, j) rrc( i, j) rwc( i, j) rhc( i, j) [0,1] O object 1 Choose maximum remove object images from match hypothesis repeat 4

22 43