Mech 296: Vision for Robotic Applications. Today s Summary

Transcription

1 Mech 296: Vision for Robotic Applications Gravity Probe B Lecture 3: Visual Sensing 3.1 Today s Summary 1. Visual Signal: Position in Time Acquire Process Obtain Camera State 2. Examples Star Tracker Jellyfish tracking 3.2 1

2 Color Video x Color y Time 3.3 Visual Positioning Signal Vision Sensor Video Hardware Acquire Process Obtain Camera State Position Signal 3.4 2

3 Video Acquisition Goal: Visual sensing requires manipulation of video with a digital computer Video Acquisition Depends on camera output type Analog Video framegrabber card required Ex: Plug NTSC signal into Matrox Meteor II board Framegrabber digitizes analog signal and outputs the digital frame into a memory buffer Digital Video direct interface Video hardware attached to computer via digital input port Digital frames loaded directly into memory (USB, Firewire) or through dedicated framegrabber (Camera Link) OR Digital frames loaded from a video file on hard disk 3.5 Analog Video Transmission Analog Video is uncompressed Common television transmission standards include NTSC (North America) and PALS (Europe) NTSC format Frames arrive at a rate of Hz Full images are constructed at Hz using interlaced lines from even and odd frames A full image consists of 525 lines Data is transmitted on approximately 480 lines Bandwidth results in an effective horizontal resolution of approximately 640 lines Color resolution is approximately half grayscale resolution and uses YIQ (also known as LUV) color space Analog video transmission is susceptible to RF interference 3.6 3

4 Interlaced Analog Video 3.7 Digital Video Compression Digital Video is compressed based on time differencing A digital video may consist of N frames The compressed video is smaller than N compressed images Compression Method: Use time differencing Keyframe: reference image Stored periodically Compressed in same manner as conventional images Intermediate frames Differenced from most recent keyframe Difference image generally contains little new information content Compression rate for intermediate frames is very high Several video standards exist for digital video File formats: MPEG, AVI, Quicktime Compression method: Microsoft Video, Cinepak, Indeo 3.8 4

5 Video Acquisition in Matlab (I) Matlab offers Acquistion Toolbox imaqhwinfo: query available video devices Ex List video devices vidinfo = imaqhwinfo; %hardware info vidinfo.installedadaptors %available inputs Ex Obtain device ID number vidinfo = imaqhwinfo( winvideo ); vidinfo.deviceids %winvideo device ID videoinput: intialize video device Ex Initialize winvideo (USB) device with ID = 1 vidobject = videoinput( winvideo,1); 3.9 Video Acquisition in Matlab (II) getsnapshot: obtain single video image (slow ~ 1 Hz) Ex Get image, im1, from video object, vidobject im1 = getsnapshot(vidobject); %get frame image(im1); %display to screen trigger: obtain one or more frames (faster > 1 Hz) Ex Get image, im1, from video object, vidobject % Configure the video object so that one image % is acquired for each trigger ( FramesPerTrigger ), and so % that the maximum number of frames acquired ( TriggerRepeat ) % is not restricted set(vidobject,'framespertrigger',1); set(vidobject,'triggerrepeat',inf); % Configure the video object to acquire images based on a % software request ( Manual ) rather than an external signal triggerconfig(vidobject, 'Manual'); % Signal the video object to begin waiting for a trigger signal start(vidobject); % Trigger image acquisition trigger(vidobject); im1 = getdata(vidobject,1); %Software trigger %Extract one frame from video buffer

6 Video Acquisition in Matlab (III) Matlab offers Acquistion Toolbox flushdata: clear frames from memory. It is necessary to use this command for visual sensing, otherwise Matlab will eventually run out of memory. Ex Clear most recently acquired frame flushdata(vidobject,'trigger'); % flush buffer imaqmem % get buffer stats stop: signal device to stop accepting triggers Ex On exit, the video object should be halted and cleared stop(vidobject); % halt the object flushdata(vidojbect); % clear the video buffer delete(vidobject); % delete the object 3.11 Matlab Acquistion Sample % Initialize video object vidobject = videoinput('winvideo',1); set(vidobject,'framespertrigger',1); set(vidobject,'triggerrepeat',inf); triggerconfig(vidobject, 'Manual'); start(vidobject); % Begin acquistion loop for n = 1:10000 tic % Begin timing trigger(vidobject); % Trigger video device im1 = getdata(vidobject,1); % Acquire image image(im1); % Displaying video makes loop slow pause(.01); % Introduce pause to force display flushdata(vidobject,'trigger'); % Clear video buffer toc % Time elapsed since tic end % Close video object stop(vidobject); flushdata(vidojbect); delete(vidobject);

7 Reading Movie File into Matlab Instead of working with video in real-time, it is also possible to post-process video files aviinfo: obtain information on an AVI movie file Ex Look up frame rate for an AVI, sample.avi movieinfo = aviinfo('sample.avi'); %get stats movieinfo.framespersecond %Display frame rate aviread: read avi frames into matlab structure Ex Extract and display first ten frames from sample.avi moviearray = aviread('sample.avi',1:10); for n=1:10 image(moviearray(n).cdata); %Display frames pause(.01); end 3.13 Visual Positioning Signal Vision Sensor Video Hardware Acquire Process Obtain Camera State Position Signal

8 Segmentation Using Time Difference Segmentation is the process of dividing an image into smaller regions (segments) Lecture 1: Pixel intensity Lecture 2: Pixel color This Lecture: Pixel temporal changes Difference between successive images in a time series Background difference between an image in a time series and a reference image 3.15 Comparison: Differencing Time Difference = g(t) - g(t -1) Background Difference = g(t) - g(0) t = 0 t = 1 t = 2 t = 3 t = 4 Example MECH 296: uses green Vision images for Robotics only Threshold Dr. Jason is Rife applied to the absolute value of the image difference

9 Visual Positioning Signal Vision Sensor Video Hardware Acquire Process Obtain Camera State Position Signal 3.17 Obtaining Camera State Camera pixel data is inherently 2D Pixels locations may be interpreted as Bearings (angles relative to camera axis) or Planar positions 3D point positions may be calculated by combining information from multiple pixels Using a reference of known geometry viewed in a single image dots from parallel lasers on flat surface (for size scaling & orientation) pattern orientation (for camera angle relative to a scene) Using multiple cameras to view a scene of unknown geometry (stereo vision) 6 DoF camera position can be calculated by combining observations of multiple points in the environment over time In this class focus on 2D information

10 Pinhole Camera Model Simplest camera model: pinpoint aperture Pinhole Cameras Dark images (little light enters pinhole) Infinite depth of focus Lens Cameras Brighter images (bigger aperture) Focus depends on distance of object from camera Plane Object D Planar Positioning Points in space are mapped onto image plane For objects lying on a plane parallel to the image plane, pixel position differences are proportional to actual physical distances For some alignment tasks, it is sufficient to express relative distances in pixel units Absolute distance (in meters) can be determined by calibration, if the distance between the camera and the plane of the objects is known For objects lying in a plane at an oblique angle relative to the image plane, pixel distances are foreshortened by perspective

11 2D Planar Positioning Application Example: Overhead Vision System Δy Δx D Bearing Pixels measurements describe the angle between an observed point and the camera centerline This angular relationship always applies, even if observed points are not coplanar Zenith angle, φ, describes angle relative to camera axis Azimuth angle, θ, describes angle around camera axis z x-x 0 φ θ

12 2D Planar Positioning Application Example: Warehouse Inventory ( x x y y ) θ = atan 2, 0 0 θ φ 2 2 d x x0 + y y0 = tanφ tanφ d ( ) ( ) Pixel distance between Constant of object and origin assuming proportionality square, MECH even 296: pixels Vision for Robotics 3.23 Lens Distortion Actual lenses have finite thickness Lens patterns result in systematic errors in vision position measurements (planar position or bearing) Pincushion Distortion No Distortion Barrel Distortion

13 Lens Distortion Calibration Calibration Target Perpendicular to camera axis Located at 15 from camera Grid lines at 2 intervals A simple calibration procedure for lens distortion: Choose points in image (i.e. intersections of grid lines) and measure two pieces of data 1. Gridline position 2. Pixel position Compute radial distance of each point relative to the image center point (assuming it aligns with the lens center) Perform polynomial fit to relate gridline radial position to pixel radial position 3.25 Polynomial Fit for Radial Distortion 1. For each data point, relate grid measurement to pixel measurement through unknown polynomial weights C 1, C 2, etc. 2. Solve for unknown coefficients with least squares method 3. Evaluate residual and adjust centerpoint if necessary to minimize residual, δ, (and compensate for misalignment of imaging array relative to lens optics) Polynomial fit: r = Cr + C r + C r grid, k 1 pixel, k 2 pixel, k 3 pixel, k Least Squares: c = A\ b Residual: rpixel, corr = Ac δ = r b pixel, corr 2 3 rpixel,1 rpixel,1 r pixel,1 2 3 rpixel,2 rpixel,2 rpixel,2 A =, 2 3 rpixel, N rpixel, N rpixel, N rgrid,1 C1 r grid,2 b =, c = C 2 C 3 r grid, N Corrected (undistorted) measurement

14 Scale-Factor Calibration After radial distortion calibration complete, it is possible to calibrate pinhole geometry The remaining scale factor (assuming square pixels) is: Planar Position Case: C xy Bearing Case: C φ θ y pixel,corr x pixel,corr rpixel, corr = C φ tanφ x meters xy pixel, corr y = C x θactual = θ pixel, corr = C y 1 φactual = atan ( Cφ rpixel, corr ) meters xy pixel, corr 3.27 Scale Factor Solution r pixel,corr z φ z = z nom + Δ grid rpixel corr = C = Cφ z nom, φ tan ( φactual ) r r grid + Δ Planar position: automatically have a scale factor C xy = 1 (in grid units). Bearing angle: in order to compute scale factor, C φ, need distance of pinhole from calibration plane, z. This distance is approximately equal to the distance from the calibration plane to the front of the camera, z nom (in grid units)

15 LSQ for Bearing Scale Factor 1. Compute bearing scale-factor, C φ, assuming the camera-to-calibration plane measurement is biased by an unknown constant, Δ Polynomial fit: z r = C r Δr nom pixel, corr φ grid pixel, corr 2. Solve for unknown coefficients with least squares method 3. Confirm that residual, δ, and distance correction, Δ, are both small Least Squares: rgrid,1 rpixel, corr,1 rgrid,2 r pixel, corr,2 A =, rgrid, N r c = A\ b pixel, corr, N znomrpixel, corr,1 znomr pixel, corr,2 Cφ b =, c = Δ znomrpixel, corr, N Residual: δ = Ac b 3.29 Viewing Cone For a conventional camera, the maximum viewing angle is set by the size of the pixel array and by its distance from the focal center (which is a function of lens shape). In the figure below, this maximum viewing angle relative to the optical axis is labeled φ max φ max Often, specs refer to a camera field of view, where FOV = 2 φ max The FOV places a significant constraint on vision-based control

16 Increasing Field of View To maximize the field of view, it may be desirable to use A wide angle lens A panoramic mirror These methods increase field of view and Decrease resolution Add angular distortion But permit easier tracking of targets and features in the environmental from Visual Positioning Signal Vision Sensor Video Hardware Acquire Process Obtain Camera State Position Signal

17 Applying Vision Signal Example: Gravity Probe B Measures position relative to stars Coarse positioning: Star tracker Fine positioning: Guide star + Telescope 3.33 Optical Sensing Hardware Example: Gravity Probe B Measures position relative to stars Coarse positioning: Star tracker FOV: 1 degree Resolution: 1 arcminute Fine positioning: Telescope FOV: 1 arcminute Resolution: 1 milliarcminute

18 Star Tracker Star Tracker View Guide Star Two pixel measurement Compares intensity on two pixels Control attempts to balance intensity on both pixels

19 Applying Vision Signal Example: Jellyfish Tracking 3.37 Filter Impacts Segmentation Four cases of thresholding with different filters: Raw Morphological Filter Narrow Gaussian Wide Gaussian

20 Visual Positioning Review Vision Sensor Video Hardware Acquire Process Obtain Camera State Position Signal Analog, Digital Video Video Capture Time Difference Background Diff. Planar Position, Bearing Calibration Field of View Spatial Filtering