Ellipse Centroid Targeting in 3D Using Machine Vision Calibration and Triangulation (Inspired by NIST Pixel Probe) Final Project EENG 510 December 7, 2015 Steven Borenstein 1
Background NIST Pixel Probe[1] NIST - Configurable Robotic Millimeter-Wave Antenna (CROMMA) facility, Boulder, CO Pixel Probe is a system of 3 cameras. A single pixel in each camera is calibrated to a laser tracker to establish a fixed point in space linked to the tracker. Robotic placement of the 3D point in space defined by 3 pixels enables measurement accurate to 25 micrometers. Currently a manual process to generate path plans based on image data. High frequency antennas have very small openings, around 1-2mm. 2
Project Goals Build an apparatus with three cameras arranged in a tetrahedron to collect data and test results. Calibrate Cameras Ellipse Detection 3D Triangulation of Ellipse Center Experimentation Identify Pixel Probe Capture Filter Detect Triangulate Core Program Command Camera Calibration Camera Calibration Intrinsic Parameters Camera/Machine Transformation Camera Calibration Extrinsic Parameters 3
Machine Apparatus 3 Sony DFW-VL500 Firewire Machine Vision Cameras Video Server using ispyserver, converts Firewire video to a MPEG4 IP video stream 3 Axis mini-cnc machine adapted with camera mounting plate and antenna mount. Control PC for manual control of CNC machine. Data processing PC (running Matlab) (not shown) 4
Camera Calibration [2] Determines: Focal Length Camera Center Skew Distortion Focal Length: fc = [ 1915.98997 1879.98008 ] +/- [ 54.31969 54.38232 ] Principal point: cc = [ 331.98704 281.97042 ] +/- [ 10.80129 13.36162 ] Skew: alpha_c = [ 0.00000 ] +/- [ 0.00000 ] => angle of pixel axes = 90.00000 +/- 0.00000 degrees Distortion: kc = [ -0.18718-11.34161-0.02335-0.00480 0.00000 ] +/- [ 0.09761 4.14249 0.00235 0.00315 0.00000 ] Pixel error: err = [ 0.36934 0.48881 ] Challenge: returning camera to same focal length after powering off. Solution: focusing stand to refocus camera to a known object location. Unfortunately, must use default zoom. 5
Extrinsic Transformations Generates a rotation and translation from the camera coordinate system and the machine. Uses a calibration target similar to calibration. (Aligned w/ laser line) Requires the cameras first be calibrated. Results: Translation vector: Tc_ext = [ -15.975610 16.304874 216.157072 ] Rotation vector: omc_ext = [ -2.336670-0.061269 0.084768 ] Rotation matrix: Rc_ext = [ 0.996611 0.018288-0.080200 0.070414-0.693689 0.716825-0.042524-0.720043-0.692626 ] Pixel error: err = [ 1.01156 0.36487 ] M = [ ] R(3x3) T(1x3) # 6
Ellipse Detection Previous Work A New Efficient Ellipse Detection Method, Yonghong Xie and Qiang Ji [3] Hough Transform Scoring Technique Based on Ellipse Geometry Canny Edge Detector (sigma 5) Loop iterates through all edge point pairs, to test if the pair meets min/max major axis constraints. Calc Major Axis and center. For each valid hypothesis pair, identify a 3 rd pair to test constraints for minor axis. Accumulate if a possible match. Accumulator with max value is most likely minor axis. Increment vote for this pair and center. If votes reach a threshold, an ellipse is detected. To improve efficiency, a subset of pairs is randomly chosen (though, at risk of missing the exact major axis). [4] 7
Ellipse Detection Filtering A few downsides to ellipse detection slow, responds to noise in the environment, may detect a part of an ellipse as a full ellipse. Other issues, noise in image may trigger detection of ellipse. Which ellipse do you use? 8
Solution 1 Filtering RGB or HSV Thresholding Mask Edges (~25 pixels) Open/Dilate to remove noise and fill gaps Regionprops to toss out large or small blobs. Upscale image to interpolate jagged edges of ellipse. R < 200, G < 150 SF=1.0 SF=2.0 SF=4.0 9
Solution 2 ROI by Object Detection Pick three representative candidate images. Candidate Images With the target image, sum the normalized cross correlation of each candidate image. ROI is centered on the pixel with maximum value. Ellipse detection in ROI. (No other filtering) [5] Similar to Wen Yuan Chen and Jung Lin, Object Detection Scheme Using Cross Correlation and Affine Transformation Techniques. 10
Solution 2 Handling Background Noise Left images show a pairing (imshowpair) of the sum of the cross correlations w/ original image. Right shows Original. 11
Improvement of Ellipse Detection Solution 1 Filtering Only Solution 2 Object Detection only within ROI 12
Putting it all together, Simulating the Pixel Probe First, Need need a single 3D point. Simulated with a stiff wire. 13
Triangulation p 3 P p 1 p 2 Identify the pixel center of the ellipse from each camera. Solve: p 1 x M 1 P = 0 p 2 x M 2 P = 0 p 3 x M 3 P = 0 p 1n = [x n y n 1] T = K -1 [x,y,1] T p 1x = 3x3 skew symmetric of p 1n K = intrinsic parameters from calibration M x = Transformation Matrix (3x4) 14
First Test Machine Zero d on Goal 15
Direct Machine Control Questions? Matlab program performs all commands to 3-axis platform. 16
Next Steps Application Possibilities Port to OpenCV, improve object detection for ROI (potentially with SURF ), model in 3D entire object. Possible applications: border tracing, real time path commands, real time path error control, automated calibration routines. http://robocv.blogspot.com/2012/02/real-time-objectdetection-in-opencv.html 17
References [1] Images and Details on the Pixel Probe can be found in the paper: A Single-Pixel Touchless Laser Tracker Probe by Joshua Gordon, David Novotny and Alexandra Curtin (NIST), The Journal of the CMSC, Autumn 2015 [2] Camera Calibration Toolbox, by Jean-Yves Bouguet, CALTECH, http://www.vision.caltech.edu/bouguetj/calib_doc/index.html [3] Yonghong Xie and Qiang Ji, A New Efficient Ellipse Detection Method, IEEE Pattern Recognition, 16th International Conf., Vol 2, pp. 957-960, 2002 [4] Basca, C.A., Randomized Hough Transform for Ellipse Detection with Result Clustering http://ieeexplore.ieee.org/xpl/articledetails.jsp?arnumber=1630222 (paper unavailable) [5] Wen Yuan Chen and Jung Lin, Object Detection Scheme Using Cross Correlation and Affine Transformation Techniques, International Journal of Computer, Consumer and Control (IJ3C), Vol. 3, No.2 (2014) 18