3D Tracking Using Two High-Speed Vision Systems
|
|
- Maximilian Andrews
- 6 years ago
- Views:
Transcription
1 3D Tracking Using Two High-Speed Vision Systems Yoshihiro NAKABO 1, Idaku ISHII 2, Masatoshi ISHIKAWA 3 1 University of Tokyo, Tokyo, Japan, nakabo@k2.t.u-tokyo.ac.jp 2 Tokyo University of Agriculture and Technology, Tokyo, Japan, iishii@cc.tuat.ac.jp 3 University of Tokyo, Tokyo, Japan, ishikawa@k2.t.u-tokyo.ac.jp Abstract When considering real-world applications of robot control with visual servoing, both 3D information and a high feedback rate are required. We have developed a 3D target tracking with a 1ms feedback rate using two high-speed vision s called Column Parallel Vision (CPV) s. To obtain 3D information, such as position, orientation, and shape parameters of the target object, a feature-based algorithm has been introduced using moment feature values extracted from vision s for a spheroidal object model. Also, we propose a new 3D self windowing method to extract the target in 3D space, which is an extension of the conventional self windowing method in 2D images. 1 Introduction To control a robot dynamically by direct visual feedback [1, 2], a servo rate of around 1kHz is required. Conventional vision s using CCD cameras cannot realize this fast feedback rate because of the slow transmission rates of video standards. To solve this problem, we have developed a 1ms highspeed vision called the Column Parallel Vision (CPV) [3] and demonstrated a high-speed grasping task [4] using our vision in our previous work. However, our previous using one vision could not extract 3D information, thus we made the assumption of constant distance between the camera and target. In many real applications, 3D information such as positon, motion, orientation or/and shape information of the target is strongly required. In this research, we have developed a 3D tracking algorithm and a for extracting 3D information in a 1ms cycle time. Our goal is to apply them for high-speed grasping tasks in a 3D real world. (see Fig 1.) We have Presently with RIKEN, Bio-Mimetic Control Research Center, Nagoya, Japan, nakabo@bmc.riken.go.jp. First active vision First CPV High speed target tracking f 1 2D image feature extraction object Feature based 3D reconstruction Hand-Arm High speed motion Object Model Position, Orientation Shape parameters Motion How to reach and grasp the target? Figure 1: 3D tracking and grasping task f 2 Second active vision Second CPV 2D image feature extraction All processes are executed in a 1ms cycle time developed the 3D tracking using two high-speed vision s, in which bottleneck-free processing has been realized by massively parallel image processing in CPV s and a feature-based 3D reconstruction in a DSP. Active vision s called AVS-II [3] enable fast control of the gazes to track the target at a 1ms visual servo rate. Also, we propose a new method for extracting the target in a 3D space called 3D self windowing, as conventional self windowing [5] is considered only in a 2D image plane. In the next section, we describe the details of the proposed algorithms. The and an implementation of these algorithms are described in section 3. In section 4, some experimental results are given.
2 2 3D Tracking Algorithms 2.1 Task-based object model Our final goal is, as shown in Fig.1, to catch the target moving fast and irregularly in a 3D space by the dynamically controlled hand-arm using direct feedback of 3D visual information. Considering these tasks, target position and motion in 3D cartesian coordinates should be obtained for tracking the target, and shape parameters and orientation of the target will be required for guiding the arm to the target and deciding the trajectories of the fingers in the hand for preshaping. In this research, we choose a spheroidal object model, whose parameters are centroid, radial length of rotation and direction and length of the rotation axis. Using these parameters, the approach of the arm can be determined from the direction of the rotaton axis and preshaping can be organized from the size and the length of each direction of the spheroid. These parameters contain sufficient information for our task, so that we are able to focus on the algorithms for extracting these 3D parameters using two high-speed vision s. 2.2 Reconstruction of ellipsoidal shape model (in 2D) In this section, we introduce the moment feature values and compute the ellipsoidal shape parameters from the moment values. Let [u, v] T be the image coordinates and I(u, v) beaninput image. The (i + j)th order bond moments m ij are defined by the following equation: m ij u i v j I(u, v) (1) u,v These moment feature values are often used in visual servoing. In our vision, we can extract these values at high speed, as shown in section 3. The center of gravity ū, v, variance σ 2 u,σ2 v and covariance C uv of an image pattern can be calculated by the moment values, as: ū = m 1 /m, v = m 1 /m (2) σ 2 u = m 2 ū 2 m (3) σ 2 v = m 2 v 2 m (4) C uv = m 11 ū vm. (5) The parameters of an ellipsoidal shape model will be computed from these moment feature values. First, we consider the basic ellipsoid pattern described as: u 2 /a 2 e + v 2 /b 2 e 1, (6) where a e and b e are the lengths of the major and minor (a e > b e ) axes of the ellipsoid. We consider the general description of ellipsoid S, by rotating θ e and translating [u e, v e ] T the basic ellipsoid given by equation (6). [ ] [ ][ ] [ ] u cos θe sin θ = e u ue v sin θ e cos θ e v + (7) v e We consider the binary image, where I(u, v) = 1 inside the ellipsoid, and I(u, v) = outside. Let us calculate the moment feature values from the ellipsoid pattern S. Clearly, the center of an ellipisoid is: u e = ū, v e = v. The second-order moments are calculated as: σ 2 u = u 2 Idudv = a eb e π S 4 (a2 e cos 2 θ e + b 2 e sin 2 θ e ) (8) σ 2 v = v 2 Idudv = a eb e π S 4 (a2 e sin 2 θ e + b 2 e cos 2 θ e ) (9) C uv = uvidudv = a eb e π S 4 (a2 e b2 e ) sin 2θ e. From these equations, θ e can be calculated as: θ e = 1 ( ) 2 arctan 2C uv σ 2 u. (1) σ2 v Solving (8) and (9), a e and b e can be obtained as: 4 a e = 4 8 (σ 2 u cos2 θ e σ 2 v sin2 θ e ) 3 π cos 2θ e σ 2 v cos2 θ e σ 2 u sin2 θ e 4 b e = 4 8 (σ 2 v cos 2 θ e σ 2 u sin 2 θ e ) 3 π cos 2θ e σ 2 u cos 2 θ e σ 2 v sin 2. θ e We have shown how to calculate all the parameters of the ellipsoid from the moment feature values. 2.3 Reconstruction of spheroidal shape model (in 3D) Next, we describe the reconstruction of a spheroidal shape model from the feature values extracted from one vision. Let x = [x, y, z] T be the cartesian coordinates of the object coordinate. We first consider the basic spheroid, whose center is positioned at the origin of the coordinate and the axis of symmetry is aligned along the x axis. The spheroid is described as: x T Σx = 1, where Σ=diag. [ ] 1/a 2 s, 1/b 2 s, 1/b 2 s. (11) We assume the weak perspective camera model. [ ] u u = f M(Rx + T), (12) v T z where M = [I 2 ], R is a 3 3 rotation matrix, T = [T x, T y, T z ] T is a translation vector from the object to the
3 camera coordinate, and f denotes focal length and scale factor of the camera. We assume that R i denotes the rotation θ i around an axis i, and introduce the following equations: R = R z R y R x (13) x = R y R x x (14) With respect to these equations, we divide the transformation of the camera projection into two phases. 1. An orthographic projective transformation from the 3D object coordinate to the x y plane. 2. A similarity transformation from the x y plane to the uv image plane. Now we consider the first transformation. At first, the relation of R x ΣR T x = Σ shows that there is no need to know the parameter θ x in this case. Substituting equation (14) into equation (11), we obtain the following equation: x T Σ x = 1, where Σ = R y ΣR y T. (15) Generally, the projection of the spheroid is an ellipsoid. In this case, an ellipsoid formed by an orthographic projection of the spheroid to the x y plane is an envelope generated by the intersecting lines of the x y plane and the tangent planes of the spheroid parallel to the z axis. We will calculate the projection practically. Any tangent planes at x on the spheroid (15) are described as: x T Σ x =. (16) If the plane is parallel to the z axis, the normal vector of the plane is orthogonal to the z axis, for example: x T Σ [,, 1] T =. (17) Substituting equation (15) into equation (17), we can eliminate z, and rewriting x x, we have the following equation, which describes an ellipsoid on the x y plane: x 2 + y 2 = 1. (18) a s2 cos 2 θ y + b 2 s sin 2 2 θ y b s Now we consider the second transformation. Substituting equation (13) and (14) into equation (12), we can obtain the following equation. [ u v ] = f T z ([ ][ ] cos θz sin θ z x sin θ z cos θ z y + [ Tx T y ]) (19) Finally, comparing equations (18),(19) and (6),(7), we obtain the following relations between ellipsoidal parameters and spheroidal parameters: θ z = θ e T x = u e T z / f (2) T y = v e T z / f (21) b s = b e T z / f a s2 cos 2 θ y + b 2 s sin 2 θ y = a e T z / f. Note that T x, T y, and θ z have been computed, but T z and θ y are unknown and we are not concerned with θ x with respect to the position and the orientation of the target. 2.4 Computing position and orientation (in 3D) In the previous section, we have calculated the parameters available from one camera. Now we will integrate information from two cameras. Let R b and T b be, respectively, rotation and translation matrices from the first camera to the second. We derive them from the initial setup of the two cameras and encoder sensor data of the active vision s. Let R i and T i be those from the target to the ith camera. Now we have: R b T 1 + T b = T 2, (22) R b R 1 = R 2. (23) Substituting equation (2) and (21) into equation (22), we have: u e1 / f u A = R e2 / f b v e1 / f v 1 e2 / f 1 A [ T z1, T z2 ] T = T b. We can solve these equations by minimizing the least squares error of the solution, as: [ T z1, T z2 ] T = A + T b = (A T A) 1 A T T b. Next, we compute the parameter θ y. Suppose R i = R zi R yi R xi. Multiplying the vector [1,, ] T to (23) from the right-hand side, R xi can be eliminated. Finally we obtain: where: cos θ y1 n sin θ y1 a = [ cos θ y2,, sin θ y2 ] T, (24) Solving (24), we obtain: [ n o a ] = R z2 T R b R z1. θ y1 = arctan (n 2 /a 2 ) ( ) a3 n 2 a 2 n 3 θ y2 = arctan. a 2 n 1 a 1 n 2 Now all the algorithms are shown for extracting 3D information from moment feature values.
4 Possible area of target in 3D space Possible area of target pattern in Image 1, known from Image 2. Frontier point Obstacle be the self window W i, as: W i = D(Tt 1 i ). (25) step 2. Create a tentative target pattern P i by masking the raw image S i from the vision sensors by the self windowing mask W i : Image 1 Epipolar geometry Image 2 Figure 2: Epipolar geometry in 3D self windowing 2.5 Conventional self windowing (in 2D) Now we will focus on a method of extracting the pattern of the target from input images on vision sensors. In our previous work, we proposed a self windowing method [5], in which self windowing masks are created from the target pattern in the previous frame cycle, so that the target can be tracked continuously providing there is a high enough frame rate of the vision. Basically, this method can be applied to the task here. However, when an occlusion occurs, this can be detected by an abrupt increase of the area of the target pattern, but we cannot distinguish the target pattern from the obstacle until the object patterns will be separated again in the images D Extended self windowing To solve this problem, we propose a 3D self windowing using an epipolar geometry of two vision s. As shown in Fig.2, assumig the pattern obtained from the conventional self windowing as a tentative target pattern, we can consider an area enclosed by the pencil of the contour of the tentative pattern and take a product space of these areas from two visions, which can be considered as a possible area where the target should be in the 3D space. This area can be used as a 3D mask to separate the target object from the obstacle, and it is certain that we can track the target continuously because of the high frame rate of the vision s D self windowing algorithm In this section, we describe our 3D self windowing algorithm. Let the number of i (i=1,2) denote each of the vision s. We assume the patterns to be binary (1,). step 1. Suppose the target pattern at the last frame T i t 1 has been obtained. Let the dilated (D) pattern of T i t 1 P i = W i S i. (26) step 3. Find the tangent lines l i max and l i min of the contour of the pattern P i passing an epipole e = [e u, e v ] t. Call the tangent points f i max and f i min the tentative frontier points. The sets of points L i max and L i min on the lines l i max and l i min can be described as: c max = max(c), subject to L P i φ (27) c min = min(c), subject to L P i φ (28) L i max = {u c max (u e u ) (v e v ) <ɛ} (29) L i min = {u c min (u e u ) (v e v ) <ɛ}, (3) where ɛ hasasufficiently small value. f i max and f i min can be chosen from the sets of points F i max and Fi min as: Fmax i = Li max Pi (31) Fmin i = Li min Pi. (32) step 4. Exchange the tentative frontier points between two vision s, and calculate epipolar lines m i max, m i min from these points as: [ m j maxt, 1] T = F [ f i maxt, 1] T, (i j) where the 3 3 matrix F is a fundamental matrix which is calculated from R b, T b, and known camera parameters. step 5. Though it cannot be ensured that the calculated epipolar lines m i max, m i min pass the true frontier points, we can say that the target object should be presented inside the area M i clipped by the epipolar lines m i max, m i min. Using Mi as the mask for extracting the target, Tt i can be described as: T i t = P i M i. (33) In conventional self windowing, the algorithm has been stopped at step 2 treating T i t = P i. In the proposed algorithm, the mask M i in (33) compresses the possible area of the target, so that an occlusion-free recognition is realized. 3 System and Implementation 3.1 Dual CPV and DSP In the following, we briefly describe our and demonstrate high-speed computing of the proposed algorithms.
5 Image input PD array 128 x 128 pixels Control signals 8bit ADC 128 Cycle time : 1ms Column parallel image transfer Instructions Controller PE array 128 x 128 PEs Figure 3: CPV Summation circuit Column parallel data inout Image feature extraction to DSP network Table 1: Processing time on CPV processing contents steps time 2D self windowing µs search frontier points µs 3D self window mask µs th order moment (m ) µs 1st order moments (m 1, m µs 2nd order moments (m 2, m µs 2nd order moment (m 11 ) µs total µs Active vision motion and position Active vision (AVS-II) - 1 Servo controller Parallel DSP First CPV and AVS (left camera) object Object image Object image Moment feature values CPV -1 3D SW mask parameters CPV -2 Active vision (AVS-II) - 2 Object model 3D SW mask parameters Moment feature values Servo controller 3D reconstruction Object model parameters, position, and orientation Obstacle Second CPV and AVS (right camera) Active vision motion and position Cycle time : 1ms Figure 5: Photo of experimental setup Figure 4: Block diagram of 3D tracking The consists of two independent vision s and a DSP. The vision s are called the CPV [3], which has photodetectors and an allpixel parallel processing array based on an S 3 PE architecture and exclusive summation circuit for calculating moment values, as shown in Fig.3. The architecture of the CPV is optimized for high-speed visual feedback, so that it can realize a 1ms feedback rate while executing various kinds of image processing algorithms. Each of the vision sensors of CPV s are attached onto the active vision s called AVS-II. Each of the active vision s enables high-speed gaze control for tracking the target independently. In the DSP, parallel DSPs (TI:C6721 4) are used for PD servo control of both actuators of AVS-II. Also, they are used for computing an integration of the feature values from two vision s. The block diagram of the entire is shown in Fig Implementation of algorithms In the 3D self windowing algorithm, identical operations on large sets of points are often used when processing images. There can be operated extremely fast by pixel parallel processings in a CPV. For example, an input image is first binarised in each pixels and the operations in (25),(26) and (31)-(33) are executed in a few steps. Also there are search processes in (27)-(3), but we only need to search limited regions of parameters, because there is a sufficiently high frame rate. The processing time of every algorithm in a CPV for proposed 3D tracking is shown in Tab.1. Note that the total processing time is less than half a millisecond, which is the cycle time of the tracking. A calculation of an epipolar geometry and a 3D reconstruction in the DSP, and an exchange of the feature values between the CPV and the DSP[3] are also sufficiently fast that, in total, a bottleneck-free processing for the goal task is realized in our. 4 Experimental Results The experimental setup is shown in Fig.5. The distance between the two vision s is 1cm and that from the camera to the object is about 8cm. The size of the spheroid is 2cm by 1cm. Though the target and the obstacle overlap in the image in Fig.5, the left camera
6 Input Image (binary) Calculated Moment extracted from CPV sysyem. Moment th order:mo=1222, 1st order:mx=69953 My= nd order:mxx= Myy= Mxy= Center X= Y= Variance Sx= Sy= Covariance Cxy= Tilt Theta= (rad) = (deg) Ellipse a= b= Figure 6: Results of reconstruction of ellipsoid From left image without obstacle with obstacle 3D self window masked Left camera image Obstacle (masked) -5 [pixel] Figure 7: Results of 3D self windowing Right camera image Table 2: Results of 3D reconstruction of spheroid parameters calculated truth error rate distance: to CPV1 71cm 65cm 9.2% distance: to CPV2 6cm 54cm 11.1% rotation: θ z 42deg. 4deg. 5.% rotation: θ y 28deg. 4deg. 3.% length of axis: a s 21cm 2cm 5.% length of radius: b s 1cm 1cm.% Left camera position and direction Right camera.4 [m].6 shows them to be separated. First, the results of a calculation of ellipsoidal parameters are shown in Fig.6. The left-hand image in the figure is the binarised input image and the right-hand image is the reconstructed ellipsoid whose parameters are calculated by the proposed algorithm, which seems to be well estimated. Next, we show the result of the 3D self windowing in Fig.7. Shown on the left are the trajectories of the target. Without a 3D self windowing mask, the trajectory is biased to the left and downwards by the disturbing obstacle. Images on the right are the results of 3D masking, so that only the target patterns are extracted. Last, we show an example of the result of the 3D reconstruction, shown in Tab.2 and Fig.8. Some of the parameters are calculated close to the true values. But some parameters (for example θ y ) are not sufficiently accurate even for a task such as the rough grasping of the target by the hand-arm. This might be caused by an inaccurate calibration of initial rotations of camera coordinates. A demonstration of the target tracking with two cameras can be viewed in a video clip in a CD-ROM of the proceedings. 5 Conclusion A 3D tracking consisting of two CPV s and a DSP has been developed. Also a momentfeature-based 3D reconstruction algorithm and a new 3D Figure 8: Result of 3D reconstruction in 3D graph self windowing method have been introduced. The processing time of the 3D tracking is less than 1ms, which is the required speed for real time and dynamic control of the robot. The accuracy of the recent will be improved by an accurate calibration of camera coordinates. References [1] K. Hashimoto, editor. Visual Servoing. World Scientific, [2] S. Hutchinson, G. D. Hager, and P. I. Corke. A Tutorial on Visual Servo Control. IEEE Trans. on Robotics and Automation, Vol. 12, No. 5, pp , [3] Y. Nakabo, M. Ishikawa, H. Toyoda, and S. Mizuno. 1ms Column Parallel Vision System and its Application of High Speed Tracking. In Proc. IEEE Int. Conf. on Robotics and Automation, pp , 2. [4] A. Namiki, Y. Nakabo, I. Ishii, and M. Ishikawa. 1-ms Sensory-Motor Fusion System. IEEE Trans. on Mechatronics, Vol. 5, No. 3, pp , 2. [5] I. Ishii, Y. Nakabo, and M. Ishikawa. Tracking Algorithm for 1ms Visual Feedback System Using Massively Parallel Processing Vision. In Proc. IEEE Int. Conf. on Robotics and Automation, pp , 1996.
Task selection for control of active vision systems
The 29 IEEE/RSJ International Conference on Intelligent Robots and Systems October -5, 29 St. Louis, USA Task selection for control of active vision systems Yasushi Iwatani Abstract This paper discusses
More informationMotion Planning for Dynamic Knotting of a Flexible Rope with a High-speed Robot Arm
The 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems October 18-22, 2010, Taipei, Taiwan Motion Planning for Dynamic Knotting of a Flexible Rope with a High-speed Robot Arm Yuji
More informationVisual Tracking of a Hand-eye Robot for a Moving Target Object with Multiple Feature Points: Translational Motion Compensation Approach
Visual Tracking of a Hand-eye Robot for a Moving Target Object with Multiple Feature Points: Translational Motion Compensation Approach Masahide Ito Masaaki Shibata Department of Electrical and Mechanical
More informationVisual Tracking of Unknown Moving Object by Adaptive Binocular Visual Servoing
Visual Tracking of Unknown Moving Object by Adaptive Binocular Visual Servoing Minoru Asada, Takamaro Tanaka, and Koh Hosoda Adaptive Machine Systems Graduate School of Engineering Osaka University, Suita,
More informationKeeping features in the camera s field of view: a visual servoing strategy
Keeping features in the camera s field of view: a visual servoing strategy G. Chesi, K. Hashimoto,D.Prattichizzo,A.Vicino Department of Information Engineering, University of Siena Via Roma 6, 3 Siena,
More informationParallel Extraction Architecture for Information of Numerous Particles in Real-Time Image Measurement
Parallel Extraction Architecture for Information of Numerous Paper: Rb17-4-2346; May 19, 2005 Parallel Extraction Architecture for Information of Numerous Yoshihiro Watanabe, Takashi Komuro, Shingo Kagami,
More information6-dof Eye-vergence visual servoing by 1-step GA pose tracking
International Journal of Applied Electromagnetics and Mechanics 52 (216) 867 873 867 DOI 1.3233/JAE-16225 IOS Press 6-dof Eye-vergence visual servoing by 1-step GA pose tracking Yu Cui, Kenta Nishimura,
More informationMachine vision. Summary # 11: Stereo vision and epipolar geometry. u l = λx. v l = λy
1 Machine vision Summary # 11: Stereo vision and epipolar geometry STEREO VISION The goal of stereo vision is to use two cameras to capture 3D scenes. There are two important problems in stereo vision:
More informationA 100Hz Real-time Sensing System of Textured Range Images
A 100Hz Real-time Sensing System of Textured Range Images Hidetoshi Ishiyama Course of Precision Engineering School of Science and Engineering Chuo University 1-13-27 Kasuga, Bunkyo-ku, Tokyo 112-8551,
More informationA Tool for Kinematic Error Analysis of Robots/Active Vision Systems
A Tool for Kinematic Error Analysis of Robots/Active Vision Systems Kanglin Xu and George F. Luger Department of Computer Science University of New Mexico Albuquerque, NM 87131 {klxu,luger}@cs.unm.edu
More informationVisual Servoing Utilizing Zoom Mechanism
IEEE Int. Conf. on Robotics and Automation 1995, pp.178 183, Nagoya, May. 12 16, 1995 1 Visual Servoing Utilizing Zoom Mechanism Koh HOSODA, Hitoshi MORIYAMA and Minoru ASADA Dept. of Mechanical Engineering
More informationShape Modeling of A String And Recognition Using Distance Sensor
Proceedings of the 24th IEEE International Symposium on Robot and Human Interactive Communication Kobe, Japan, Aug 31 - Sept 4, 2015 Shape Modeling of A String And Recognition Using Distance Sensor Keisuke
More information3D Environment Measurement Using Binocular Stereo and Motion Stereo by Mobile Robot with Omnidirectional Stereo Camera
3D Environment Measurement Using Binocular Stereo and Motion Stereo by Mobile Robot with Omnidirectional Stereo Camera Shinichi GOTO Department of Mechanical Engineering Shizuoka University 3-5-1 Johoku,
More informationComputer Vision Projective Geometry and Calibration. Pinhole cameras
Computer Vision Projective Geometry and Calibration Professor Hager http://www.cs.jhu.edu/~hager Jason Corso http://www.cs.jhu.edu/~jcorso. Pinhole cameras Abstract camera model - box with a small hole
More information955-fps Real-time Shape Measurement of a Moving/Deforming Object using High-speed Vision for Numerous-point Analysis
27 IEEE International Conference on Robotics and Automation Roma, Italy, 1-14 April 27 FrA2.4 955-fps Real-time Shape Measurement of a Moving/Deforming Object using High-speed Vision for Numerous-point
More information3D Terrain Sensing System using Laser Range Finder with Arm-Type Movable Unit
3D Terrain Sensing System using Laser Range Finder with Arm-Type Movable Unit 9 Toyomi Fujita and Yuya Kondo Tohoku Institute of Technology Japan 1. Introduction A 3D configuration and terrain sensing
More informationRobot Vision Control of robot motion from video. M. Jagersand
Robot Vision Control of robot motion from video M. Jagersand Vision-Based Control (Visual Servoing) Initial Image User Desired Image Vision-Based Control (Visual Servoing) : Current Image Features : Desired
More informationFlexible Calibration of a Portable Structured Light System through Surface Plane
Vol. 34, No. 11 ACTA AUTOMATICA SINICA November, 2008 Flexible Calibration of a Portable Structured Light System through Surface Plane GAO Wei 1 WANG Liang 1 HU Zhan-Yi 1 Abstract For a portable structured
More informationContents. 1 Introduction Background Organization Features... 7
Contents 1 Introduction... 1 1.1 Background.... 1 1.2 Organization... 2 1.3 Features... 7 Part I Fundamental Algorithms for Computer Vision 2 Ellipse Fitting... 11 2.1 Representation of Ellipses.... 11
More information3D Geometry and Camera Calibration
3D Geometry and Camera Calibration 3D Coordinate Systems Right-handed vs. left-handed x x y z z y 2D Coordinate Systems 3D Geometry Basics y axis up vs. y axis down Origin at center vs. corner Will often
More informationcalibrated coordinates Linear transformation pixel coordinates
1 calibrated coordinates Linear transformation pixel coordinates 2 Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration with partial
More informationEpipolar geometry. x x
Two-view geometry Epipolar geometry X x x Baseline line connecting the two camera centers Epipolar Plane plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections
More informationMeasurement of Pedestrian Groups Using Subtraction Stereo
Measurement of Pedestrian Groups Using Subtraction Stereo Kenji Terabayashi, Yuki Hashimoto, and Kazunori Umeda Chuo University / CREST, JST, 1-13-27 Kasuga, Bunkyo-ku, Tokyo 112-8551, Japan terabayashi@mech.chuo-u.ac.jp
More informationEllipse fitting using orthogonal hyperbolae and Stirling s oval
Ellipse fitting using orthogonal hyperbolae and Stirling s oval Paul L. Rosin Abstract Two methods for approximating the normal distance to an ellipse using a) its orthogonal hyperbolae, and b) Stirling
More informationStructure from Motion and Multi- view Geometry. Last lecture
Structure from Motion and Multi- view Geometry Topics in Image-Based Modeling and Rendering CSE291 J00 Lecture 5 Last lecture S. J. Gortler, R. Grzeszczuk, R. Szeliski,M. F. Cohen The Lumigraph, SIGGRAPH,
More informationMidterm Exam Solutions
Midterm Exam Solutions Computer Vision (J. Košecká) October 27, 2009 HONOR SYSTEM: This examination is strictly individual. You are not allowed to talk, discuss, exchange solutions, etc., with other fellow
More informationRectification and Distortion Correction
Rectification and Distortion Correction Hagen Spies March 12, 2003 Computer Vision Laboratory Department of Electrical Engineering Linköping University, Sweden Contents Distortion Correction Rectification
More informationOmni-Directional Visual Servoing for Human-Robot Interaction
Omni-Directional Visual Servoing for Human-Robot Interaction Peng Chang and Martial Hebert [Peng, hebert]@ri.cmu.edu Robotics Institute, Carnegie Mellon University, Pittsburgh PA 15213 Abstract In this
More informationMOTION. Feature Matching/Tracking. Control Signal Generation REFERENCE IMAGE
Head-Eye Coordination: A Closed-Form Solution M. Xie School of Mechanical & Production Engineering Nanyang Technological University, Singapore 639798 Email: mmxie@ntuix.ntu.ac.sg ABSTRACT In this paper,
More information1 Projective Geometry
CIS8, Machine Perception Review Problem - SPRING 26 Instructions. All coordinate systems are right handed. Projective Geometry Figure : Facade rectification. I took an image of a rectangular object, and
More information55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography Estimating homography from point correspondence
More informationComputer Vision Projective Geometry and Calibration. Pinhole cameras
Computer Vision Projective Geometry and Calibration Professor Hager http://www.cs.jhu.edu/~hager Jason Corso http://www.cs.jhu.edu/~jcorso. Pinhole cameras Abstract camera model - box with a small hole
More informationLUMS Mine Detector Project
LUMS Mine Detector Project Using visual information to control a robot (Hutchinson et al. 1996). Vision may or may not be used in the feedback loop. Visual (image based) features such as points, lines
More informationSelf-calibration of a pair of stereo cameras in general position
Self-calibration of a pair of stereo cameras in general position Raúl Rojas Institut für Informatik Freie Universität Berlin Takustr. 9, 14195 Berlin, Germany Abstract. This paper shows that it is possible
More informationModel Based Perspective Inversion
Model Based Perspective Inversion A. D. Worrall, K. D. Baker & G. D. Sullivan Intelligent Systems Group, Department of Computer Science, University of Reading, RG6 2AX, UK. Anthony.Worrall@reading.ac.uk
More informationStereo Vision. MAN-522 Computer Vision
Stereo Vision MAN-522 Computer Vision What is the goal of stereo vision? The recovery of the 3D structure of a scene using two or more images of the 3D scene, each acquired from a different viewpoint in
More informationToday. Stereo (two view) reconstruction. Multiview geometry. Today. Multiview geometry. Computational Photography
Computational Photography Matthias Zwicker University of Bern Fall 2009 Today From 2D to 3D using multiple views Introduction Geometry of two views Stereo matching Other applications Multiview geometry
More informationTool Center Position Determination of Deformable Sliding Star by Redundant Measurement
Applied and Computational Mechanics 3 (2009) 233 240 Tool Center Position Determination of Deformable Sliding Star by Redundant Measurement T. Vampola a, M. Valášek a, Z. Šika a, a Faculty of Mechanical
More informationMotion. 1 Introduction. 2 Optical Flow. Sohaib A Khan. 2.1 Brightness Constancy Equation
Motion Sohaib A Khan 1 Introduction So far, we have dealing with single images of a static scene taken by a fixed camera. Here we will deal with sequence of images taken at different time intervals. Motion
More informationSilhouette Coherence for Camera Calibration under Circular Motion
Silhouette Coherence for Camera Calibration under Circular Motion Carlos Hernández, Francis Schmitt and Roberto Cipolla Appendix I 2 I. ERROR ANALYSIS OF THE SILHOUETTE COHERENCE AS A FUNCTION OF SILHOUETTE
More informationTask analysis based on observing hands and objects by vision
Task analysis based on observing hands and objects by vision Yoshihiro SATO Keni Bernardin Hiroshi KIMURA Katsushi IKEUCHI Univ. of Electro-Communications Univ. of Karlsruhe Univ. of Tokyo Abstract In
More informationAbsolute Scale Structure from Motion Using a Refractive Plate
Absolute Scale Structure from Motion Using a Refractive Plate Akira Shibata, Hiromitsu Fujii, Atsushi Yamashita and Hajime Asama Abstract Three-dimensional (3D) measurement methods are becoming more and
More informationImage Based Visual Servoing Using Algebraic Curves Applied to Shape Alignment
The 29 IEEE/RSJ International Conference on Intelligent Robots and Systems October 11-15, 29 St. Louis, USA Image Based Visual Servoing Using Algebraic Curves Applied to Shape Alignment Ahmet Yasin Yazicioglu,
More informationVisual Recognition: Image Formation
Visual Recognition: Image Formation Raquel Urtasun TTI Chicago Jan 5, 2012 Raquel Urtasun (TTI-C) Visual Recognition Jan 5, 2012 1 / 61 Today s lecture... Fundamentals of image formation You should know
More informationUnit 3 Multiple View Geometry
Unit 3 Multiple View Geometry Relations between images of a scene Recovering the cameras Recovering the scene structure http://www.robots.ox.ac.uk/~vgg/hzbook/hzbook1.html 3D structure from images Recover
More informationTwo-view geometry Computer Vision Spring 2018, Lecture 10
Two-view geometry http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2018, Lecture 10 Course announcements Homework 2 is due on February 23 rd. - Any questions about the homework? - How many of
More informationOptical Flow-Based Person Tracking by Multiple Cameras
Proc. IEEE Int. Conf. on Multisensor Fusion and Integration in Intelligent Systems, Baden-Baden, Germany, Aug. 2001. Optical Flow-Based Person Tracking by Multiple Cameras Hideki Tsutsui, Jun Miura, and
More informationChapter 3 Image Registration. Chapter 3 Image Registration
Chapter 3 Image Registration Distributed Algorithms for Introduction (1) Definition: Image Registration Input: 2 images of the same scene but taken from different perspectives Goal: Identify transformation
More informationMultiple Views Geometry
Multiple Views Geometry Subhashis Banerjee Dept. Computer Science and Engineering IIT Delhi email: suban@cse.iitd.ac.in January 2, 28 Epipolar geometry Fundamental geometric relationship between two perspective
More informationA Stratified Approach for Camera Calibration Using Spheres
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. Y, MONTH YEAR 1 A Stratified Approach for Camera Calibration Using Spheres Kwan-Yee K. Wong, Member, IEEE, Guoqiang Zhang, Student-Member, IEEE and Zhihu
More informationExpanding gait identification methods from straight to curved trajectories
Expanding gait identification methods from straight to curved trajectories Yumi Iwashita, Ryo Kurazume Kyushu University 744 Motooka Nishi-ku Fukuoka, Japan yumi@ieee.org Abstract Conventional methods
More informationRecognizing Buildings in Urban Scene of Distant View ABSTRACT
Recognizing Buildings in Urban Scene of Distant View Peilin Liu, Katsushi Ikeuchi and Masao Sakauchi Institute of Industrial Science, University of Tokyo, Japan 7-22-1 Roppongi, Minato-ku, Tokyo 106, Japan
More informationDevelopment of 3D Positioning Scheme by Integration of Multiple Wiimote IR Cameras
Proceedings of the 5th IIAE International Conference on Industrial Application Engineering 2017 Development of 3D Positioning Scheme by Integration of Multiple Wiimote IR Cameras Hui-Yuan Chan *, Ting-Hao
More informationRobot vision review. Martin Jagersand
Robot vision review Martin Jagersand What is Computer Vision? Computer Graphics Three Related fields Image Processing: Changes 2D images into other 2D images Computer Graphics: Takes 3D models, renders
More information1D camera geometry and Its application to circular motion estimation. Creative Commons: Attribution 3.0 Hong Kong License
Title D camera geometry and Its application to circular motion estimation Author(s Zhang, G; Zhang, H; Wong, KKY Citation The 7th British Machine Vision Conference (BMVC, Edinburgh, U.K., 4-7 September
More informationOctree-Based Obstacle Representation and Registration for Real-Time
Octree-Based Obstacle Representation and Registration for Real-Time Jaewoong Kim, Daesik Kim, Junghyun Seo, Sukhan Lee and Yeonchool Park* Intelligent System Research Center (ISRC) & Nano and Intelligent
More informationCS201 Computer Vision Camera Geometry
CS201 Computer Vision Camera Geometry John Magee 25 November, 2014 Slides Courtesy of: Diane H. Theriault (deht@bu.edu) Question of the Day: How can we represent the relationships between cameras and the
More informationVisualisation Pipeline : The Virtual Camera
Visualisation Pipeline : The Virtual Camera The Graphics Pipeline 3D Pipeline The Virtual Camera The Camera is defined by using a parallelepiped as a view volume with two of the walls used as the near
More informationEpipolar Geometry and the Essential Matrix
Epipolar Geometry and the Essential Matrix Carlo Tomasi The epipolar geometry of a pair of cameras expresses the fundamental relationship between any two corresponding points in the two image planes, and
More informationCS223b Midterm Exam, Computer Vision. Monday February 25th, Winter 2008, Prof. Jana Kosecka
CS223b Midterm Exam, Computer Vision Monday February 25th, Winter 2008, Prof. Jana Kosecka Your name email This exam is 8 pages long including cover page. Make sure your exam is not missing any pages.
More informationComputer Vision cmput 428/615
Computer Vision cmput 428/615 Basic 2D and 3D geometry and Camera models Martin Jagersand The equation of projection Intuitively: How do we develop a consistent mathematical framework for projection calculations?
More informationEpipolar Geometry in Stereo, Motion and Object Recognition
Epipolar Geometry in Stereo, Motion and Object Recognition A Unified Approach by GangXu Department of Computer Science, Ritsumeikan University, Kusatsu, Japan and Zhengyou Zhang INRIA Sophia-Antipolis,
More informationThe end of affine cameras
The end of affine cameras Affine SFM revisited Epipolar geometry Two-view structure from motion Multi-view structure from motion Planches : http://www.di.ens.fr/~ponce/geomvis/lect3.pptx http://www.di.ens.fr/~ponce/geomvis/lect3.pdf
More informationProf. Fanny Ficuciello Robotics for Bioengineering Visual Servoing
Visual servoing vision allows a robotic system to obtain geometrical and qualitative information on the surrounding environment high level control motion planning (look-and-move visual grasping) low level
More informationA Fast Linear Registration Framework for Multi-Camera GIS Coordination
A Fast Linear Registration Framework for Multi-Camera GIS Coordination Karthik Sankaranarayanan James W. Davis Dept. of Computer Science and Engineering Ohio State University Columbus, OH 4320 USA {sankaran,jwdavis}@cse.ohio-state.edu
More informationThree-Dimensional Sensors Lecture 2: Projected-Light Depth Cameras
Three-Dimensional Sensors Lecture 2: Projected-Light Depth Cameras Radu Horaud INRIA Grenoble Rhone-Alpes, France Radu.Horaud@inria.fr http://perception.inrialpes.fr/ Outline The geometry of active stereo.
More informationAssignment 3. Position of the center +/- 0.1 inches Orientation +/- 1 degree. Decal, marker Stereo, matching algorithms Pose estimation
Assignment 3 1. You are required to analyze the feasibility of designing a vision system for the robot gas station attendant. Assume that the driver parks the car so that the flap and the cap are in a
More informationMultiview Stereo COSC450. Lecture 8
Multiview Stereo COSC450 Lecture 8 Stereo Vision So Far Stereo and epipolar geometry Fundamental matrix captures geometry 8-point algorithm Essential matrix with calibrated cameras 5-point algorithm Intersect
More informationN-Views (1) Homographies and Projection
CS 4495 Computer Vision N-Views (1) Homographies and Projection Aaron Bobick School of Interactive Computing Administrivia PS 2: Get SDD and Normalized Correlation working for a given windows size say
More informationMCE/EEC 647/747: Robot Dynamics and Control. Lecture 3: Forward and Inverse Kinematics
MCE/EEC 647/747: Robot Dynamics and Control Lecture 3: Forward and Inverse Kinematics Denavit-Hartenberg Convention Reading: SHV Chapter 3 Mechanical Engineering Hanz Richter, PhD MCE503 p.1/12 Aims of
More informationComputer Vision. Coordinates. Prof. Flávio Cardeal DECOM / CEFET- MG.
Computer Vision Coordinates Prof. Flávio Cardeal DECOM / CEFET- MG cardeal@decom.cefetmg.br Abstract This lecture discusses world coordinates and homogeneous coordinates, as well as provides an overview
More information2. Give an example of a non-constant function f(x, y) such that the average value of f over is 0.
Midterm 3 Review Short Answer 2. Give an example of a non-constant function f(x, y) such that the average value of f over is 0. 3. Compute the Riemann sum for the double integral where for the given grid
More informationRobotics 2 Visual servoing
Robotics 2 Visual servoing Prof. Alessandro De Luca Visual servoing! objective use information acquired by vision sensors (cameras) for feedback control of the pose/motion of a robot (or of parts of it)
More informationEpipolar Geometry Prof. D. Stricker. With slides from A. Zisserman, S. Lazebnik, Seitz
Epipolar Geometry Prof. D. Stricker With slides from A. Zisserman, S. Lazebnik, Seitz 1 Outline 1. Short introduction: points and lines 2. Two views geometry: Epipolar geometry Relation point/line in two
More informationFactorization Method Using Interpolated Feature Tracking via Projective Geometry
Factorization Method Using Interpolated Feature Tracking via Projective Geometry Hideo Saito, Shigeharu Kamijima Department of Information and Computer Science, Keio University Yokohama-City, 223-8522,
More informationA Framework for 3D Pushbroom Imaging CUCS
A Framework for 3D Pushbroom Imaging CUCS-002-03 Naoyuki Ichimura and Shree K. Nayar Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology (AIST) Tsukuba,
More informationarxiv: v1 [cs.cv] 18 Sep 2017
Direct Pose Estimation with a Monocular Camera Darius Burschka and Elmar Mair arxiv:1709.05815v1 [cs.cv] 18 Sep 2017 Department of Informatics Technische Universität München, Germany {burschka elmar.mair}@mytum.de
More informationThree-Dimensional Measurement of Objects in Liquid with an Unknown Refractive Index Using Fisheye Stereo Camera
Three-Dimensional Measurement of Objects in Liquid with an Unknown Refractive Index Using Fisheye Stereo Camera Kazuki Sakamoto, Alessandro Moro, Hiromitsu Fujii, Atsushi Yamashita, and Hajime Asama Abstract
More informationAssist System for Carrying a Long Object with a Human - Analysis of a Human Cooperative Behavior in the Vertical Direction -
Assist System for Carrying a Long with a Human - Analysis of a Human Cooperative Behavior in the Vertical Direction - Yasuo HAYASHIBARA Department of Control and System Engineering Toin University of Yokohama
More informationMultiple View Geometry of Projector-Camera Systems from Virtual Mutual Projection
Multiple View Geometry of rojector-camera Systems from Virtual Mutual rojection Shuhei Kobayashi, Fumihiko Sakaue, and Jun Sato Department of Computer Science and Engineering Nagoya Institute of Technology
More informationCalibration and Synchronization of a Robot-Mounted Camera for Fast Sensor-Based Robot Motion
IEEE Int. Conf. on Robotics and Automation ICRA2005, Barcelona, Spain, April 2005 Calibration and Synchronization of a Robot-Mounted Camera for Fast Sensor-Based Robot Motion Friedrich Lange and Gerd Hirzinger
More informationCatadioptric camera model with conic mirror
LÓPEZ-NICOLÁS, SAGÜÉS: CATADIOPTRIC CAMERA MODEL WITH CONIC MIRROR Catadioptric camera model with conic mirror G. López-Nicolás gonlopez@unizar.es C. Sagüés csagues@unizar.es Instituto de Investigación
More informationSubpixel Corner Detection Using Spatial Moment 1)
Vol.31, No.5 ACTA AUTOMATICA SINICA September, 25 Subpixel Corner Detection Using Spatial Moment 1) WANG She-Yang SONG Shen-Min QIANG Wen-Yi CHEN Xing-Lin (Department of Control Engineering, Harbin Institute
More informationA New Algorithm for Measuring and Optimizing the Manipulability Index
DOI 10.1007/s10846-009-9388-9 A New Algorithm for Measuring and Optimizing the Manipulability Index Ayssam Yehia Elkady Mohammed Mohammed Tarek Sobh Received: 16 September 2009 / Accepted: 27 October 2009
More informationPrecise Omnidirectional Camera Calibration
Precise Omnidirectional Camera Calibration Dennis Strelow, Jeffrey Mishler, David Koes, and Sanjiv Singh Carnegie Mellon University {dstrelow, jmishler, dkoes, ssingh}@cs.cmu.edu Abstract Recent omnidirectional
More informationCamera Model and Calibration
Camera Model and Calibration Lecture-10 Camera Calibration Determine extrinsic and intrinsic parameters of camera Extrinsic 3D location and orientation of camera Intrinsic Focal length The size of the
More information1 Introduction. 2 Real-time Omnidirectional Stereo
J. of Robotics and Mechatronics, Vol. 14, 00 (to appear) Recognizing Moving Obstacles for Robot Navigation using Real-time Omnidirectional Stereo Vision Hiroshi Koyasu, Jun Miura, and Yoshiaki Shirai Dept.
More informationA Simulation Study and Experimental Verification of Hand-Eye-Calibration using Monocular X-Ray
A Simulation Study and Experimental Verification of Hand-Eye-Calibration using Monocular X-Ray Petra Dorn, Peter Fischer,, Holger Mönnich, Philip Mewes, Muhammad Asim Khalil, Abhinav Gulhar, Andreas Maier
More informationDynamic Time Warping for Binocular Hand Tracking and Reconstruction
Dynamic Time Warping for Binocular Hand Tracking and Reconstruction Javier Romero, Danica Kragic Ville Kyrki Antonis Argyros CAS-CVAP-CSC Dept. of Information Technology Institute of Computer Science KTH,
More information3D Sensing. 3D Shape from X. Perspective Geometry. Camera Model. Camera Calibration. General Stereo Triangulation.
3D Sensing 3D Shape from X Perspective Geometry Camera Model Camera Calibration General Stereo Triangulation 3D Reconstruction 3D Shape from X shading silhouette texture stereo light striping motion mainly
More informationStructure from Motion
11/18/11 Structure from Motion Computer Vision CS 143, Brown James Hays Many slides adapted from Derek Hoiem, Lana Lazebnik, Silvio Saverese, Steve Seitz, and Martial Hebert This class: structure from
More informationGraphics and Interaction Transformation geometry and homogeneous coordinates
433-324 Graphics and Interaction Transformation geometry and homogeneous coordinates Department of Computer Science and Software Engineering The Lecture outline Introduction Vectors and matrices Translation
More informationVisual servo control of mobile robots
Visual servo control of mobile robots homas Blanken 638483 DC 21.XXX Bachelor Final Project Dynamics and Control Department of Mechanical Engineering Eindhoven University of echnology Supervisor: dr. Dragan
More information2 1/2 D visual servoing with respect to planar. contours having complex and unknown shapes
2 /2 D visual servoing with respect to planar contours having complex and unknown shapes E. Malis, G. Chesi and R. Cipolla Abstract In this paper we present a complete system for segmenting, matching,
More informationIMAGE MOMENTS have been widely used in computer vision
IEEE TRANSACTIONS ON ROBOTICS, VOL. 20, NO. 4, AUGUST 2004 713 Image Moments: A General Useful Set of Features for Visual Servoing François Chaumette, Member, IEEE Abstract In this paper, we determine
More informationPOME A mobile camera system for accurate indoor pose
POME A mobile camera system for accurate indoor pose Paul Montgomery & Andreas Winter November 2 2016 2010. All rights reserved. 1 ICT Intelligent Construction Tools A 50-50 joint venture between Trimble
More informationCOMP30019 Graphics and Interaction Transformation geometry and homogeneous coordinates
COMP30019 Graphics and Interaction Transformation geometry and homogeneous coordinates Department of Computer Science and Software Engineering The Lecture outline Introduction Vectors and matrices Translation
More informationComputer Vision I - Algorithms and Applications: Multi-View 3D reconstruction
Computer Vision I - Algorithms and Applications: Multi-View 3D reconstruction Carsten Rother 09/12/2013 Computer Vision I: Multi-View 3D reconstruction Roadmap this lecture Computer Vision I: Multi-View
More informationIndoor Positioning System Based on Distributed Camera Sensor Networks for Mobile Robot
Indoor Positioning System Based on Distributed Camera Sensor Networks for Mobile Robot Yonghoon Ji 1, Atsushi Yamashita 1, and Hajime Asama 1 School of Engineering, The University of Tokyo, Japan, t{ji,
More informationRecovering structure from a single view Pinhole perspective projection
EPIPOLAR GEOMETRY The slides are from several sources through James Hays (Brown); Silvio Savarese (U. of Michigan); Svetlana Lazebnik (U. Illinois); Bill Freeman and Antonio Torralba (MIT), including their
More information