Object Pose from a Single Image

Object Pose from a Single Image

How Do We See Objects in Depth? Stereo Use differences between images in or left and right eye How mch is this difference for a car at 00 m? Moe or head sideways Or, the scene is moing Or we are moing in a car We know the sie and shape of objects raffic lights, car headlights and taillights

Headlights in the Dark A robot cold ealate its distance from incoming cars at night partly from a model of cars Distance between headlights known D m d Z Z = f d D Image plane f Center of projection 3

Object Pose with D Image Plane What happens if we don t know object s angle? 4

More Points Limited nmber of object poses ( or ) Head lights and one taillight ransparent car 5

Correspondence Problem When we know correspondences (i.e. matchings), pose is easier to find When we know the pose, correspondences are easier to find. Bt we need to find both at the same time Below, we first assme we know correspondences and describe how to sole the pose gien n corresponding points in image and object Perspectie n-point Problem hen we explore what to do when we don t know correspondences 6

7 Pose s. Calibration Problem Now we know calibration matrix K We can transform image points by K - transformation Projection matrix is now Soling pose problem consists of finding R and 6 nknowns = 0 0 0 K 0 0 y x s y x α α = ' ' ' w w - K [ ] = S S S Z Y X w 3 3 3-0 R 0 I K K [ ] = S S S Z Y X w R ] P = [R Canonical perspectie projection with f =

Iteratie Pose Calclation First we derie a linear system for the nknown parameters of rotation and translation that contains the known world coordinates of points and the homogenos coordinates of their images. Problem: Does not contain the wi components he wi components are reqired for compting homogeneos coordinates of images from the pixel locations hey can be compted once the rotation and translation parameters are estimated Soltion: Make a gess on wi, compte R and, then recompte wi, and recompte R and, etc 8

9 Iteratie Pose Calclation = S S S y x Z Y X w 3 r r r X r r r 3 = y x w i i i i Z Y X w ),,.( + r 3 = X r r = y x [ ] = y x r r X = y x Z Y X Z Y X Z Y X Z Y X 4 4 4 3 3 3 4 4 3 3 r r = 4 4 3 3 y x - M r r Non coplanar points needed (otherwise matrix M is singlar). At least 4 points.

0 Iteratie Pose Calclation Compte model matrix M and its inerse Assme Compte i = w i x i, i = w i y i Compte Compte, x, y, r, r, then r 3 = r x r Compte Go back to step and iterate ntil conergence i i i i Z Y X w ),,.( + r 3 = = 4 4 3 3 y x - M r r ),, ( = = i i i i w Z Y X. r 3

Iteratie Pose Calclation. Find object pose nder scaled orthographic projection. Project object points on lines of sight 3. Find scaled orthographic projection images of those points 4. Loop sing those images in step r 3

POSI for a Cbe Left: Actal perspectie image for cbe with known model op: Eoltion of perspectie image dring iteration Bottom: Eoltion of scaled orthographic projection

Application: 3D Mose 3

3 Points Each correspondence between scene point and image point determines eqations Since there are 6 degrees of freedom in the pose problems, the correspondences between 3 scene points in a known configration and 3 image points shold proide enogh eqations for compting the pose of the 3 scene points the pose of a triangle of known dimension is defined from a single image of the triangle Bt nonlinear method, to 4 soltions 4

riangle Pose Problem here are two basic approaches Analytically soling for nknown pose parameters Soling a 4th degree eqation in one pose parameter, and then sing the 4 soltions to the eqation to sole for remaining pose parameters problem: errors in estimating location of image featres can lead to either large pose errors or failre to sole the 4th degree eqation Approximate nmerical algorithms find soltions when exact methods fail de to image measrement error more comptation 5

Nmerical Method for riangle Pose A α B C' A' B' γ δ C Center of Projection β If distance R c to C is known, then possible locations of A (and B) can be compted they lie on the intersections of the line of sight throgh A' and the sphere of radis AC centered at C Once A and B are located, their distance can be compted and compared against the actal distance 6 AB

Nmerical Method for riangle Pose H A A' α C' γ δ B' C β B Not practical to search on R c since it is nbonded Instead, search on one anglar pose parameter, α. R c = AC cos α sin δ R a = R c cos δ ± AC sin α R b = R c cos γ ± [(BC -(RC sin γ) ] his reslts in for possible lengths for side AB Keep poses with the right AB length 7

Choosing Points on Objects Gien a 3-D object, how do we decide which points from its srface to choose for its model? Choose points that will gie rise to detectable featres in images For polyhedra, the images of its ertices will be points in the images where two or more long lines meet hese can be detected by edge detection methods Points on the interiors of regions, or along straight lines are not 8easily identified in images.

Example images 9

Choosing the Points Example: why not choose the midpoints of the edges of a polyhedra as featres midpoints of projections of line segments are not the projections of the midpoints of line segments if the entire line segment in the image is not identified, then we introdce error in locating midpoint 0

Strategy: Objects and Unknown Correspondences Pick p a small grop of points (3 or 4) on object, and candidate image points in image Find object pose for these correspondences Check or accmlate eidence by one of following techniqes: Clstering in pose space Image-Model Alignment and RANSAC

4-3--? 4 - point perspectie soltion Uniqe soltion for 6 pose parameters Comptational complexity of n 4 m 4 3 - point perspectie soltion Generally two soltions per triangle pair, bt sometimes for. Redced complexity of n 3 m 3

Redcing the Combinatorics of Pose Estimation How can we redce the nmber of matches Consider only qadrples of object featres that are simltaneosly isible extensie preprocessing 3

Redcing the Combinatorics of Pose Estimation Redcing the nmber of matches Consider only qadrples of image featres that Are connected by edges Are close to one another Bt not too close or the ineitable errors in estimating the position of an image ertex will lead to large errors in pose estimation Generally, try to grop the image featres into sets that are probably from a single object, and then only constrct qadrples from within a single grop 4

Image-Model Alignment Gien: A 3-D object modeled as a collection of points Image of a scene sspected to inclde an instance of the object, segmented into featre points Goal Hypothesie the pose of the object in the scene by matching (collections of) n model points against n featre points, enabling s to sole for the rigid body transformation from the object to world coordinate systems, and Verify that hypothesis by projecting the remainder of the model into the image and matching Look for edges connecting predicted ertex locations Srface markings 5

RANSAC RANdom SAmple Consenss Randomly select a set of 3 points in the image and a select a set of 3 points in the model Compte triangle pose and pose of model Project model at compted pose onto image Determine the set of projected model points that are within a distance threshold t of image points, called the consenss set After N trials, select pose with largest consenss set 6

Clstering in Pose Space Each matching of n model points against n featre points proides R and Each correct matching proides a similar rotation and translation Represent each pose by a point in a 6D space. hen points from correct matchings shold clster Or find clsters for points and find the clster where the rotations are most consistent Generalied Hogh transform if bins are sed 7

Scope of the Problem Flat objects s. 3D objects Grabbing flat tools on a tray s. grabbing handle of cp Ronded objects s. polyhedral objects Cp s. keyboard or CD cassette Rigid objects s. deformable objects 8

Pose and Recognition Soling the Pose Problem can be sed to sole the Recognition Problem for 3D objects: ry to find the pose of each item in the database of objects we want to identify Select the items whose projected points match the largest amonts of image points in the erification stage, and label the corresponding image regions with the item names. Bt many alternatie recognition techniqes do not proide the pose of the recognied item. 9

References VAS literatre on the sbject (larger than for calibration) Search for pose & object & model & USC Search in citeseer.nj.nec.com 30