System 1 Overview. System 1 Overview. Geometric Model-based Object Recognition. This Lecture: Geometric description Next Lecture: Model matching

Transcription

1 System 1 Overview How to discriminate between and also estimate image positions System 1 Overview Geometric Model-based Object Recognition This Lecture: Geometric description Next Lecture: Model matching vs Geometric Model-based Object Recognition Pose estimation Verification AV: 2D Geometric vision Fisher system 1 slide 1 AV: 2D Geometric vision Fisher system 1 slide 2 Motivation - automated visual inspection Manufacturing High speed product verification Largest use of computer vision systems worldwide Most western manufacturing has some visual quality control Given: Isolated binary image object Assume: Introduction 1. Geometric shape models for parts to be recognized (,) (12,) (4,4) (8,4) (,4) (12,4) (4,12) (8,12) AV: 2D Geometric vision Fisher system 1 slide 3 AV: 2D Geometric vision Fisher system 1 slide 4

2 2. Image feature positions Data Description Goal: describe parts in same vocabulary of boundary shapes as model Get object pixels that lie on the boundary Split pixels into straight line sets System does: 1. Matches image and model features 2. Estimates transformation mapping model onto data Find corners where the lines meet (Here we ignore curved boundaries.) AV: 2D Geometric vision Fisher system 1 slide 5 AV: 2D Geometric vision Fisher system 1 slide 6 Boundary Finding 1) Get points that lie on boundary: [r,c] = find( bwperim(image,4) == 1 ) Removing Dangling Spurs Spur: any boundary pixel with only 1 neighbor inside a 3x3 neighborhood ) Remove any spurs on boundary, track and segment [sr,sc] = removespurs(r,c,h,w); [tr,tc] = boundarytrack(sr,sc); [cr,cc] = findcorners(tr,tc); AV: 2D Geometric vision Fisher system 1 slide 7 AV: 2D Geometric vision Fisher system 1 slide 8

3 Removing Unnecessary Boundary Pixels changed=1; while changed==1 changed = ; [sr,sc] = find(work==1); % work: boundary pixels for i = 1 : length(sr) % check each boundary point neigh = work(sr(i)-1:sr(i)+1,sc(i)-1:sc(i)+1); count=sum(sum(neigh)); if count < 3 % only point and at most work(sr(i),sc(i)) = ; % 1 neighbor so remove it changed=1; Trailing s omitted. Find unnecessary corners: * - boundary point to keep c - boundary point to remove - boundary point thru here somehow shaded box - interior or exterior pixel thick red box - pixel neighbourhood inspected * c * * c * c * * * * c AV: 2D Geometric vision Fisher system 1 slide 9 AV: 2D Geometric vision Fisher system 1 slide 1 Raw boundary: Boundary Cleaning Results 23 Cleaned boundary: AV: 2D Geometric vision Fisher system 1 slide 11 AV: 2D Geometric vision Fisher system 1 slide 12

4 Getting a Consecutive Boundary Track Tracking Results Despurred boundary (unorganized point set): TRACK TO FIRST UNTRACKED BOUNDARY PIXEL ENCOUNTERED AS i GOES NEXT DIRECTIONS x NEXT = (LAST i) MOD EXAMPLE TRACKING X X LAST MOVE = 3 NEXT MOVE = 8,1,2,3,4,5, Tracked boundary (consecutive point set): AV: 2D Geometric vision Fisher system 1 slide 13 AV: 2D Geometric vision Fisher system 1 slide 14 Midlecture Problem Given the following tracking sequence: a b c d f X X e g What is the order of pixels to be considered for tracking to the next pixel AV: 2D Geometric vision Fisher system 1 slide A Recursive splitting the boundary into linear segments B Find leftmost point A 2. Find rightmost point B 3. Split points in set A >B and B >A: (a) Find line thru current segment points X & Y (b) Find point Z furthest from the line at distance d (c) If d is less than a threshold, then this segment finished (d) Otherwise, create new sets X >Z and Z >Y and recurse AV: 2D Geometric vision Fisher system 1 slide 16

5 Recursive Splitting Algorithm Y Recursive Splitting Code function recsplit(r,c,threshold) global numlines lines n = length(r); % total number of points vec = [c(n)-c(1), r(1)-r(n)]; % unit vector vec = vec/norm(vec); % perpicular to XY X d1 d2 Z % find point furthest from line maxdist = ; for i = 1 : n dist = abs( [r(i) - r(1), c(i) - c(1)] * vec ); if dist > maxdist maxdist = dist; maxindex = i; % where furthest AV: 2D Geometric vision Fisher system 1 slide 17 AV: 2D Geometric vision Fisher system 1 slide 18 % check for splitting by testing maximum point distance if maxdist < threshold % then it s a single line - save it numlines = numlines + 1; lines(numlines,1) = r(1); lines(numlines,2) = c(1); lines(numlines,3) = r(n); lines(numlines,4) = c(n); else % otherwise it needs to be split up recsplit(r(1:maxindex),c(1:maxindex),threshold); recsplit(r(maxindex:n),c(maxindex:n),threshold); Segmented boundary: Splitting Results AV: 2D Geometric vision Fisher system 1 slide 19 AV: 2D Geometric vision Fisher system 1 slide 2

6 Describing Lines Endpoints Length True Length Full Result Set (249,123)-(261,127) 13 - (261,127)-(373,289) (373,289)-(39,316) 32 - (39,316)-(388,33) 14 - (388,33)-(85,536) (85,536)-(13,437) (13,437)-(25,268) (25,268)-(186,171) (186,171)-(249,123) Input into matcher: extra lines, short lines, longer lines AV: 2D Geometric vision Fisher system 1 slide 21 AV: 2D Geometric vision Fisher system 1 slide 22 Running Program >> doall Input model to image scale factor (float) 3.9 Want to use live test data (,1) Test image file stem (filestem) TESTDATA1/f initial_split = ans = Want to process another image 2 (,1) 1 numlines = 1 AV: 2D Geometric vision Fisher system 1 slide 23 AV: 2D Geometric vision Fisher system 1 slide 24

7 Discussion 1. Simple boundary track and segment process 2. Gives compact line-based description 3. May have some extra segments 4. Segments may be too long or short 5. Description is input into matcher What Have We Learned Introduction to Data cleaning Boundary/Curve tracking Curve segmentation From pixels to descriptions Next: Matching descriptions to models AV: 2D Geometric vision Fisher system 1 slide 25 AV: 2D Geometric vision Fisher system 1 slide 26 System 1 Overview How to discriminate between these How to estimate object positions System 1 Overview Geometric model-based recognition processes Last Lecture: geometric description This Lecture: model matching vs Geometric model-based recognition pose estimation Verification AV: 2D Geometric vision Fisher system 1 slide 27 AV: 2D Geometric vision Fisher system 1 slide 28

8 Introduction Given: Sets of model lines {m i } in a scene coordinate system Set of image lines {d j } in an image coordinate system Image to scene scale conversion factor σ (pixels to cm) Do: 1. Match image and model lines {(m i, d j )} 2. Estimate transformation mapping model onto data: R, t 3. Verify matching and pose estimate Output: identity and position (R, t) Interpretation Tree matching Goal: Correspondence between subset of M model features {m i } and D data features {d j } Complete (exhaustive, depth-first) search - if a match exists, it will be found Needs a wildcard ( * ) data feature to match model features with no corresponding data feature (occlusion, segmentation failure) Can find multiple solutions Result: {(m i, d ji )} set of matched features AV: 2D Geometric vision Fisher system 1 slide 29 AV: 2D Geometric vision Fisher system 1 slide 3 Search Tree Expand by model feature at each new level m1 m2 mm d1 d1 d2 d1 d2... d dd... dd * dd * Any given node in tree represents a set of matches {(m i, d ji )} * Reducing Search Complexity Do we need to consider all paths in search tree No: Suppose current match state has these pairs matched: {(m i, d ji )}, i = 1..k Given a new pair (m k+1, d jk+1 ) 1. unary test(m k+1, d jk+1 ) - terminates exting search path if new pair has incompatible properties 2. binary test(m k+1, d jk+1, m x, d jx ) for all x = 1..k - terminates exting search path AV: 2D Geometric vision Fisher system 1 slide 31 AV: 2D Geometric vision Fisher system 1 slide 32

9 if new pair has incompatible properties with each previous pairing on this tree branch (as all parts of the same object are compatible). 3. Early success limit L - can stop search when have {(m i, d ji )}, i = 1..L compatible pairs Midlecture Problem What are good unary/binary properties to test if matching parts with sets of circular holes Eg: 4. Early failure limit L - can stop search when can never get L pairs on this path. If have t non-wildcard matches on this path out of k pairings, then fail if t + (M k) < L AV: 2D Geometric vision Fisher system 1 slide 33 AV: 2D Geometric vision Fisher system 1 slide 34 Computational Complexity M model feature tree levels. D data features on each level plus 1 wildcard Worst case: (D + 1) M nodes in tree to visit p u - probability that any random model feature and any random data feature pass unary test p b - probability that any 2 random model features and any 2 random data features pass binary test Then, if p b MD < 2, then the average case complexity of ITREE search is O(LD 2 ) Much smaller, but can still be substantial AV: 2D Geometric vision Fisher system 1 slide 35 IT algorithm matlab code % interpretation tree - match model and data lines until % Limit are successfully paired or can never get Limit % model - current model % numm - number of lines in the model % mlevel - last matched model feature % Limit - early termination threshold % pairs(:,2) - paired model-data features % numpairs - number of paired features function ok=itree(model,numm,mlevel,limit,pairs,numpairs) global Models numlines datalines % check for termination conditions if numpairs >= Limit % enough pairs to verify AV: 2D Geometric vision Fisher system 1 slide 36

10 [theta,trans] = estimatepose(model,numpairs,pairs) for p = 1 : 4 ok = verifymatch(theta(p),trans(p,:),model, numpairs,pairs); if ok return % successful verification return % failure to verify - continue search % never enough pairs if numpairs + numm - mlevel < Limit ok=; return % normal case - see if we can ext pair list mlevel = mlevel+1; for d = 1 : numlines % try all data lines % do unary test if unarytest(model,mlevel,d) % do all binary tests passed=1; for p = 1 : numpairs if ~binarytest(model,mlevel,d,pairs(p,1),pairs(p,2)) passed=; break AV: 2D Geometric vision Fisher system 1 slide 37 AV: 2D Geometric vision Fisher system 1 slide 38 if passed % passed all tests: add to matched pairs and recurse pairs(numpairs+1,1)=mlevel; pairs(numpairs+1,2)=d; ok=itree(model,numm,mlevel,limit,pairs,numpairs+1); if ok return % successful verification Algorithm Block Diagram % wildcard case - go to next model feature ok = itree(model,numm,mlevel,limit,pairs,numpairs); AV: 2D Geometric vision Fisher system 1 slide 39 AV: 2D Geometric vision Fisher system 1 slide 4

11 Line matching unary test DATA LINES Line matching binary tests MODEL LINES DATA LINE MODEL LINE l d l m Pass test if σl m (1 δ u ) l d σl m (1 + δ u ) Allows for calibration and segmentation errors Position indepent property (δ u =.3 typical) α β Pass test if α β δ b Allows for calibration and segmentation errors Position indepent property (δ b =.2 radians typical) Also: don t allow duplicate use of model or data lines AV: 2D Geometric vision Fisher system 1 slide 41 AV: 2D Geometric vision Fisher system 1 slide 42 Matching performance Limit L = number of model lines - 1 Tries all models Stops at first verified model instance for each model Different Matched Models & Instances Image True Model Tee Thin L Thick L 1 Tee Tee Tee Tee Thin L Thin L Thin L Thin L Thick L Thick L Thick L Thick L 2 3 AV: 2D Geometric vision Fisher system 1 slide 43 AV: 2D Geometric vision Fisher system 1 slide 44

12 Pose Estimation Goal: eliminate invalid matches & find object pose Given a set {(m i, d ji )}, i = 1..L of compatible pairs Find the rotation R and translation t that transforms the model onto the data features. This is the pose or position Let R = cos(θ) sin(θ) sin(θ) cos(θ) be the rotation matrix If p is a model point, then R p + t is the transformed model point Usually estimate rotation R first and then translation t Estimating Rotation Given model line i points {( m i1, m i2 )} Corresponding data line points {( d i1, d i2 )} u m i1 m i2 Model line direction unit vector: i u i = Data line direction unit vector: m i2 m i1 m i2 m i1 d v i = i2 d i1 d i2 d i1 AV: 2D Geometric vision Fisher system 1 slide 45 AV: 2D Geometric vision Fisher system 1 slide 46 If no data errors, want R such that v i = ±R u i (± as don t know if points are in same order) But, as we have errors least squares solution Step 1: compute vectors perpicular to v i If v i = (v x1, v y1 ), then perpicular is ( v yi, v xi ) Step 2: compute error between v i and R u i Use dot product of R u i and perpicular, which equals sin() of angular error, which is small, so sin(error). = error Step 3: Reformulate error Let R = cos(θ) sin(θ) sin(θ) cos(θ) Multiplying out and grouping terms: ǫ i = (v xi u yi v yi u xi, v yi u yi + v xi u xi )(cos(θ), sin(θ)) Make a matrix equation ǫ = D(cos(θ), sin(θ)) Each row of L vector ǫ is ǫ i and each row of L 2 matrix D is (v xi u yi v yi u xi, v yi u yi + v xi u xi ) The least square error is ǫ ǫ = (cos(θ), sin(θ))d D(cos(θ), sin(θ)) ǫ i = ( v yi, v xi )R(u xi, u yi ) AV: 2D Geometric vision Fisher system 1 slide 47 AV: 2D Geometric vision Fisher system 1 slide 48

13 Step 4: Finding rotation that minimizes least square error Let D D = e f g h Then, we minimize (cos(θ), sin(θ)) e f (cos(θ), sin(θ)) = g h ecos(θ) 2 + (f + g)cos(θ)sin(θ) + hsin(θ) 2 Differentiate wrt θ and set equal to gives: (f + g)cos(θ) 2 + 2(h e)cos(θ)sin(θ) (f + g)sin(θ) 2 = Solving gives: tan(θ) = (h e) ± (e h) 2 + (f + g) 2 (f + g) Four θ solutions (2 for ±, 2 for tan(θ) = tan(π + θ)). Try to verify all 4. Divide by cos(θ) 2 (if cos(θ) = then use special case) gives: (f + g)tan(θ) 2 + 2(e h)tan(θ) (f + g) = AV: 2D Geometric vision Fisher system 1 slide 49 AV: 2D Geometric vision Fisher system 1 slide 5 Estimating Translation By Least Squares d i2 Verification Transform model lines into place: for each m i compute σr m i + t w i d i1 ε i σrm i2+ t For each model-data line pair, do 3 tests: Test 1: Are model and data lines parallel (For simplicity, use m i in notation instead of σr m i + t) d i2 σrm + t w i is perpicular to rotated model line i i1 d i1 m i2 Offset error ǫ i = ( d i1 σr m i1 t) w i Differentiate i ǫ2 i wrt t, set equal to and solve for t gives: t = ( w i w i) 1 w i w i(d i1 σr m i1 ) If m i1 m i1 m i2 m i1 m i2 then OK (threshold =.9) d i1 d i2 d i1 d i2 > threshold AV: 2D Geometric vision Fisher system 1 slide 51 AV: 2D Geometric vision Fisher system 1 slide 52

14 Test 2: Are model and data lines close Test 3: Do model and data lines overlap d i2 d i2 u i m i2 w i d i1 ε i d i1 m i2 m i1 Let (r, s) = m i1 m i2 m i1 m i2 and w i = ( s, r) For k = i1, i2, compute ǫ i = ( d k m i1 ) w i If ǫ i < threshold then OK (threshold = 15 pixels) m i1 λ i For k = i1, i2, compute λ k = ( d k m i1 ) u i If tolerance m i1 m i2 λ k (1 + tolerance) m i1 m i2, then OK (tolerance =.3) AV: 2D Geometric vision Fisher system 1 slide 53 AV: 2D Geometric vision Fisher system 1 slide 54 Confusion Matrix Verified Position Result Examples Limit = number of model lines - 1 Est Est Est No Tee Thin L Thick L Est True Tee 4 True Thin L 3 1 True Thick L 4 Image 8 had Thin L model flipped over. Matching process can be exted to allow this. AV: 2D Geometric vision Fisher system 1 slide 55 AV: 2D Geometric vision Fisher system 1 slide 56

15 Discussion Efficient if good unary/binary tests Suitable for 5% (estimated) flat parts Similar techniques for shapes other than straight lines: circular arcs, corners, holes,... Extable to 3D (future lectures) Extensions for perspective projection What Have We Learned Introduction to Geometric Model-based Object Recognition General Feature Matching Algorithm 2D Least Squares rotation and translation estimation algorithms 2D Geometric Verification Algorithm AV: 2D Geometric vision Fisher system 1 slide 57 AV: 2D Geometric vision Fisher system 1 slide 58