Fttng: Votng and the Hough Transform Thurs Sept 4 Krsten Grauman UT Austn Last tme What are groupng problems n vson? Inspraton from human percepton Gestalt propertes Bottom-up segmentaton va clusterng Algorthms: Mode fndng and mean shft: k-means, mean-shft Graph-based: normalzed cuts Features: color, texture, Quantzaton for texture summares Images as graphs q p wpq w Fully-connected graph node (vertex) for every pxel lnk between every par of pxels, p,q affnty weght wpq for each lnk (edge) wpq measures smlarty» smlarty s nversely proportonal to dfference (n color and poston ) Source: Steve Setz 1
Segmentaton by Graph Cuts q p wpq w A B C Break Graph nto Segments Want to delete lnks that cross between segments Easest to break lnks that have low smlarty (low weght) smlar pxels should be n the same segments dssmlar pxels should be n dfferent segments Source: Steve Setz Cuts n a graph: Mn cut A Lnk Cut set of lnks whose removal makes a graph dsconnected cost of a cut: cut( A, B) w p, q p A, q B Fnd mnmum cut gves you a segmentaton fast algorthms exst for dong ths B Source: Steve Setz Measurng affnty for edge weghts One possblty: Small sgma: group only nearby ponts Large sgma: group dstant ponts
Measurng affnty for edge weghts σ=. Data ponts Affnty matrces σ=.1 σ=. σ=1 Cuts n a graph: Mn cut A Lnk Cut set of lnks whose removal makes a graph dsconnected cost of a cut: cut( A, B) w p, q p A, q B Fnd mnmum cut gves you a segmentaton fast algorthms exst for dong ths B Source: Steve Setz Mnmum cut Problem wth mnmum cut: Weght of cut proportonal to number of edges n the cut; tends to produce small, solated components. [Sh & Malk, 000 PAMI] 3
Cuts n a graph: Normalzed cut A B Normalzed Cut fx bas of Mn Cut by normalzng for sze of segments: cut( A, B) cut( A, B) assoc( A, V ) assoc( B, V ) assoc(a,v) = sum of weghts of all edges that touch A Ncut value small when we get two clusters wth many edges wth hgh weghts, and few edges of low weght between them Approxmate soluton for mnmzng the Ncut value : generalzed egenvalue problem. J. Sh and J. Malk, Normalzed Cuts and Image Segmentaton, CVPR, 1997 Source: Steve Setz Example results Normalzed cuts: pros and cons Pros: Generc f ramework, f lexble to choce of f uncton that computes weghts ( af f ntes ) between nodes Does not requre model of the data dstrbuton Cons: Tme complexty can be hgh Dense, hghly connected graphs many affnty computatons Solvng egenvalue problem Pref erence f or balanced parttons 4
Segments as prmtves for recognton Multple segmentatons B. Russell et al., Usng Multple Segmentatons to Dscover Objects and ther Extent n Image Collectons, CVPR 006 Slde credt: Lana Lazebnk Top-down segmentaton E. Borensten and S. Ullman, Class-specfc, top-down segmentaton, ECCV 00 A. Levn and Y. Wess, Learnng to Combne Bottom-Up and Top-Down Segmentaton, ECCV 006. Slde credt: Lana Lazebnk Top-down segmentaton Normalzed cuts Top-down segmentaton E. Borensten and S. Ullman, Class-specfc, top-down segmentaton, ECCV 00 A. Levn and Y. Wess, Learnng to Combne Bottom-Up and Top-Down Segmentaton, ECCV 006. Slde credt: Lana Lazebnk 5
Moton segmentaton Input sequence Image Segmentaton Moton Segmentaton Input sequence Image Segmentaton Moton Segmentaton A.Barbu, S.C. Zhu. Generalzng Swendsen-Wang to samplng arbtrary posteror probabltes, IEEE Trans. PAMI, August 005. Image groupng K. Grauman & T. Darrell, Unsupervsed Learnng of Categores from Sets of Partally Matchng Image Features, CVPR 006. Recap on groupng Segmentaton to fnd object boundares or mdlevel regons, tokens. Bottom-up segmentaton va clusterng General choces -- f eatures, af fnty functons, and clusterng algorthms Groupng also useful for quantzaton, can create new feature summares Texton hstograms f or texture wthn local regon Example clusterng methods K-means (and EM) Mean shf t Graph cut, normalzed cuts 6
Now: Fttng Want to assocate a model wth observ ed f eatures [Fg from Marszalek & Schmd, 007] For example, the model could be a lne, a crcle, or an arbtrary shape. Fttng: Man dea Choose a parametrc model to represent a set of features Membershp crteron s not local Can t tell whether a pont belongs to a gven model just by lookng at that pont Three man questons: What model represents ths set of features best? Whch of several model nstances gets whch feature? How many model nstances are there? Computatonal complexty s mportant It s nfeasble to examne every possble set of parameters and every possble combnaton of features Slde credt: L. Lazebnk Example: Lne fttng Why f t lnes? Many objects characterzed by presence of straght lnes Wat, why aren t we done just by runnng edge detecton? 7
Dffculty of lne fttng Extra edge ponts (clutter), multple models: whch ponts go wth whch lne, f any? Only some parts of each lne detected, and some parts are mssng: how to fnd a lne that brdges mssng evdence? Nose n measured edge ponts, orentatons: how to detect true underlyng parameters? Votng It s not f easble to check all combnatons of f eatures by f ttng a model to each possble subset. Votng s a general technque where we let the f eatures vote for all models that are compatble wth t. Cycle through features, cast votes for model parameters. Look for model parameters that receve a lot of votes. Nose & clutter f eatures wll cast v otes too, but ty pcally ther v otes should be nconsstent wth the majorty of good f eatures. Fttng lnes: Hough transform Gv en ponts that belong to a lne, what s the lne? How many lnes are there? Whch ponts belong to whch lnes? Hough Transform s a v otng technque that can be used to answer all of these questons. Man dea: 1. Record v ote f or each possble lne on whch each edge pont les.. Look f or lnes that get many v otes. 8
Fndng lnes n an mage: Hough space y b b0 x mage space m0 m Hough (parameter) space Connecton between mage (x,y ) and Hough (m,b) spaces A lne n the mage corresponds to a pont n Hough space To go from mage space to Hough space: gven a set of ponts (x,y), fnd all (m,b) such that y = mx + b Slde credt: Steve Setz Fndng lnes n an mage: Hough space y b y0 x0 x mage space m Hough (parameter) space Connecton between mage (x,y ) and Hough (m,b) spaces A lne n the mage corresponds to a pont n Hough space To go from mage space to Hough space: gven a set of ponts (x,y), fnd all (m,b) such that y = mx + b What does a pont (x0, y0) n the mage space map to? Answer: the solutons of b = -x0m + y0 ths s a lne n Hough space Slde credt: Steve Setz Fndng lnes n an mage: Hough space y y0 (x0, y0) (x1, y1) b x0 x mage space b = x1m + y1 m Hough (parameter) space What are the lne parameters f or the lne that contans both (x 0, y 0) and (x 1, y 1)? It s the ntersecton of the lnes b = x0m + y0 and b = x1m + y1 9
Image rows 9/3/015 Fndng lnes n an mage: Hough algorthm y b x mage space m Hough (parameter) space How can we use ths to f nd the most lkely parameters (m,b) f or the most promnent lne n the mage space? Let each edge pont n mage space vote f or a set of possble parameters n Hough space Accumulate v otes n dscrete set of bns; parameters wth the most v otes ndcate lne n mage space. Polar representaton for lnes Issues wth usual (m,b) parameter space: can take on nf nte v alues, undef ned f or v ertcal lnes. [0,0] y d Image columns x d : perpendcular dstance f rom lne to orgn : angle the perpendcular makes wth the x-axs xcos ysn d Pont n mage space snusod segment n Hough space Hough transform algorthm Usng the polar parameterzaton: xcos ysn d H: accumulator array (votes) Basc Hough transf orm algorthm d 1. Intalze H[d, ]=0. for each edge pont I[x,y] n the mage for = [ mn to max ] // some quantzaton d xcos ysn H[d, ] += 1 3. Fnd the value(s) of (d, ) where H[d, ] s maxmum 4. The detected lne n the mage s gven by d xcos ysn Tme complexty (n terms of number of v otes per pt)? Source: Steve Setz 10
Orgnal mage Canny edges Vote space and top peaks Showng longest segments found Impact of nose on Hough y d Image space edge coordnates x Votes What df f culty does ths present f or an mplementaton? 11
Impact of nose on Hough Image space edge coordnates Votes Here, ev ery thng appears to be nose, or random edge ponts, but we stll see peaks n the v ote space. Extensons Extenson 1: Use the mage gradent 1. same. for each edge pont I[x,y] n the mage = gradent at (x,y) d xcos ysn H[d, ] += 1 3. same 4. same (Reduces degrees of freedom) Extenson gve more votes for stronger edges Extenson 3 change the samplng of (d, ) to gve more/less resoluton Extenson 4 The same procedure can be used wth crcles, squares, or any other shape Extensons Extenson 1: Use the mage gradent 1. same. for each edge pont I[x,y] n the mage compute unque (d, ) based on mage gradent at (x,y) H[d, ] += 1 3. same 4. same (Reduces degrees of freedom) Extenson gve more votes for stronger edges (use magntude of gradent) Extenson 3 change the samplng of (d, ) to gve more/less resoluton Extenson 4 The same procedure can be used wth crcles, squares, or any other shape Source: Steve Setz 1
Hough transform for crcles Crcle: center (a,b) and radus r ( x a) ( y b) r For a f xed radus r, unknown gradent drecton b Image space Hough space a Hough transform for crcles Crcle: center (a,b) and radus r ( x a) ( y b) r For a f xed radus r, unknown gradent drecton Intersecton: most votes for center occur here. Image space Hough space Hough transform for crcles Crcle: center (a,b) and radus r ( x a) ( y b) r For an unknown radus r, unknown gradent drecton r? Image space a Hough space b 13
Hough transform for crcles Crcle: center (a,b) and radus r ( x a) ( y b) r For an unknown radus r, unknown gradent drecton r b Image space a Hough space Hough transform for crcles Crcle: center (a,b) and radus r ( x a) ( y b) r For an unknown radus r, known gradent drecton x Image space Hough space Hough transform for crcles For every edge pxel (x,y) : For each possble radus value r: For each possble gradent drecton : // or use estmated gradent at (x,y) a = x r cos() // column b = y + r sn() // row H[a,b,r] += 1 end end Tme complexty per edgel? Check out onlne demo : http://www.markschulze.net/java/hough/ 14
Example: detectng crcles wth Hough Orgnal Edges Votes: Penny Note: a df f erent Hough transf orm (wth separate accumulators) was used f or each crcle radus (quarters v s. penny ). Example: detectng crcles wth Hough Combned Orgnal detectons Edges Votes: Quarter Con fndng sample mages from: Vvek Kwatra Example: rs detecton Gradent+threshold Hough space (fxed radus) Max detectons Hemerson Pstor and Eduardo Rocha Costa http://rsbweb.nh.gov/j/plugns/hough-crcles.html 15
Example: rs detecton An Irs Detecton Method Usng the Hough Transform and Its Evaluaton for Facal and Eye Movement, by Hdek Kashma, Htosh Hongo, Kunhto Kato, Kazuhko Yamamoto, ACCV 00. Votng: practcal tps Mnmze rrelevant tokens frst Choose a good grd / dscretzaton Too fne? Too coarse Vote for neghbors, also (smoothng n accumulator array) Use drecton of edge to reduce parameters by 1 To read back w hch ponts voted for w nnng peaks, keep tags on the votes. Hough transform: pros and cons Pros All ponts are processed ndependently, so can cope wth occluson, gaps Some robustness to nose: nose ponts unlkely to contrbute consstently to any sngle bn Can detect multple nstances of a model n a sngle pass Cons Complexty of search tme ncreases exponentally wth the number of model parameters Non-target shapes can produce spurous peaks n parameter space Quantzaton: can be trcky to pck a good grd sze 16
Generalzed Hough Transform What f w e w ant to detect arbtrary shapes? Intuton: Dsplacement vectors x x x Ref. pont Model mage Novel mage x Vote space x Now suppose those colors encode gradent drectons Generalzed Hough Transform Defne a model shape by ts boundary ponts and a reference pont. p1 x a Model shape p Offlne procedure: At each boundary pont, compute dsplacement vector: r = a p. Store these vectors n a table ndexed by gradent orentaton. [Dana H. Ballard, Generalzng the Hough Transform to Detect Arbtrary Shapes, 1980] Generalzed Hough Transform Detecton procedure: For each edge pont: Use ts gradent orentaton to ndex nto stored table Use retrev ed r v ectors to v ote f or ref erence pont x xx xx p1 Novel mage Assumng translaton s the only transformaton here,.e., orentaton and scale are fxed. 17
Generalzed Hough for object detecton Instead of ndexng dsplacements by gradent orentaton, ndex by matched local patterns. tranng mage vsual codeword wth dsplacement vectors B. Lebe, A. Leonards, and B. Schele, Combned Object Categorzaton and Segmentaton wth an Implct Shape Model, ECCV Workshop on Statstcal Learnng n Computer Vson 004 Source: L. Lazebnk Generalzed Hough for object detecton Instead of ndexng dsplacements by gradent orentaton, ndex by vsual codew ord test mage B. Lebe, A. Leonards, and B. Schele, Combned Object Categorzaton and Segmentaton wth an Implct Shape Model, ECCV Workshop on Statstcal Learnng n Computer Vson 004 Source: L. Lazebnk Summary Groupng/segmentaton useful to make a compact representaton and merge smlar f eatures assocate features based on defned smlarty measure and clusterng objectve Fttng problems requre f ndng any supportng ev dence f or a model, ev en wthn clutter and mssng f eatures. assocate features wth an explct model Votng approaches, such as the Hough transform, make t possble to f nd lkely model parameters wthout searchng all combnatons of f eatures. Hough transform approach for lnes, crcles,, arbtrary shapes defned by a set of boundary ponts, recognton from patches. 18
Comng up Fttng w th deformable contours A s out, due n tw o w eeks 19