Estimating Human Motion

Size: px

Start display at page:

Download "Estimating Human Motion"

Earl Reeves
5 years ago
Views:

1 Estmatng Human Moton Leond Sgal Mchael Isard * Stefan Roth Sd Bahta and Department of Computer Scence, Brown Unversty * Mcrosoft Research

2 Percepton * nference from uncertan, ambguous, ncomplete, nosy data. * combnes measurements wth a pror knowledge.

3 Capturng Humans n Moton If a shadow s a two dmensonal projecton of the three-dmensonal world, then the three-dmensonal world as we know t s the projecton of the four dmensonal unverse. Marcel Duchamp MARCEL DUCHAMP (1912) Nude Descendng A Starcase

4 Capturng Humans n Moton E T I E N N E -J U L E S M A R E Y, 1882 chronophotograph.

5 Capturng Humans n Moton Loss of depth and moton n projecton to 2D mages. EADWEARD MUYBRIDGE, Multple cameras. E T I E N N E -J U L E S M A R E Y, Marker-based trackng

6 Artculated Body Model Knematc tree: Marr&Nshhara 78 0 [ τ [ θ [ θ 1,0 x, τ, τ 0, g 0, g 0, g x y z, θ, θ 1,0 y, θ, θ 1,0 z 1 2,1 [ ] 2 θ x 0, g 0, g 0, g x y z ] ] ] Also model angular veloctes ~ 50D space Represent a pose at tme t by a vector of these parameters: X t

7 Mocap Today Cameras CMU Mocap lab. 2003

8 Mocap Today

9 Mocap n the Wld Humans n captvty Humans n ther natural habtat

10 Detectng and Trackng People * Where are the people? * What are ther poses? * How are they movng? * What are they dong?

11 Detecton: The Pure ML Approach f f f M f N Classfer Person/Not-person Sngle mage

Detecton: The Pure ML Approach f f f M 1 2 3 f N

12 Detecton: The Pure ML Approach f f f M f N Mappng 3D pose? Sngle mage Hugely hgh dmensonal. Ambguous.

13 Markerless Moton Capture

14 Markerless Moton Capture Fnd the pose X t * No specal clothng * Unknown, cluttered, envronment * Incremental estmaton such that the projecton matches the mage data.

15 Markerless Moton Capture Fnd the pose X t Generate and Test. Computng the lkelhood of a pose s easy. such that the projecton matches the mage data.

16 Markerless Moton Capture Fnd the pose X t Searchng over all possble poses s hard. such that the projecton matches the mage data.

17 Markerless Moton Capture Fnd the pose X t such that the projecton matches the mage data.

18 Problems The appearance/sze/shape of people can vary dramatcally (hgh-d space). Occluson and partal vews. Underlyng structure (bones and jonts) s unobservable (obscured by muscle, skn, clothng).

19 Problems Loss of 3D n 2D projecton Unusual poses Self occluson Low contrast

20 Problems Multple people and occluson leads to ambguty. Movng cameras & complex changng backgrounds.

21 Problems Accdental algnment Moton blur. (nothng to match)

22 Brghtness Inconstancy

23 Requrements 1. Represent uncertanty and multple hypotheses. 2. Model complex knematcs of the body. Correlatons between jonts and over tme. 3. Explot rch set of mage cues n a robust fashon. 4. Integrate nformaton over tme. The recovery of human pose/moton s fundamentally a problem of nference from ambguous and uncertan measurements.

24 State of the Art Deutscher, North, Bascle, & Blake 99 Sdenbladh, Black and Fleet, 00 Cham and Rehg 99 Smnchsescu &Trggs 01

25 Problems * Search n a huge-dmensonal space: 30+ dmensons. -Non-Gaussan, mult-modal, posteror p(x t ). * Multple cameras or strong prors needed for relable trackng. * Manual ntalzaton. * Requre good slhouettes (background subtracton). * Unprncpled collecton of so-so mage lkelhood models. * Brttle can t recover when they get lost.

26 Herarchy of Representatons Brooks, ACRONYM, 1981.

27 Approach Stuff: Rch set of flters appled to mages Shouters: Smple feature (part) detectors * faces, lmbs, feet, hands Glue: Loose-lmbed model * local constrants n space and tme nference Thngs: Artculated body model * expresses model knowledge about humans and how they move cf recent work n object recognton

28 Shouters p( X Yt ) Fndng a person s hard but fndng plausble lmbs s much easer.

29 Shouters p( X Yt ) Bottom-up processes * provde proposal dstrbutons they shout

30 Shouters * Varous ML technques (egen-x, SVM, RVM, Adaboost, etc). * Relatvely low dmensonal problem. * Accuracy not crtcal.

31 Glue: Loose-lmbed People Loose-lmbed body (graphcal model) Push puppet toy

32 Glue: Loose-lmbed People Loose-lmbed body (graphcal model) From Ballard and Brown Pctoral structures Fschler and Elschlager 73

33 Glue: Loose-lmbed People Soft constrants between lmbs (messages). Pose estmaton as nference n a graphcal model (Belef Propagaton). Allows bottom up ntalzaton. Deals well wth unobserved lmbs. More recently (2D, dscretzed): * Felzenszwalb & Huttenlocher 00, Ronfard et al 02, Forsyth et al

34 Glue: Loose-lmbed People X * = poston & orentaton * 6D dscretzaton not practcal

35 Inference: Belef random vector (poston and orentaton of node ) local evdence observatons (mage and flter responses) p ( X Y) = α λ( Y X ) m ( X ) neghbors of node k A k ncomng messages

36 Inference: Belef local evdence p ( X Y) = α λ( Y X ) m ( X ) k A k Learn ths from tranng data.

37 Messages message from node to node j local evdence m j ( X ) = α ψ ( X, X ) λ( X ) m ( X ) dx j j j k A \ j k probablty of X j condtoned on X neghbors of ( spatal or temporal pror), not ncludng j ncomng messages at ths s the hard part f thngs aren t Gaussan

$dx j j j k A \ j k probablty of X j condtoned on X ( spatal or temporal$

38 Messages Learn ths from tranng data m j ( X ) = α ψ ( X, X ) λ( X ) m ( X ) dx j j j k A \ j k probablty of X j condtoned on X ( spatal or temporal pror)

39 Recpe for Fndng People 1. Learn local evdence * Model appearance. 2. Learn spatal and temporal prors * How jonts connect and move. 3. Search * Develop an effectve nference algorthm for message passng. * Explot shouters as a proposal dstrbuton.

40 Recpe for Fndng People 1. Learn local evdence * Model appearance. 2. Learn spatal and temporal prors local evdence * How jonts connect and move. 3. Search p ( X Y) = α λ( Y X ) m ( X ) k * Develop an effectve nference algorthm for k A message passng. * Develop bottom-up feature detectors.

41 Ground Truth (MoVd) 3D Vcon data projected onto vdeo streams.

42 Towards a Rgorous Lkelhood Learn from examples: λ( Y X ) = p( f, f2, K, 1 f n X )

43 Example Edge Lkelhood p(edge flter responses lmb edge locaton and orentaton) Deutscher, Blake & Red, CVPR 00

44 Human-Specfc Image Statstcs Frst dervatve flter f flter response p( f X j Non-Gaussan margnal statstcs. )

45 Naïve Lkelhood 1D margnals λ( Y X = = j ) p ( f1,..., fk X j ) p( f X j ) Not a great assumpton: Roth, S., Sgal, L., Black, M. J., Gbbs lkelhoods for Bayesan trackng, CVPR 04

46 Recpe for Fndng People 2. Learn spatal and temporal prors * How jonts connect and move. m j ( X ) = α ψ ( X, X ) λ( X ) m ( X ) dx j j j k A \ j k probablty of X j condtoned on X ( spatal or temporal pror)

47 Learned Condtonals * represented by a mxture of Gaussans (learned from mocap data).

48 Spatal and Temporal * Also learn condtonals backwards and forwards n tme. * Introduces loops

49 3. Search Recpe for Fndng People * Develop an effectve nference algorthm for message passng. Non-Gaussan 6D contnuous m j ( X ) = α ψ ( X, X ) λ( X ) m ( X ) dx j j j k A \ Product of mxtures Mxture of Gausans * Gbbs sampler j k

50 Illustraton of the message product.

51 Algorthm Hghlghts m j ( X ) = α ψ ( X, X ) λ( X ) m ( X ) dx j j j k A \ j k Non-parametrc Belef Propagaton: (Isard CVPR 03, Sudderth et al CVPR 03) * Gbbs sample from message product (Sudderth et al) * Monte Carlo ntegraton (lke partcle flterng) * mportance sample from a proposal dstrbuton * ncludng bottom-up shouters * propagate through potental functon

54 Automatc Intalzaton and Trackng Sgal et al, Trackng loose-lmbed people, CVPR 04

55 Problems and Next Steps Loose-lmbed model Dffcult to model self occluson. Dffcult to model hgh-level knowledge about human poses and motons. Full-body model Add a new knematc-tree node. Loose-lmbed nodes are combned to nfer 3D pose Hgh level constrants are mposed top-down.

56 Summary We have tackled four mportant parts of the problem: lkelhood pror search 1. Probablstcally modelng human appearance. 2. Learn relatonshps between lmbs from tranng data weak model. 3. Bayesan nference usng non-parametrc belef propagaton. These general tools are wdely applcable.

57 Lessens for Optcal Flow Need rcher prors * Flow methods have used week prors on spatal and temporal coherence don t model the rch structure of movng scenes well. Need rcher lkelhoods * The probablty of appearance change should be learned from examples. * Model condtonal dependences among flter responses Need new computatonal models * Belef propagaton for spatal/temporal coherence wth non-gaussan lkelhoods and prors.

58 What s stll far off? Moton nterpretaton. Heder&Smmel, 1944 * Here estmaton problem s trval but explanaton s hard. Move by Emre Ylmaz

59 Graduate Students at Brown: Leond Sgal, Computer Scence Stefan Roth, Computer Scence Alex Balan, Computer Scence Sdharth Bhata, Engneerng Collaborators Undergrads: Jonathan Bankard, & Davd Erckson Mocaplab Mchael Isard, Mcrosoft Research Alumn: Hedvg Sdenbladh, Swedsh Defense Research Inst. Davd Fleet, Unversty of Toronto Ben Sgelman, Brown Unversty (now Google).

Inferring 3D People from 2D Images

Inferring 3D People from 2D Images Department of Computer Science Brown University http://www.cs.brown.edu/~black Collaborators Hedvig Sidenbladh, Swedish Defense Research Inst. Leon Sigal, Brown University