Prof. Feng Lu Sprng 2017 ttp://www.cs.pd.edu/~flu/courses/cs510/ 05/24/2017
Last me Compostng and Mattng 2
oday Vdeo Stablzaton Vdeo stablzaton ppelne 3
Orson Welles, ouc of Evl, 1958 4
Images courtesy Peter Sand and Flckr user Carles W. Brown 5
6
7
[Lu et al. 09] 8
Input Stablzaton result 9
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output Feature trajectores
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output Feature trajectores
Brgtness Constancy Equaton: Lnearzng te rgt sde usng aylor epanson: ), ( 1),, ( ),, ( ), ( t y y v y u I t y I ), ( ), ( ),, ( 1),, ( y v I y u I t y I t y I y Feature rackng I(,y,t 1) I(,y,t) 0 t y I v I u I Hence, B. Lucas and. Kanade. An teratve mage regstraton tecnque wt an applcaton to stereo vson. In Proceedngs of te Internatonal Jont Conference on Artfcal Intellgence, pp. 674 679, 1981.
Spatal Coerence Constrant I u I y v It 0 How many equatons and unknowns per pel? One equaton, two unknowns How to get more equatons for a pel? Spatal coerence constrant: pretend te pel s negbors ave te same (u,v) 16
Solvng te rackng Problem Least squares problem: Wen s ts system solvable? Wat f te wndow contans just a sngle stragt edge? B. Lucas and. Kanade. An teratve mage regstraton tecnque wt an applcaton to stereo vson. In Proceedngs of te Internatonal Jont Conference on Artfcal Intellgence, pp. 674 679, 1981.
Condtons for Solvablty Bad case: sngle stragt edge
Lucas-Kanade Flow Least squares problem: Soluton gven by e summatons are over all pels n te wndow B. Lucas and. Kanade. An teratve mage regstraton tecnque wt an applcaton to stereo vson. In Proceedngs of te Internatonal Jont Conference on Artfcal Intellgence, pp. 674 679, 1981.
Lucas-Kanade Flow Recall te Harrs corner detector: M = A A s te second moment matr We can fgure out weter te system s solvable by lookng at te egenvalues of te second moment matr e egenvectors and egenvalues of M relate to edge drecton and magntude e egenvector assocated wt te larger egenvalue ponts n te drecton of fastest ntensty cange, and te oter egenvector s ortogonal to t
Interpretng te egenvalues Classfcaton of mage ponts usng egenvalues of te second moment matr: 2 Edge 2 >> 1 Corner 1 and 2 are large, 1 ~ 2 1 and 2 are small Flat regon Edge 1 >> 2 1
Unform Regon gradents ave small magntude small 1, small 2 system s ll-condtoned
Edge gradents ave one domnant drecton large 1, small 2 system s ll-condtoned
Hg-teture or Corner Regon gradents ave dfferent drectons, large magntudes large 1, large 2 system s well-condtoned
Feature trackng So far, we ave only consdered feature trackng n a par of mages If we ave more tan two mages, we can track feature from eac frame to te net Gven a pont n te frst mage, we can n prncple reconstruct ts pat by smply followng te arrows
rackng over Many Frames Select features n frst frame For eac frame: Update postons of tracked features Dscrete searc or Lucas-Kanade (or a combnaton of te two) ermnate nconsstent tracks Compute smlarty wt correspondng feature n te prevous frame or n te frst frame were t s vsble Fnd more features to track
S-omas Feature racker Fnd good features usng egenvalues of secondmoment matr Key dea: good features to track are te ones wose moton can be estmated relably From frame to frame, track wt Lucas-Kanade s amounts to assumng a translaton model for frame-to-frame feature movement Ceck consstency of tracks by affne regstraton to te frst observed nstance of te feature Affne model s more accurate for larger dsplacements Comparng to te frst frame elps to mnmze drft J. S and C. omas. Good Features to rack. CVPR 1994.
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output Frames 2 2D moton model (omograpy)
Homograpy 1 1 33 32 31 23 22 21 13 12 11 y y epand
Fttng a omograpy Equaton for omograpy: 1 1 33 32 31 23 22 21 13 12 11 y y 0 y y y 3 2 1 1 2 3 1 2 3 1 0 0 0 0 3 2 1 y y 3 equatons, only 2 lnearly ndependent
Drect lnear transform H as 8 degrees of freedom (9 parameters, but scale s arbtrary) One matc gves us two lnearly ndependent equatons Four matces needed for a mnmal soluton (null space of 89 matr) More tan four: omogeneous least squares 0 0 0 0 0 3 2 1 1 1 1 1 1 1 n n n n n n y y 0 A
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output
Moton Plan S t N t t G, were t t j j Image courtesy: Matsusta et al. 06 33
34
35
36
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output 3D reconstructon va structure from moton Voodoo Camera racker (ttp://www.dglab.un-annover.de)
Voodoo Camera racker (ttp://www.dglab.un-annover.de)
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output Novel vew syntess va mage based renderng
Unstructured lumgrap renderng [Bueler et al. 01] 41
42 F Lu, M Glecer, H Jn, A Agarwala. Content-preservng warps for 3D vdeo stablzaton, SIGGRAPH 09
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output Novel vew syntess mage based renderng
Input rajectory Estmaton Moton Model Estmaton Moton Plan Vdeo ransform Output Our metod for novel vew syntess One nput frame One output frame
45
46
nput frame and ponts 47
nput frame and ponts output ponts 48
nput frame and ponts output frame 49
50
51
52
53
54
[Igaras et al. 05] 55
Input Vsual salency map [Itt et al. 99] Vsual salency: te dstnct subjectve perceptual qualty wc makes some tems n te world stand out from ter negbors and mmedately grab our attenton from [Itt 07] 56
Input Output 57
Input Output teture mappng [Srley et al. 2005] 58
Grd mes & ponts Output 59
Net me More on Vdeo Stablzaton 3D Vdeo Stablzaton Subspace Vdeo Stablzaton 60