Targil 8 : Image warping. Forward warping. Motion Transformations and Image Warping (cont.) Automatic Image Alignment: Lucas Kanade (cont.

Transcription

1 Hebrew Uniersi mage Processing Hebrew Uniersi mage Processing Moion Transformaions and mage Warping - conine Targil 8 : Moion Transformaions and mage Warping con. Aomaic mage Alignmen: Lcas Kanade con. Opical Flow hp:// Man slides from Aleei fros ans See Seiz Man slides from See Seiz and Aleei fros mage warping Hebrew Uniersi mage Processing Forward warping Hebrew Uniersi mage Processing T f g T f g Gien a coordinae ransform h and a sorce image f how do we compe a ransformed image g ft? Send each piel f o is corresponding locaion T in he second image Q: wha if piel lands beween wo piels? 3 4 Forward warping Hebrew Uniersi mage Processing nerse warping Hebrew Uniersi mage Processing T f g T - f g Send each piel f o is corresponding locaion T in he second image Q: wha if piel lands beween wo piels? A: disribe color among neighboring piels Ge each piel g from is corresponding locaion T - in he firs image Q: wha if piel comes from beween wo piels? Known as splaing 5 6

2 nerse warping Hebrew Uniersi mage Processing Bilinear inerpolaion Hebrew Uniersi mage Processing Sampling a f: T - f g Ge each piel g from is corresponding locaion T - in he firs image Q: wha if piel comes from beween wo piels? A: nerpolae color ale from neighbors neares neighbor bilinear bicbic 7 8 Hebrew Uniersi mage Processing Aomaic mage Alignmen direc reminder from las lecre Hebrew Uniersi mage Processing Wh do we wan o esimae moion? Los of ses Correc for camera jier sabilizaion Align images mosaics 9 0 Hebrew Uniersi mage Processing Moion Consrains grascale images Hebrew Uniersi mage Processing Local Talor approimaion in D: T ' ' f h f f ' h ff h f Le s look a hese consrains more closel brighness consanc: Q: wha s he eqaion? small moion: and are less han piel sppose we ake he Talor series epansion of : f Local Talor approimaion in D for images: f f f f

3 3 3 Hebrew Uniersi mage Processing We wan o find ha minimize he moion error The consan brighness consrain [ ] Using Talor approimaion 4 Hebrew Uniersi mage Processing Wriing i in simple form : The deriaie of image. : The deriaie of image. : The difference beween he images -. The fncion o minimize oer he image The final LK eqaion ] [ 5 Hebrew Uniersi mage Processing Translaion moion The final eqaions for finding he ranslaion moion parameers: 0 0 A b 6 Hebrew Uniersi mage Processing nsead of minimizing for consan We sbsie as a fncion of heir locaion in he picre: aking in consideraion he moion model How can we represen and in his model? : Beond Translaion: A simple modificaion o he same idea Θ Θ Θ Θ ' ' 0 0 cos sin sin cos Θ Θ 7 Hebrew Uniersi mage Processing Reisiing he small moion assmpion s his moion small enogh? Probabl no i s mch larger han one piel nd order erms dominae How migh we sole his problem? 8 Hebrew Uniersi mage Processing Redce he resolion!

4 LK Algorihm : sing pramids Hebrew Uniersi mage Processing Hebrew Uniersi mage Processing Coarse-o-fine opical flow esimaion.5 piels.5 piels 5 piels rn ieraie L-K warp & psample rn ieraie L-K... image H 0 piels image image J image Gassian pramid of image Gassian pramid of image 9 Gassian pramid of image Gassian pramid of image 0 Hebrew Uniersi mage Processing LK Algorihm for image ranslaion A Creae pramids P P for he images. niialize e.g. 00 For pramid leels k Updae mlipl b Compe he deriaies and mari A erae nil conergence P k -P k compe he ecor b sole he eqaions pdae d d. b d A b d Opical Flow Hebrew Uniersi mage Processing Defining he opical flow problem The aperre problem Opical flow eqaion gradien descen on he error fncion Lkas-Kanade flow eraie refinemen Corse o Fine solion pramids Mli-resolion Lcas Kanade Algorihm Hebrew Uniersi mage Processing Moion esimaion: Opical flow Hebrew Uniersi mage Processing Problem definiion: opical flow We hae seen how o esimae he image of he enire image How o esimae piel moion from image H o image? Wh is i sefl Deph 3D reconsrcion Moion deecion - racking moing objecs Compression... now we wan o esimaing moion of each piel separael 3 4 4

5 Wha is Opical Flow? Hebrew Uniersi mage Processing Hebrew Uniersi mage Processing Problem definiion: opical flow p p p 3 Opical Flow r r r 3 p 4 r 4 { p } i { r Veloci ecors i } How o esimae piel moion from image H o image? Sole piel correspondence problem gien a piel in H look for nearb piels of he same color in Common assmpion: The appearance of he image paches do no change brighness consanc r pi pi i 5 Ke assmpions color consanc: a poin in H looks he same in For grascale images his is brighness consanc small moion: poins do no moe er far This is called he opical flow problem 6 Classes of Techniqes Hebrew Uniersi mage Processing Direc mehods Hebrew Uniersi mage Processing Feare-based mehods rac isal feares corners ered areas and rack hem oer mliple frames Sparse moion fields b possibl robs racking Siable especiall when image moion is large 0-s of piels gradien descen on he error fncion Same assmpion we sed in finding global moion image alignmenn Direc-mehods Direcl recoer image moion from spaio-emporal image brighness ariaions Moion ecors direcl recoered wiho an inermediae feare moion calclaion Dense moion fields b more sensiie o appearance ariaions Siable for ideo and when image moion is small < 0 piels 7 8 Hebrew Uniersi mage Processing Opical flow consrains grascale images Opical flow eqaion Hebrew Uniersi mage Processing Q: how man nknowns and eqaions per piel? Le s look a hese consrains more closel brighness consanc: Q: wha s he eqaion? niiel wha does his consrain mean? The componen of he flow in he gradien direcion is deermined The componen of he flow parallel o an edge is nknown small moion: and are less han piel sppose we ake he Talor series epansion of : This eplains he Barber Pole illsion hp://

6 Normal Flow Hebrew Uniersi mage Processing Aperre problem Hebrew Uniersi mage Processing We ge a mos Normal Flow wih one poin we can onl deec moemen perpendiclar o he brighness gradien. Solion is o ake a pach of piels Arond he piel of ineres. 3 3 Slide from Michael Black CS Aperre problem Hebrew Uniersi mage Processing Aperre problem Hebrew Uniersi mage Processing The Aperre Problem Le M Algorihm: A each piel compe U b soling T and Hebrew Uniersi mage Processing b MU b Geing more qaions Hebrew Uniersi mage Processing How o ge more eqaions for a piel? Basic idea: impose addiional consrains mos common is o assme ha he flow field is smooh locall one mehod: preend he piel s neighbors hae he same» f we se a 55 window ha gies s 5 eqaions per piel! M is singlar if all gradien ecors poin in he same direcion e.g. along an edge of corse riiall singlar if he smmaion is oer a single piel or here is no ere i.e. onl normal flow is aailable aperre problem Corners and ered areas are OK

7 Lkas-Kanade flow Prob: we hae more eqaions han nknowns Hebrew Uniersi mage Processing Condiions for solabili Opimal saisfies Lcas-Kanade eqaion Hebrew Uniersi mage Processing Solion: sole leas sqares problem minimm leas sqares solion gien b solion in d of: The smmaions are oer all piels in he K K window This echniqe was firs proposed b Lkas & Kanade When is This Solable? A T A shold be inerible igenalesλ and λ of A T A shold no be oo small A T A shold be well-condiioned λ / λ shold no be oo large λ larger eigenale A T A is solable when here is no aperre problem 38 Local Pach Analsis Hebrew Uniersi mage Processing dge Hebrew Uniersi mage Processing large gradiens all he same largeλ small λ 40 Low ere region Hebrew Uniersi mage Processing High ered region Hebrew Uniersi mage Processing gradiens hae small magnide gradiens are differen large magnides smallλ small λ largeλ large λ 4 4 7

8 Obseraion Hebrew Uniersi mage Processing This is a wo image problem BUT Can measre sensiii b js looking a one of he images! This ells s which piels are eas o rack which are hard er sefl laer if o wan o do feare racking... rrors in Lkas-Kanade Hebrew Uniersi mage Processing Wha are he poenial cases of errors in his procedre? Sppose A T A is easil inerible Sppose here is no mch noise in he image When or assmpions are iolaed Brighness consanc is no saisfied The moion is no small A poin does no moe like is neighbors window size is oo large wha is he ideal window size?