Learning Depth from Single Still Images: Approximate Inference 1

Transcription

1 Learnng Depth from Sngle Stll Images: Approxmate Inference 1 MS&E 211 course project Ashutosh Saxena, Ilya O. Ryzhov Channng Wong, Janln Wang June 7th, In ths report, Saxena, et. al. [1] somethng wll mean the work was not one specfcally for the class; an we an our wll mean the work was one by four stuents Ashutosh, Ilya, Channng or Janln specfcally for ths course.

2 I. INTRODUCTION In ths project, we conser the problem of nferrng epth-map 2 from a sngle monocular mage. There are two phases n ths: (a) Learnng on 250 mages, Saxena et. al. [1] use supervse learnng to prect the epth-map as a functon of the mage; an (b) Inference on fferent 64 mages, for whch Saxena et. al. [1] use a nave metho for nference, whch was very slow an oes not scales well to epth-maps of large sze. We are gven correct epth-map 3 to compare the performance of the algorthm, hence, urng nference phase, the error of the algorthm s measure as average fference between correct groun truth epth-map, an nferre epth-map from our algorthm. The nference problem can be reuce to a norm-mnmzaton problem. Saxena et. al. [1] formulate t as a large-scale lnear optmzaton problem that s extremely tmeconsumng to solve. For MS&E211, we (a) Formulate the 1-norm mnmzaton as a LP wth half the number of varables use n [1]. (b) Bult a mn-cost moel to nfer a approxmate soluton, whch has a upper-boun on error from the correct groun-truth soluton. Ths soluton can also act as a ntal soluton for LP n (a), thus reucng the run tme sgnfcantly. I. Backgroun [1] Ths secton gves backgroun for the problem, one n Saxena, et. al. [1]. Recoverng 3-D epth from mages s a basc problem n computer vson, an has mportant applcatons to robotcs, scene unerstanng an 3-D reconstructon. Humans perceve epth usng many fferent epth cues, such as bnocular vson, as well as a number of monocular cues. In the past, most work on 3-D reconstructon has focuse on bnocular vson (stereopss) an very goo stereovson systems have been esgne. In ths project, we look at how monocular cues from a sngle mage can be use to estmate epth an evelop practcal moel an algorthms for computaton. Estmatng epth from a sngle mage requres a sgnfcant amount of pror knowlege about the envronment an global structure of the mage, snce there s an ntrnsc ambguty between local mage features an epth varatons. 2 Depthmap means epth at each pxel n the mage, hence s large matrx. 3 Saxena, bult a custom-laser scanner to get correct groun-truth epths.

3 Our approach s base on the capturng epths an relatonshps between epths at fferent spatal scales usng Markov Ranom Fels (MRFs), whch a workhorse of machne learnng an have been successfully apple to numerous applcatons. Usng the exstng tranng set, the MRF s scrmnatvely trane to prect epth. In other wors, rather than moelng the jont strbuton of mage features an epths, we moel the posteror strbuton of the epths gven the mage features an sparty. Our moel uses L 2 (Gaussan) terms n the MRF nteracton potentals, an captures epths an nteracton between epths at multple spatal scales. B) Vsual cues for epth percepton Humans have an amazng ablty to juge epth from a sngle monocular mage. Ths s one usng monocular cues such as texture varatons, texture graents, occluson, known object szes, haze, an efocus etc. Many objects texture appears fference epenng on the stance to t. Texture graents, whch capture the strbuton of the recton of eges, also help to ncate epth. Most of the monocular cues are contextual nformaton, n the sense that they are global propertes of an mage an cannot be nferre from small mage patches. For example, occluson cannot be etermne f we look at just a small porton of an occlue object. Although local nformaton such as the texture an colors of a patch can gve some nformaton about ts epth, ths s usually nsuffcent to accurately etermne ts absolute epth. One nees to look at the overall organzaton of the mage to etermne epths. Most of the monocular cues are memory base. For example, humans remember that a structure of a partcular shape s a bulng, sky s blue, grass s green, an so on. These cues help to prect epth n envronments smlar to those they have seen before. C) Monocular features In our project, we ve the mage nto small patches, an estmate a sngle epth value for each patch. We use two types of features: absolute epth features, use to estmate the absolute epth at a partcular patch, an relatve features, use to estmate relatve epths (magntue of the fference n epth between two patches). Gven some patch n the mage I(x, y), the sum square energy base the local cues, e.g., texture varatons, texture graents, an color are compute. D) Probablstc moel Our learnng moel s: II. LINEAR PROGRAMMING REDUCTION

4 where M s the total number of patches n the mage; x s the epth feature vector for the patch ; θ an λ are the parameters of the moels; Z s the normalzaton constant; an s the epth of the -th patch. Refer to [1] for more etals. Ths looks lke a non-lnear programmng. However, after takng logarthm, the problem can be converte to a lnear optmzaton problem. II. Lnear Programmng Reucton for MS&E211 In ths secton, we wll explan n etal how the above problem can be reuce to a lnear program an bul a mn-cost moel to prove better approxmate solutons, whch also act as better ntal solutons n case exact nference s esre. A) The essence of the problem After all the prelmnares an mathematcal ervatons for maxmum lkelhoo nference, we arrve at the followng optmzaton problem 4 : mn α m + c j N ( ) s.t. l g. j j In ths problem, the ecson varables are a vector of sze represent the estmate epths at each pxel. We woul lke to make our estmates reasonably close to some target values m, as well as to each other. The values m represent the results of localze epth estmaton at every pxel. Ths estmaton s performe earler n the epth estmaton algorthm. For our purposes here, we wll treat the values m as gven constants. The values α represent the costs of mssng these target values. Smlarly, the values c j represent the costs of makng the epth estmates at two ajacent pxels fferent from each other. Notce that our objectve functon only mposes costs on the fferences between ajacent epth values; here, j N() means that j s a neghbor of, or pxel j s next to pxel n the pcture. Although ths problem oes not look lnear, t can be turne navely t nto a lnear program by ntroucng some artfcal varables nave, as one n [1]: Ths lnear program can be solve by stanar nteror-pont algorthms, usng = m as ntal solutons. But solvng t n ths way takes too much tme. Even for a small epth-map of 80x60 pxels, the solver woul run for several mnutes (~200 secons.) 4 Ths oes not nclues mult-resoluton epths as n [1] for the purpose of keepng t smple to explan. The mn-cost was run on sngle-resoluton moel as above, however, the actual LP mnmzaton erve for MS&E211 n half the varables was o

5 Total number of nequalty constrants = 4*6897 (beses lower an upper boun) The appearance of ths problem suggests a mn-cost flow structure. The pcture tself can be vewe as a graph, n whch nvual pxels are noes, an two noes are connecte by eges f they are neghbours n the sense of the objectve functon. We are tryng to set a number of varables, subject to upper bouns (capactes) an lower bouns, n such a way as to mnmze a lnear objectve functon. In the followng sectons, we wll outlne the way n whch we use the graph structure of the problem to obtan goo epth estmates by solvng a number of small mn-cost problems. B) Better reucton to LP for exact nference For MS&E211, we reuce the problem LP problem gven n prevous secton as (to be complete)

6 Total number of nequalty constrants = 2*6897 (beses lower an upper boun) Therefore, we mprove the mplementaton of LP as a part of MS&E211 reucng the effectve lnear program sze n half, whch was solve my Matlab/Mosek. C) The mn-cost flow moel We wsh to somehow turn our optmzaton problem nto a mn-cost flow problem. It s clear that the varables have to represent flow n such a problem, snce these are the values that we wsh to set. Frst, we observe that the objectve functon n our lnear program contans two components: the cost ncurre when a epth estmate evates from a gven target value, an the cost ncurre when two ajacent epth estmates evate from each other. It s not har to solve these problems separately. For nstance, to mnmze the quantty α m, we can use the followng structure: l g cos t = 0 cost = α capacty = eman= m cost = α capacty = One of the eges comng nto the eman noe represents the value an has zero cost, but lmte capacty. The other ncomng ege, as well as the outgong ege, has nfnte capacty but mposes a cost α. These two eges represent the fference m. If > m, the excess wll go out along the outgong ege; f < m, the remaner wll come through the other ncomng ege. We can create one such structure for every pxel, an a global sources an snks as necessary to balance out total supply an eman. Smlarly, to mnmze c j j for two ajacent pxels, we can use ths structure: l g cos t = 0 cos t = cj capacty = l eman= 0 cos t = cj capacty = g cos t = 0 j

7 It works lke the prevous one, except now, the ncomng flow s not eaten up as eman, but sent out as j. The fference between the two s mae up by the two other eges that mpose cost. Taken nvually, these two mechansms are correct but also useless, because the ffculty of the problem arses n the nterplay between the two kns of costs across the whole graph. Somehow, we nee to moel the whole pcture usng the mechansms. Ths s precsely the man ffculty that we encountere n moelng the problem. Smply put, we beleve that t s mpossble to exten these mechansms to the whole graph. The reason for ths s that we have no way of recoverng the value of any once we make one comparson wth another value. Conser a typcal pxel n our graph. The epth estmate for that pxel shoul be reasonably close, not only to m, but also to four fferent values j, one for each neghbour. However, f we use our mn-cost moel to compare an m, we lose the value of. It gets eaten up as eman at one noe, an there s no way to recover t afterwars. Smlarly, f we compare an any j, we have no way of recoverng to compare t agan wth another j. However, t s possble to combne the above mechansms to solve a much more smplfe problem wth only two ajacent noes: 2 1 The two vertcal eges wth zero cost n the mle of the graph represent the varables 1, 2. All the horzontal eges represent excess. The top an bottom row of eges

8 represent 2 m2 an 1 m1, as we explane prevously, an the mle row represents 1 2. The last ege gong from the source to the snk s use to balance the total supply an eman n the graph. The supply an eman values of the source an the snk can be ajuste accorngly (they shoul just be large enough to ensure that the mn-cost soluton s correct for our problem). In fact, we can see from ths graph that t woul be mpossble to exten ths moel to nclue, say, another epth value 3 ajacent to 1. The value of 1 s lost after the comparson wth 2. However, the values of 1, 2 obtane through ths mn-cost problem are clearly more useful to us than our ntal guesses = m, because at least they take nto conseraton some of the ajacency costs. D) The mn-cost flow approxmaton We coul obtan a set of feasble epth estmates by ong the followng. For every two ajacent pxels, j, we solve the mn-cost flow problem escrbe n the prevous secton to obtan values of, j. Thus, we obtan four fferent values of for the typcal pxel. Now, we can just average them together to obtan one feasble value. Obvously, s not necessarly the optmal value n our orgnal problem. However, we argue that t s better than = m, because ths average oes conser the neghbours of the pxel, f not smultaneously. Then, we can use these newly obtane averages as new, mprove ntal solutons for an nteror-pont algorthm, or take them by themselves as a quck estmate of the epths wthout ong further optmzaton. III. RESULTS We teste the usefulness of the averages n both scenaros, as ntal solutons for the usual nteror-pont metho, an as estmates n ther own rght, by runnng the entre epth estmaton algorthm wth the mn-cost aton on 64 fferent sample pctures for whch the true epth values are known. Frst of all, we foun that the averages are always more accurate than the guesses = m, sometmes much more so. The Euclean stance between approxmaton an optmal LP soluton was about ½ to 1/3 of the stance between m an optmal LP soluton. That mn-cost soluton s always closer to optmal LP soluton than m can be prove theoretcally also. Next, we foun that usng averages as ntal nteror-pont solutons oes not apprecably reuce the runnng tme of the nteror-pont algorthm, even though they are the optmal soluton than the guesses. However, the tme requre to run all the mn-cost flow problems an obtan the averages s extremely small compare to the full nteror-pont algorthm. For 64 pctures usng monocular magng, the mn-cost problems

9 take just 0.35 secons on average per mage, where the full algorthm takes sx secons per mage. For the more complcate bnocular magng, the fference s even greater: the mn-cost algorthm takes 4.2 secons per mage where the full algorthm takes 220 (almost four mnutes). Wth such a fference n runnng tme, t makes sense to use the mn-cost approxmaton as a stan-alone epth estmate n an of tself for certan applcatons n whch tme s of the essence. In fact, the accuracy of the mn-cost approxmaton s enough to justfy such applcatons, as we can see from the followng tables. The mn-cost estmate s less accurate than the optmal soluton acheve by the nterorpont algorthm, but t s not substantally less accurate. There are many applcatons n whch t s acceptable to sacrfce some accuracy an thereby gan n runnng tme. For nstance, a walkng robot nees to be able to make quck epth estmates base on what t sees. It s better to have a goo, fast approxmaton n four secons than to wat four mnutes for a fully accurate soluton an crash n the process. Table 1. The error s calculate from actual epths collecte from laser scanner. The smulatons were run overnght, snce for mult-resoluton bnocular case, t takes 3-4 mnutes per mage. 64 mages, monocular case Average error n log scale (E) Average Multplcatve error (=10^E) Mn-cost sec Full algorthm (full exact LP) sec Ieal Error (cannot be acheve, by the moel n [1]) ^0 = 1-64 mages, mult-resoluton bnocular case Average error n log scale Average Multplcatve error (=10^E) Average runnng tme Mn-cost sec Full algorthm (full exact LP) sec Ieal Error (cannot be acheve, by the moel n [1]) 0 10^0 =1 - Average runnng tme 5 Ths s the lmt of the learnng metho (mae of probablstc moel an features) propose n [1], NOT A LIMIT OF THE INFERENCE. Snce, the nference s convex an exact, a LP wll always gve the accurate an optmal soluton, therefore, t s not possble to make nference better. The error can be reuce by proposng new Artfcal Intellgence an learnng technques, e.g. By mprovng learnng an proposng new probablstc moels, whch obvously s outse the scope of the class.

10 VI. CONCLUSION We formulate the nference problem for the 1-norm mnmzaton as a LP wth half the number of varables use n [1]. We bult a mn-cost moel to nfer a approxmate soluton, whch has a upper-boun on error from the correct groun-truth soluton. If exact optmal soluton s neee, ths soluton can also act as a ntal soluton for exact LP, thus reucng the run tme sgnfcantly. Note: The costs c an m an alpha, also neee to be generate by learnng; whch requre a computer tme of about 9-10 hours. Snce, the coe wrtten for ths was not part of MS&E211, therefore t s not reporte here. However, we nveste tme for MS&E211 to generate all these numbers for ths project's purpose, whch took sgnfcant amount of tme, spent specfcally for ths project. V. REFERENCES [1] Ashutosh Saxena, Sung H. Chung, Anrew Y. Ng, Learnng Depth from Sngle Monocular Images. NIPS 18, 2005.