Online Navigation Summaries

Transcription

1 2010 IEEE Intenational Confeence on Robotics and Automation Anchoage Convention Distict May 3-8, 2010, Anchoage, Alaska, USA Online Navigation Summaies Yogesh Gidha and Gegoy Dudek Cente fo Intelligent Machines McGill Univesity Abstact Ou objective is to find a small set of images that summaize a obot s visual expeience along a path. We pesent a novel on-line algoithm fo this task. This algoithm is based on a new extension to the classical Secetaies Poblem. We also pesent an extension to the idea of Bayesian Supise, which we then use to measue the fitness of an image as a summay image. I. INTRODUCTION Navigation summaies ae specialization of video summaies, whee the focus is on video collected by a mobile obot, on a specific tajectoy. We ae inteested in finding a few images that epitomize the visual infomation about the pat of the wold, which was tavesed by the obot. Figue 1 shows a schematic example of a navigation summay. This poblem is elated to the Vacation Snapshot Poblem[1], whee a touist is tying to captue the most inteesting obsevations of the jouney, with a limited amount of film. A obot suveying an aea might be taking continuous video as it is moving aound, but the human monitoing the data might only be inteested in a few images, which summaize what obot has seen; howeve, deciding which images ae supising o inteesting, and epitomize the visual appeaance of the wold is highly context depent. A. Statistical vesus Semantic Reasoning Navigation summaies can boadly be classified into two categoies, based on how the decisions to choose summay images ae made. Images can be selected based on semantic cues egading scene content, pio knowledge of the obseve, and the exact context of the poblem at hand; o by puely statistical easoning, whee the focus is on the infomation contained in the image, and peviously selected images. The latte appoach is, howeve, moe tactable. In this pape we will focus on the task of compiling navigation summaies based on statistical and infomation theoetic tools. B. Online vesus Offline Navigation summaies can eithe be made offline, once the path tavesal is ove, o online. By online we mean that the decision to include an image in the summay set is made ievocably, and immediately afte it is acquied. Fo a mobile obot collecting thousands of images continuously, an offline algoithm compaing each image with evey othe image can be pohibitively expensive. Apat fom computational cost, the ability to identify an image as being pat of the Fig. 1. An illustated example of a navigation summay. The sequence of images epesent the obsevations made by a obot as it is tavesing a teain. The dotted boxes indicates one possible choice of the summay images. summay has seveal applications. Fo example, conside the task whee we would like to dop a senso node and take additional measuements at chaacteistic egions of a teain. In that case, assuming each summay image epesents a diffeent egion of the teain, the obot could dop a senso wheneve a decision to include an image to the summay set is made. Conside the special case of this poblem whee we only want one summay image, o want to dop only one senso node. We can fomalize and abstact this as a game wee you ae pesented with a finite sequence of andom numbes fom an unknown distibution and then the goal is to identify the maximum numbe in this sequence. Hee the maximum numbe might coespond to the amount of infomation an image has about the teain, and hence maximizing it will give us the most impotant image. This poblem is known as the Secetay Poblem [2]. In this fomulation, the task is posed as one of making a choice to select and hie ievocably, the best secetay, with the inteview scoe of the candidates epesented as a sequence of andom numbes. If we ae allowed to go back to a peviously ejected sample(i.e. secetay), then ou algoithm is tivial. We can just go though all the samples, identify the maximum value, and then in the go back and choose the sample with the maximum value. Hence, the main challenge in the poblem aises due to the fact that the decision made is ievocable. In this pape we will pesent an extension to the secetay /10/$ IEEE 5035

2 poblem, which can then be used to select not just one, but seveal images that fom a summay set; on-line. We also pesent a scoing function based on genealization of Bayesian Supise, used to select the summay images. II. PREVIOUS WORK A. The Secetay Hiing Poblem The secetay poblem has a long and vaied histoy. Due to its boad elevance to diffeent domains, it has been consideed by diffeent authos in seveal diffeent contexts. It is geneally accepted that Dynkin [2] was the fist one to solve the poblem fomally. Fo an inteesting discussion on the oigins of this poblem see [3]. Hee is a desciption of the poblem in its simplest fom with the numbe of secetaies k = 1 : You ae given the task of hiing the best possible candidate fo the job of a secetay. Thee ae n applicants who have signed up fo the inteview. Afte inteviewing each candidate you can ank them elative to all othe candidates seen so fa. You must eithe hie o eject the candidate immediately afte the inteview. You ae not allowed to go back and hie a peviously ejected candidate. A typical stategy would be to just obseve the fist candidates without accepting any, then find the highest scoe among them, and then hie the fist candidate with scoe highe than that. This is known to be the povably optimal stategy fo this poblem. The poblem now is to select the best value fo. Let Φ() be the pobability of successfully finding the highest scoing candidate, when we set the taining inteval to be. We then have: Φ() = i=+1 P (S i ), (1) whee S i is the event that the ith candidate is the highest scoing candidate, and that ou algoithm did not select any of the pevious candidates. Hence we have: Φ() = i=+1 n n 1 n i 1 (2) 1 dx (3) i = (log n log ). (4) n Now to optimize Φ(), we set the deivative equal to 0: d log n log Φ() = 1 dx n n = 0 (5) = = n e. (6) Hee 1/n is the pobability that the ith candidate is the highest scoing one, and /(i 1) is the pobability none of the pevious candidates wee selected. B. The Multiple Choice Secetay Poblem Fo the case when the numbe of positions which need to be filled is moe than one (k > 1), thee ae seveal possible ways in which the above single secetay solution can be genealized. Kleinbeg [4] suggested an algoithm to maximize the expected sum of the scoes of the candidates. The algoithm woks by splitting the candidates into two oughly equal intevals, whee the bounday is chosen andomly using a binomial distibution B(n, 1/2). We then ecusively apply the classic(k = 1) secetay algoithm to the fist half of the candidates, choosing l = k/2 candidates. While doing this we also find the lth highest scoing candidate fom the fist half and use this as a fixed theshold to select the emaining candidates in the second half. Babaioff et al. [5], [6] suggest a simple algoithm with the same goal of maximizing the expected sum of the scoes of the selected candidates. A sliding theshold to choose the candidates. Algoithm 1 descibes this appoach. Find the top k scoes in the fist = n/e candidates, without selecting any. Call this list of thesholds, T = {t 1,.., t k }. foeach emaining candidate (x +1,, x n ) do if candidate has scoe highe than the minimum scoe in T then Hie the candidate. Remove the minimum value fom the set T. if T is empty then beak Algoithm 1: MaxExpectedSumScoes({x 1,.., x n }, k) Note that both Klenbeg and Babaioff algoithms, ae optimal in maximizing the sum of candidate scoes. Howeve, they assumes that the scoing function does not dep on the what has aleady been selected. Hence using these algoithms to select ou summay images will only povide an optimal expected case solution when all images ae statistically indepent and do not shae any infomation. This is typically not tue. Fo example, images adjacent in time ae bound to be quite simila. In Section III-A, we will pesent a new algoithm which does not have this deficiency. Thee ae seveal othe vaiants of this poblem which exist in the liteatue. Feeman [9] eviews some of these vaiants. C. Video Summaies Most of the existing wok elated to the poblem of navigation summaies is in fact on the moe geneal poblem of video summaization. The most familia and commonplace method fo summaizing video is to subsample in tempoal domain (i.e. viewing while fast fowading). This appoach woks well fo limited compession ates, but is aely used fo speeds above 32 times eal-time. Moeove, the appoach woks well 5036

3 fo manually cafted (commecial) video whee the length of time that a topic emains on-sceen in geneally popotional to its impotance. Smith and Kanade looked at poducing video skims by depicting statistical summaies of video content accompanied by selected audio extacts [10]. That wok employed the TFIDF (Tem Fequency/Invese Document Fequency) metapho fom text document indexing to index the audio data in a video steam. They used colo histogams fo video segmentation in the tempoal domain. In that wok, subsystems wee also used to detect text captions and human fontal head views, in ode to place emphasis on content that was paticulaly impotant in the context of summaizing television footage. In elated wok, Ngo, Ma ana Zhang cluste video fames based on an motion model and then used nomalized cuts to segment the esulting poximity gaph, and epesent each cluste by an exempla [12]. The motion model was based on MPEG motion vectos. A skim was poduced that educe the video length by up to 90 pe cent by etaining the fames that made the lagest contibution to the successive changes in the image o audio content. In the wok of Gong and Liu [14], video skims ae poduced based on a selection of key-fames with high infomation content. The accomplished by pojection into a subspace that computed with espect to colo histogams of a set of 9 (3x3) sub-windows that cove each input fames. This estimate is poduced by computing a singula value decomposition ove video sequence to estimate both the aveage fame, and the distance of each fom the Eigenspace defined by the SVD. The distinctiveness of a fame is then computed by measuing its distance fom the mean of the aveage fame in the SVD subspace, and this povides a citeion fo selecting key-fames. While acceptable video summaies ae epoted, the authos suggest that the technique has shotcomings when colo infomation is not distinctive enough. In [15], Ju et al. consideed the task of summaizing videos of a specific context, in this case, videos of confeence pesentations. Takeuchi et al. in [16] poduced video summaies using a set of a-pioi manually classified images. Related to this poblem is the poblem of clusteing images that depict the same envionmental stuctue, albeit fom diffeent vantage points [18]. That wok diectly compaes all n 2 images in the video sequence with one anothe using wide-baseline steeo methods. While the esults of such an appoach appea somewhat pomising, the computational cost fo a full video sequence ae pohibitive. A. Algoigthm Oveview III. APPROACH Ou objective is to find a small set of images that summaize a obot s visual expeience along a path. We ae pesented with a steam of images taken by the obot and afte obseving each image, we must make an ievocable decision accepting o ejecting the image as a summay image. We fist conside the easie offline vesion of the poblem, whee we ae allowed to choose ou summay images afte doing all the paiwise compaison of the images. If we assume that each selected image coves infomation about some faction of the input images and the tajectoy, then this poblem is educes to an instance of the Set Cove poblem, which is known to be NP-had [19]. Hence we ae motivated to look fo appoximate solutions. Algoithm 2 pesents a geedy appoach to this poblem. Hee k is the numbe of desied images in the summay set, X is the set of input images, and S is the cuent set of summay images. The scoing function Scoe(X S), computes a fitness scoe of the given image X, given the peviously selected set of summay images S. In Section IV, we will pesent an infomation theoy based scoing function, optimizing the summaies fo infomation gain. This algoithm has O(N 2 ) computational complexity in numbe of images in the input set ( X ). if k = 0 then etun S X max agmax X X Scoe(X S) S S {X max } X X \ {X max } etun SummaizeOffline(k 1, X, S) Algoithm 2: SummaizeOffline(k, X, S) Algoithm 3 is on-line algoithm, which appoximates the esults of the geedy off-line algoithm pesented above. Hee t is the cuent time, X t is the latest image acquied by the obot, k is the appoximate numbe of images we want in the summay, n is constant and epesents the total numbe of images we ae expecting to see in this un, and S is the set of cuently selected summay images. The algoithm fist decides on an obsevation inteval (t, t obs ), whee it find the theshold scoe v theshold. Afte that, the fist image which exceeds this theshold is chosen. It then ecusively calls itself to pocess futue images. if t > n o k = 0 then etun S t obs t + (k, n t) v theshold max {Scoe(X t S),, Scoe(X tobs S)} t t obs epeat t t + 1 until Scoe(X t S) v theshold o t n S S {X t } etun SummaizeOnline(t + 1, k 1, n, S) Algoithm 3: SummaizeOnline(t, k, n, S) Apat fom the scoing function Scoe(X S), pefomance of this algoithm mainly deps on the obsevation 5037

4 inteval function (k, n), which decides on how many images to obseve, befoe stating the selection pocess. We use analysis simila to the the classical Secetaies Hiing poblem, to compute the optimal value fo this obsevation inteval. B. Top k Secetaies Using Fixed Theshold Ou goal now is to compute the obsevation inteval, which maximizes the pobability of finding the k highest scoing images. We will pesent the appoach as an extension to the classical secetaies poblem discussed in Section II-A, whee a single theshold is used to optimally select the top k highest scoing candidates. The theshold is set to be the maximum obseved scoe in the fist candidates. We would like to be a function of k, the numbe of secetaies we want, such that if k inceases, then deceases. We can compute optimal value fo by maximizing the pobability of success Φ(), whee success is defined by the event that all of the top k highest scoing candidates have been selected. Let J i be the event that with the selection of the ith candidate, we have succeeded. We can then wite : Φ k () = P (Success) (7) = P ( n i=1j i ) (8) = P ( n i=+kj i ) (9) We ignoe the fist candidates since those candidates ae neve selected as pe ou algoithm definition, and then we can ignoe the next k 1 candidates since its impossible to select k candidates fom k 1 possibilities. Analogous to Equation 2, we can then wite Φ k () as: Φ k () = = i=+k i=+k = k n P (J i ) (10) k n i 1 ( n ) n 1 i=+k+1 ) ( i 1 ( n ( i ) (11) i ), (12) whee ( n k) is the binomial coefficient. Lets us examine the thee components of Equation 11. The fist tem: k/n is the pobability that the ith candidate is one of the top k candidates. The second tem: /(i 1) is the pobability that none of the pevious candidates wee the last of the top k selected candidates. These two tems ae simila to the two tems in Equation 2. The thid tem: ( ) ( i 1 / n ) is the pobability that all of the emaining k 1 candidates have been selected. Combining these tems we get the pobability of the event that we have successfully selected last of the top k candidates. C. Optimal Taining Inteval Let ˆ be the values of fo which the pobability of success Φ k () is maximum. We computed the ˆ fo n = 1000, and k = We appoximate this in closed fom as: n ˆ(k) (13) ke 1/k Fig. 2. Optimal taining inteval as a function of numbe of secetaies k, fo n = 1000 secetaies. The dots epesent the actual taining inteval, and we found that the function ˆ(k) = n/ke 1/k, epesented by the cuve above, is a good appoximation of these points. Figue 3 shows a plot of this appoximation. We can hence use this expession to set the obsevation inteval in the algoithm descibed in Sec. 3. This will allow us to exploit this method in pactice. IV. SCORING FUNCTION We would like ou set of summay images to be chosen in such a way that they minimize the supise in obseving any of the othe images. A. Bayesian Supise Itti and Baldi [21] fomally define supise in tems of diffeence between posteio and pio beliefs about the wold. Let ou pio hypothesis H of the wold be defined using the pobability distibution P (H). When a new data obsevation D is made, it can be called supising if the posteio distibution P (H D) is significantly diffeent fom the pio distibution P (H). One of the best ways to compae these distibutions is using elative entopy o Kullback- Leible(KL) divegence [22], which measue the infomation gain. We can then define supise R as: R(D, H) = d KL (P (H D) P (H)) (14) B. Set Theoetic Supise Bayesian supise measues the distance using KL divegence, between two distibutions: P (H D) and P (H). We popose modeling ou pio and posteio using a set of distibutions, whee P (H) is eplaced by the set {P (H)} H H, and posteio P (H D) is eplaced by the set {P (H + )} H + H {D}. We would now like to measue the distance between these two sets. Hausdoff metic povides a natual way to compute distance between two such sets. It defines this distance d as the maximum distance of a set to the neaest point in the othe set: d(a, B) = max min a A b B d(a, b). (15) 5038

5 Hence taking Hausdoff distance between ou sets of posteio and pio hypothesis, whee distance is KL divegence, we define Set Theoetic Supise R as: R (D, H) = d(h {D}, H) (16) = max min d KL(P (H + ) P (H))(17) H + H D H H = min d KL(P (D) P (H)). (18) H H Applying this poblem to ou task of selecting summay images, we model the hypothesis using the set of selected images. Scoe of an image epesenting its suitability as a summay image, given a set of aleady selected summay images can then be defined as: Scoe(X S) = max min d KL(P (S + ) P (S))(19) S + S X S S = min d KL(P (X) P (S)), (20) S S whee X is image whose scoe we would like to calculate, P ( ) defines the distibution of image featues, and S is an image fom the set of selected images S. In this pape we used a simple pixel colo histogam as the image featues. V. RESULTS AND DISCUSSION We tested ou algoithm on seveal data sets, fou of which ae shown hee. Each set used in this pape contains 64 images, and we want 8 samples fom each set. Note that the offline algoithm we have descibed in this pape is guaanteed to give us 8 images. Howeve, the online algoithm does not, because if in an iteation no images ae found exceeding the theshold, then none ae selected. We tested ou algoithm on thee diffeent types of teains data. Figue 5 show images fom the steet view data sets. We show the images selected by the online algoithm, and fo compaison also show the 8 images etuned by the offline algoithm. We see that 4/5 images in the online summay ae pat of the offline summay. Figue 4 shows images fom a simulated Matian analogue envionment. We see that both the online and offline algoithms captue the visual appeaance of the teain well. Oveall we find the selections made by ou algoithms clealy captue the divesity of the image types in input sample set. VI. CONCLUSION AND FUTURE WORK In this pape we looked at the poblem of automatic geneation of navigation summaies. A navigation summay is a small set of images, which captue the visual expeience of a mobile obot, as it taveses a path. O contibution to solving this poblem is focused in two aeas. Fistly we pesent a new extension to the classical secetaies poblem, and use it to fomulate a new online algoithm to build navigation summaies. Secondly, we pesent a genealization to concept of Bayesian Supise, and then use it to pick the summay set. In futue we hope to impove ou scoing function by consideing moe featues. Some of these featues could be a) Input Set: b) Off-line Summay: c) On-line Summay: Fig. 3. Mas Dataset. a) 64 images acquied fom the Mas Analogue Site opeated by the Canadian Space Agency. b) Result of unning the off-line algoithm. c) Result of unning the on-line algoithm. We see good coveage of the vaiance in appeaance of the teain in both the off-line and on-line summaies. explicitly defined, and be poblem depent; fo example featues like human faces. We also plan on looking at othe statistical featues which ae poblem indepent; fo example, fequency of diffeent kind of textues in the images, o SIFT o SURF o MSER featues. VII. ACKNOWLEDGEMENTS We would like to thank Google Inc. fo geneously allowing us to use thei Steet View data, and Canadian Space Agency fo allowing us collect data at thei Mas Analogue site. We would also like to thank Pof. Luc Devoye fo his helpful guidance. REFERENCES [1] E. Bouque and G. Dudek, Automated image-based mapping, in IEEE Compute Vision and Patten Recognition (CVPR) Wokshop on Peception of Mobile Agents, June 1998, pp [2] E. Dynkin, The optimum choice of the instant fo stopping a makov pocess, Soviet Math. Dokl, vol. 4, [3] T. S. Feguson, Who solved the secetay poblem? Statistical Science, vol. 4, no. 3, pp , [4] R. Kleinbeg, A multiple-choice secetay algoithm with applications to online auctions, in SODA 05: Poceedings of the sixteenth annual ACM-SIAM symposium on Discete algoithms. Philadelphia, PA, USA: Society fo Industial and Applied Mathematics, 2005, pp [5] M. Babaioff, N. Immolica, and R. Kleinbeg, Matoids, secetay poblems, and online mechanisms, in SODA 07: Poceedings of the eighteenth annual ACM-SIAM symposium on Discete algoithms. Philadelphia, PA, USA: Society fo Industial and Applied Mathematics, 2007, pp

6 [6] M. Babaioff, N. Immolica, D. Kempe, and R. Kleinbeg, Online auctions and genealized secetay poblems, SIGecom Exch., vol. 7, no. 2, pp. 1 11, [7] M. Sakaguchi, Dowy poblems and ola policies, Rep. Statist. Appl. Res. JUSE, [8] M. Babaioff, N. Immolica, D. Kempe, and R. Kleinbeg, A knapsack secetay poblem with applications, in APPROX 07/RANDOM 07: Poceedings of the 10th Intenational Wokshop on Appoximation and the 11th Intenational Wokshop on Randomization, and Combinatoial Optimization. Algoithms and Techniques. Belin, Heidelbeg: Spinge-Velag, 2007, pp [9] P..Feeman, The secetay poblem and its extensions: A eview, Intenational Statistical Review, [10] M. A. Smith and T. Kanade, Video skimming and chaacteization though the combination of image and language undestanding, Content-Based Access of Image and Video Databases, Wokshop on, vol. 0, p. 61, [11] S. Uchihashi, J. Foote, A. Gigensohn, and J. Boeczky, Video manga: Geneating semantically meaningful video summaies, in Poceedings of the seventh ACM intenational confeence on Multimedia. ACM Pess, 1999, pp [12] C. wah Ngo, Y. fei Ma, and H. jiang Zhang, Automatic video summaization by gaph modeling, in in Poceedings of the 9th IEEE Intenational Confeence on Compute Vision, 2003, pp [13] Y. fei Ma, L. Lu, H. jiang Zhang, and M. Li, A use attention model fo video summaization, in In Poceedings of ACM Multimedia, 2002, pp [14] Y. Gong and X. Liu, Video summaization using singula value decomposition, in Poc. of CVPR, 2000, pp [15] S. M. S.X. Ju, M.J. Black and D. Kimbe, Summaization of videotaped pesentations: Automatic analysis of motion and gestue, in IEEE Tansactions on Cicuits and Systems fo Video Technologies, [16] Y. Takeuchi and M. Sugimoto, Use-adaptive home video summaization using pesonal photo libaies, in Poceedings of the 6th ACM intenational confeence on Image and video etieval CIVR 07, [17] J. Sivic and A. Zisseman, Video google: A text etieval appoach to object matching in videos, in In Poc. ICCV, 2003, pp [18] F. Schaffalitzky and A. Zisseman, Automated scene matching in movies, in In Poceedings of the Challenge of Image and Video Retieval, London, LNCS Spinge-Velag, 2002, pp [19] R. M. Kap, Reducibility among combinatoial poblems, in Complexity of Compute Computations, R. E. Mille and J. W. Thatche, Eds. Plenum Pess, 1972, pp [20] Y. Gidha and G. Dudek, Optimal online data sampling o how to hie the best secetaies, in Canadian Confeence on Compute and Robotic Vision(CRV), Kelowna, Bitish Columbia, May [21] L. Itti and P. Baldi, Bayesian supise attacts human attention, Vision Reseach, vol. 49, no. 10, pp , 2009, visual Attention: Psychophysics, electophysiology and neuoimaging. [22] S. Kullback, Infomation theoy and statistics. John Wiley and Sons, NY, [23] J. Satta, G. Dudek, O. Chiu, I. Rekleitis, P. Giguèe, A. Mills, N. Plamondon, C. Pahacs, Y. Gidha, M. Nahon, and J.-P. Lobos, Enabling autonomous capabilities in undewate obotics, in Poceedings of the IEEE/RSJ Intenational Confeence on Intelligent Robots and Systems, IROS, Nice, Fance, Septembe a) Input set: b) Offline summay: c) Online summay: Fig. 4. Steet View Dataset 1. a) 64 images fom ou steet view data set. b) Result of unning the off-line summay algoithm, equesting eight summay images. Hee the fist image coesponds to the mean appeaance of the scene. c) Result of unning the on-line summay algoithm. We see that 4/5 images in the selected set ae eithe same o simila to the off-line selection set. 5040