Semantic Scene Concept Learning by an Autonomous Agent

Transcription

1 Semantc Scene Concept Learnng by an Autonomous Agent Weyu Zhu Illnos Wesleyan Unversty PO Box 29, Bloomngton, IL 672 Abstract Scene understandng addresses the ssue of what a scene contans. Exstng research on scene understandng s typcally focused on classfyng a scene nto classes that are of the same category type. These approaches, although they solve some scene-understandng tasks successfully, n general fal to address the semantcs n scene understandng. For example, how does an agent learn the concept label red and ball wthout beng told that t s a color or a shape label n advance? To cope wth ths problem, we have proposed a novel research called semantc scene concept learnng. Our proposed approach models the task of scene understandng as a mult-labelng classfcaton problem. Each scene nstance perceved by the agent may receve multple labels comng from dfferent concept categores, where the goal of learnng s to let the agent dscover the semantc meanngs,.e., the set of relevant vsual features, of the scene labels receved. Our prelmnary experments have shown the effectveness of our proposed approach n solvng ths specal ntra- and ntercategory mxng learnng task.. Introducton Scene understandng addresses the ssue of what a scene contans. Exstng research on scene understandng s based typcally on ether scene modelng (Belonge, Malk and Puzcha 22; Selnger and Nelson 999) or supervsed learnng (Murase and Nayar 995; Mel 997). In both cases, a detector s bult to dfferentate one scene label from another, where all labels of nterest come from the same category type. Although the exstng approaches solve some scene understandng tasks successfully, they n general fal to address another mportant ssue n vsual percepton: the semantcs. For example, how does an agent learn mutually non-exclusve labels such as red, ball, and cat wthout beng told of ther category types n advance? The capablty of learnng the semantcs of a label s crucal for ntellgent human-computer nteracton and robotc natural language acquston. To learn semantcs of scene labels, supervsed learnng usually fals the task because one learnng system can only classfy the labels of one category type. For example, semantcally t s vald that a scene receves both labels of red and ball f t contans a ball object n red color. However, supervsed learnng fals to address ths scenaro because the label red and ball belong to dfferent category types (.e., the color category and shape category). To tackle ths problem, we have proposed a novel research called mult-labelng scene concept learnng. The fundamental dea s to learn semantcs of scene labels va creatng the assocatons between labels and the relevant vsual features contaned n mages. Scene labels that an agent receves may come from multple concept categores that are unknown to the agent beforehand. For example, gven a scene contanng a coke can, the vald labels nclude red, can, or coke. However, the robot s not told that red s a label related to colors whle can refers to a shape. Fgure llustrates the dfference between the task that we are dealng wth and that of a supervsed learnng problem. For both cases, the agent receves labels resded n the leaf nodes (the l-nodes). The challenge of the multlabelng learnng case les n the unknown hdden layers,.e., the category type of the label receved (the C-nodes). In addton, the mult-labelng learnng has to deal wth the scenaro that a gven scene nstance receves multple labels of dfferent category types nstead of only one as n the supervsed learnng case. S l l 2 l n hdden C C 2 C m (a) l l 2 l n l m l m2 l mk Fgure. (a) Supervsed learnng; mult-labelng. S-node: scene nstance; C-node: category type; l-node: label A smlar research on scene concept understandng s termed symbol groundng (Duygulu et al. 22; Mor, Takahash and Oka 999; Gornak and Roy 24). However, to our best knowledge, none of the exstng symbol groundng research addresses the case that a concept label comes from multple concept categores. For nstance, the study n (Duygulu et al. 22) s to match keywords, whch could be more than one, wth the relevant components n a pcture, where all of the labels are of the same category type,.e., the category of the objects of nterest n a scene. In ths study, we have proposed a generc model, whch s based on jont probablty densty functon of vsual fea- S AAAI-5 / 962

2 tures, for mult-labelng scene concept learnng. As a prelmnary step, we have developed a small-szed system, whch s based on the proposed learnng model, for semantc scene concept learnng. The proposed method uses a two-level Bayesan nference network to determne the category type of a scene label. Our prelmnary experments have shown the effectveness of ths method n catchng the semantcs of labels of unknown category types. The proposed learnng methods are formulated n Secton 2. Secton 3 and 4 presents the detals of our approach, ncludng feature extracton and concept category nferences. Experments are gven n Secton 5, followed by the concluson n Secton 6. To avod confusons, some termnologes used n ths paper are summarzed below. Scene labels: Semantc descrptons of a scene, such as red, square, and Peps. All scene label terms are n talcs font n ths paper. Concept (label) category: A class of labels characterzng the same type of vsual attrbute. The categores studed n ths paper nclude color, shape, and the objects of nterest. Concept category terms are captalzed n ths paper, e.g., color category s denoted as COLOR. Features: Vsual nformaton extracted from an mage. A feature s a vector of data. For example, a color feature s a trplet of {hue, saturaton, value} and a shape feature could be a vector of seven nvarant moments. 2. Task Formulaton 2. A General Purpose Learnng Framework The goal of semantc scene concept learnng s to dscover the assocatons between relevant vsual features and scene concept labels that characterze certan semantc meanngs of a scene. Gven a set of vsual features, the task of concept learnng can be formulated as dscoverng the Jont Probablty Densty Functons (JPDFs) of the vsual features of a concept label. For example (Fgure 2), gven that a scene of nterest s characterzed wth color and shape features, the jont vsual feature space s defned as the drect sum of the color and shape feature spaces (for smplcty, the color and shape feature spaces are represented as the two axes n the fgure). The JPDF of a typcal COLOR label s gven n (a), and dsplays the JPDF of a typcal SHAPE label. Smlarly, one may obtan the JPDF of an arbtrary object of nterest gven that the semantcs of an OBJECT label can be represented adequately wth the combnatons of color and shape features only. Once the assocatons between labels and feature JPDFs are bult, scene labels are retreved by matchng the scene features detected n a pcture to the JPDFs of the labels learned. By thresholdng the degrees of matches, a set of scene labels are retreved and used to descrbe the gven scene. For nstance, when the robot sees a Peps soda can, t wll retreve the related labels blue, can and Peps, etc. shape a COLOR label color a SHAPE label (a) Fgure 2. JPDFs of typcal COLOR and SHAPE labels 2.2 Proposed Approach The feature JPDF-based representaton scheme may serve as a general model for semantc scene concept learnng. However, wthout ncorporatng heurstc knowledge, the computaton of the JPDF of a scene label has to be based on statstcal countng only, whch could be tme consumng n practce. To cope wth ths dffculty, our strategy s to utlze the doman knowledge of scene labels to facltate the learnng of feature JPDFs. Specfcally, we parameterze,.e., set the format of, the JPDF of each concept category type accordng to our knowledge. By fxng the format of feature JPDFs, the learnng s formulated as solvng two problems: ) determne whether a scene label belongs to a category by evaluatng how well the observed features agree wth the format of the JPDF of that category type; 2) compute the parameter set accordngly f the category type of a label can be (or almost be) determned. At ths stage, our study has been focused on learnng the labels that can be used to descrbe a scene contanng an object. Prelmnarly, we are nterested n color and shape features of an object scene. Scene labels come from three categores: COLOR, SHAPE, and the group of the objects of nterest (denoted as OBJECT). The formats of feature JPDFs of these category types are defned heurstcally as: JPDF COLOR = Φ(μ c,ψ c ) JPDF SHAPE =Φ(μ s,ψ s ) () JPDF OBJECT = Φ(μ k s,ψ k s ) Φ(μ l c,ψ l c ) k shape where Φ(μ, Ψ) s a normal dstrbuton centered at the vector μ wth Ψ beng the covarance matrx. The subscrpts c and s stand for a color or shape feature vector, respectvely. The ndex k ndcates the possble vews of an object and l ndexes the colors that are assocated wth a certan vew of the object. The defnton n () assumes that the JPDF of a COLOR label must follow a certan normal dstrbuton centered at a color feature vector and be ndependent to the shape features. Smlarly, the JPDF of a SHAPE label s characterzed wth a shape feature vector (plus the covarance matrx) and s ndependent to the color features. The JPDF of an OBJECT label s a Gaussan mxture of all possble vews of that object, each of whch conssts of a l color AAAI-5 / 963

3 shape feature and a combnaton of several color features snce the object may contan multple colors. Accordng to the above defnton, the nature of scene concept learnng s to compute the lkelhood of a label belongng to a certan category type and meanwhle determne the parameter set accordngly. The frst problem s solved usng a two-level Bayesan nference network that wll be descrbed n the followng sectons. The second one s solved usng the Maxmum Lkelhood Estmaton (MLE) algorthm (Duda, Hart, and Stork 2) accordng to the observed scene examples of the concept label. 3. Feature Extracton and Representaton 3. Preprocessng Snce our focus s semantc scene concept learnng based on extracted vsual features, the processng of feature extracton was smplfed by settng the background of a scene unform and smple. For each scene nstance, an ntensty based mage segmentaton algorthm was used to extract the regon of nterest from the background (Faugerous 983). 3.2 Color Feature Extracton Each extracted regon was decomposed nto several sgnfcant color components represented n HSV {hue, saturaton, value} standard. A color component s sad sgnfcant f the number of pxels of that color account for over 3% of the extracted regon. Snce we are nterested more n the hue nformaton n color comparson, the three components n an HSV trplet were weghted by factors of.,.2, and.5 n smlarty comparson. 3.3 Shape Representaton Three factors were consdered n choosng a shape descrptor n ths study. Frst, t should be nvarant to shfts, scalng, and rotatons. Second, the descrptor s deally n a fxed length for the purpose of comparson. Thrd, some nner propertes, such as holes, need to be addressed. Wth these consderatons n mnd, we have chosen to use nvarant moments (Hall 979; Hu 962) plus a so-called centralzed edge dstrbuton hstogram for shape representaton. A thorough study of shape descrptons can be found n (Mehtre, Kankanhall and Lee 997; Scassellat, Alexopoulos and Flckner 994). Invarant Moments. The nvarant moments method was frst proposed by Hu (Hu 962). The formula used n ths study was borrowed from Hall (Hall 979). An nvarant moment set conssts of seven shape coeffcents calculated from the extracted regon of nterest. The obtaned descrptor s nvarant to shfts, scalng, and rotatons. Centralzed Edge Dstrbuton Hstogram. The nvarant moment descrptor s effectve n dfferentatng between rregular shapes whle t s farly nsenstve to regular symsymmetrc shapes such as crcles, squares, and pentagons. To overcome ths shortage, we have proposed to use another shape descrptor called Centralzed Edge Dstrbuton Hstogram (CEDH), whch s defned as follows.. Compute the dstance d k from each edge pont k to the geometrc center of the shape. 2. Create hstogram of d k / d max, where d max = max k d k. 3. Quantze the hstogram nto unts unformly, correspondng to the normalzed dstances from an edge pont to the center of the regon. The CEDH descrptor s nvarant to shape translatons, scalng, and rotatons. Emprcally ths measurement s a good complement to the nvarant moment descrptor snce t s effectve n dfferentatng between symmetrc shapes. An example s gven n Fgure 3, n whch the nvarant moments and CEDH of two symmetrc shapes: round and square, are compared. The nvarant moment descrptor s nsenstve to these shapes whle CEDH dfferentates them qute well. By combnng the two features, a descrptor consstng of 7 real values (7 nvarant moments + CEDH) s bult. Although the resultant descrptor does not characterze object shapes unquely, ts performance on shape dfferentaton s, however, emprcally satsfactory Invarant Moments round square CEDH (a) Fgure 3. Comparson of two shape descrptors round square 4. Concept Category Inferences The key of scene concept learnng s to determne the category type of a label accordng to the examples observed n learnng. Once the category type s determned, the related set of parameters (defned n equaton ) s calculated accordng to the MLE method and used to represent ths label. The category type of a label s estmated usng a two level Bayesan nference network, whch consst of (and referred to as) local nference and global nference. 4. Local Inference The am of local nference s to calculate the probablty of the category type of a label of nterest accordng to the vsual features obtaned from the scene examples receved n learnng. The term local ndcates that the category types of other labels are not used for the nference. The basc dea of local nference s to evaluate the evdence of the observed scene examples of a label agreeng 9 AAAI-5 / 964

4 wth the format of the feature JPDFs of a gven category type defned n equaton. The network for local nference s gven n Fgure 4. The root node (Category Type) has three values correspondng to the three category types. The evdence that a certan category type, say category, s supported by the observed scene examples s computed as follows. Frst, the set of parameters of category defned n equaton s calculated accordng to the examples observed usng the MLE method (t s worth notcng that ths operaton has nothng to do wth the one ntroduced n secton 2.2 and the one n the last step n the summary n secton 4.3). The resultant set of parameters s used to calculate the evdence that the observed examples havng a label of category. Denote an observed example of the label of nterest as x k, where k N wth N beng the number of the observed examples of ths label. The evdence that supports the category type s calculated as N P( X CT ) = J ( x k ) (2) k= where X = {x, x 2,..., x N }; J s the feature JPDF wth respect to category type dscussed above; the term CT can take one of the three values of a category type: OBJECT, COLOR, or SHAPE. By applyng the Bayesan nference rules (Pearl 988), the posteror probablty P(CT X) s calculated as P( CT X) P( X CT ) π ( CT ) = J ( x ) π ( CT ) (3) N k = where the pror π (CT ) s set unformly as /3. Gven a label of nterest, say label j, the output of local nference s a set of posteror probablty P(CT X j ) that ndcates the category type lkelhood of ths label gven the scene examples observed n learnng. {OBJECT, COLOR, SHAPE} Category Type Example_Support_Evdence Fgure 4. Inference network based on local features only 4.2 Global Inference The dea of global nference s to adjust the category type probablty of a label usng the category type nformaton of other labels. The motvaton of dong ths adjustment s as follows. Gven a label, denoted as a, that could be a COLOR or OBJECT label accordng to the output of the local nference network,.e., k P( OBJECT X a ) P(COLOR X a ) >> P(SHAPE X a ). Meanwhle, there exsts a COLOR label, denoted as b, that has already been learned (.e., P(COLOR X b ) s hgh) and has the same or smlar color mean (μ) as that of label a. In ths case, t s safe to say that the label a s unlkely to be a COLOR label because we assume that each label must represent unquely a certan semantcs of a scene. That s, t s mpossble to have one physcal color receve two dfferent color labels n learnng. The global nference network s gven n Fgure 5. The nput of the network s the sets of category type probabltes of all the labels calculated from the local nference network,.e., the set of P(CT X j ), where ndexes the category types and X j s the set of observed examples of label j. The output of the network s a set of adjusted category type probabltes for each label. {yes, no} The root node (Category Type) of the nference network s defned the same as that n the local nference network. The color or shape conflcton node takes one of two values: yes or no. The evdence of havng a conflcton (the case of yes) s gven by ev m, whch s calculated as ev m k j ( f ) ( ) m fm P CTm Xk = max exp (4) k j 2 where m ndexes the modules of color () or shape (2); ndex j refers to ths label and k s for any other labels; f m represents the mean feature (color or shape) vector of a label. The term P(CT m X k ) s the probablty of label k beng a color or shape label calculated from the local nference network. Accordng to the defnton n (4), the factor ev m changes from to.5, whch corresponds to the cases from non-conflctng to conflctng, respectvely. Consequently, the evdence of a conflcton node takng the value of yes () or no (2) s gven by e m,e m 2 { }={ev m, -ev m ), respectvely. The adjusted category type probablty s therefore calculated accordng to (Pearl 988): P(CT e) P(e m CT ) π (CT ) = e l l [ m P(c m CT )] π (CT ) m= {OBJECT, COLOR, SHAPE} P(cc CT) color conflcton Color Conflct Evdence Category Type {yes, no} m= Shape Conflct Evdence l= P(sc CT) shape conflcton Fgure 5. Inference network based on global nformaton AAAI-5 / 965

5 Where the pror π(ct ) s the probablty P(CT X j ) obtaned from the local nference network (equaton 3). The condtonal probablty matrx P(cc CT) and P(sc CT) s defned heurstcally as T P(cc CT ) =, P(sc CT ) = By applyng the local and global nference engnes a set of consstent category probabltes for each label s obtaned, whch s used for scene nformaton retreval later. The effectveness of ntroducng the global nference engne s dscussed further n the expermental secton. 4.3 Summary of the Learnng Method We summarze the proposed approach for semantc scene concept learnng by an autonomous agent as below:. Collect examples of scene labels va the nteractons between the agent and the envronment. For example, the teacher shows the agent an object once a tme and meanwhle assgns a relevant concept label accordngly. 2. Calculate the concept category probabltes for each label usng the local nference network (secton 4.). 3. Do global nference based on the category probabltes of all of the labels calculated from local nference (secton 4.2). 4. Calculate the parameter sets formulated n equaton accordngly usng the MLE method. Use the resultant feature JPDF for future scene nformaton retreval. 5. Expermental Results The proposed semantc scene concept learnng system has been tested on both a smulated and real learnng robot. 5. Smulatons Our smulatons were conducted over a set of artfcal shapes, colors and objects of nterest. Fgure 6 dsplays the set of artfcal shapes used for learnng. Nne color labels were defned consstng of red, yellow, orange, lght blue, dark blue, green, pnk, purple, and black. Color samples were collected from the palette n MS wndows accordng to humans perceptual judgments. 4 vrtual objects were defned, each of whch conssted of to 4 combnatons of the predefned colors and shapes (each combnaton corresponds to a possble vew of the vrtual object). Overall a total of 63 (4 shapes, 9 colors, 4 objects) scene labels were defned. Fgure 6. Artfcal shapes used n smulatons T The learnng process was smulated as follows. A vrtual object, along wth a vrtual vew, was selected randomly one at a tme. Accordng to the vew hypothetcally observed, the system ssued randomly one of the 63 concept labels that correctly descrbed the current scene. For example, f a scene contans a red square, the canddate labels are red, square, and the name of the correspondng vrtual object. To make the learnng more close to real, the observed color and shape features were perturbed at each observaton. For colors, the perturbaton was to add Gaussan nose to the RGB values of a regstered color. For shapes, a perturbng affne transform defned as x τ + ε = ε 2 x + α (5) y ε 3 τ + ε 4 y β was used, where (α, β) s a par of random shftng factors; τ s a random scalar between.5 and 2, and ε k are small random perturbng values. The learnng performance was evaluated n terms of scene label recalls. Specfcally, the agent was presented wth an arbtrary vew of an arbtrary object and was asked to retreve all of the learned scene labels that best descrbe the current scene. Fgure 7 dsplays the recalls wth a varyng sze of tranng examples wth and wthout usng the global nference (GI) engne. The statstcs were collected over ndependent runs, each of whch conssted of 5 random testng vews. The resultant curves show clearly the contrbuton of the global nference engne, wth whch the label recalls were sgnfcantly boosted. Recall rates Scene Label Recalls number of tranng samples Fgure 7. Recalls of scene label retrevals 5.2 Implementaton on a Real Robot w/o GI w/ GI We have used the proposed method to teach a real robot to learn labels of colors (COLOR), shapes (SHAPE), and the names of the objects of nterest (OBJECT). Fgure 8 dsplays the set of objects used n the experments. Each object conssts of one or more colors. The shape of an object may change from dfferent perspectves. Durng the tranng, the teacher pcked an object to show the robot and meanwhle assgned a relevant label (usng the keyboard) accordng to the judgment of the teacher. AAAI-5 / 966

6 Fgure 8. The set of objects used n learnng The testng phase was smlar to that n the smulaton, n whch a total of 22 labels (6 colors, 3 shapes, and 3 objects) were covered. Table compares the recalls after 4 tranng examples (data were collected n addtonal 4 testng examples). Encouragngly, the proposed method receved a qute satsfactory recall after a small perod of tranng. An example of tranng and testng s gven n Fgure 9, n whch (a) dsplays the scenaro that the agent receved a label red when a red ball was presented n tranng. In testng (fgure ), the agent successfully retreved all of the labels,.e., blue, can, and Peps, that were related to the scene contanng a Peps can. Learnng mode W/ GI W/O GI Recalls 86.5% 65.2% Table. Recalls n real robot learnng n 4 testng examples. Label receved: red Retreval: blue, can, Peps (a) Fgure 9. Example of tranng (a) and testng 6. Concluson and Future Work The objectve of ths research s to explore a new world n vson study,.e., learnng the semantcs of scene labels by an agent. The learnng task s formulated n a way of assocatve memory where the am s to dscover the assocatons between scene labels and relevant vsual features. The capablty of semantc-level scene concept learnng s crucal for ntellgent human-computer nteracton. A generc model for semantc scene concept learnng s proposed n ths study, based on whch a small-szed concept learnng system was developed usng a two-level Bayesan nference network. Whle the assumpton and setup of our prelmnary study were to some extent artfcal and smple, the expermental result has dsplayed the effectveness of the proposed approach n catchng the semantcs of scene labels by an agent. Based on our prelmnary work, the future research wll be carred out n the followng two drectons:. Develop an ntegrated learnng approach for generalpurpose scene concept understandng. So far, the agent s able to learn scene labels that come from several fxed categores whose doman knowledge,.e., the format of feature JPDFs, s known. Although the proposed learnng method s expansble, t s more desrable to have an agent be able to learn adaptvely and autonomously. 2. Explore rcher vsual features. Our current study s focused on statc attrbutve features such as colors and shapes. Ths restrcton mposes lmts on many sceneunderstandng tasks. As another focus n future study, we wll nvestgate rcher scene features, ncludng both spatal and temporal vsual patterns, to address more comprehensve semantcs of a natural scene. References Belonge, S., Malk, J., and Puzcha, J., 22. Shape Matchng and Object Recognton Usng Shape Context. PAMI. Duda, R., Hart, P., and Stork, D., 2. Pattern Classfcaton. 2ed, John Wley & Sons, Duygulu, P., Barnard, K., Fretas, N., and Forsyth, D., 22. Object recognton as machne translaton: Learnng a lexcon for a fxed mage vocabulary. 7 th ECCV Faugerous, O, 983. Fundamentals n Computer Vson. Cambrdge Unversty Press. Gornak, P., and Roy, D., 24. Grounded Semantc Composton for Vsual Scenes, Journal of Artfcal Intellgence Research, 2: Hall, E., 979. Computer Image Processng and Recognton. Academc Press. Hu, M., 962. Vsual pattern recognton by moment nvarents. IRE Transactons on Informaton Theory 8: Mehtre, B., Kankanhall, M., and Lee, W., 997. Shape Measures for Content Based Image Retreval: A Comparson. Informaton Processng & Management 33(3): Mel, B., 997. SEEMORE: Combnng Color, Shape, and Texture Hstogrammng n a Neurally-Inspred Approach to Vsual Object Recognton. Neural Computng 9(4): Mor, Y., Takahash, H., and Oka, R., 999. Image-to-word transformaton based on dvdng and vector quantzng mages wth words. st Int l Workshop on Multmeda Intellgent Storage and Retreval Management. Murase, H., and Nayar, S., 995. Vsual Learnng and Recognton of 3-D Objects from Appearance. Internatonal Journal of Computer Vson 4(): Pearl, J., 988. Probablstc Reasonng n Intellgent Systems: Networks of Plausble Inference. Morgan Kaufmann Press. Scassellat, B., Alexopoulos, S., and Flckner, M., 994, Retrevng mages by 2D shape: a comparson of computaton methods wth human perceptual judgments. Proc. of Spe - the Int l socety for Optcal Engneerng, (285): 2-4. Selnger, A., and Nelson, R., 999. A Perceptual Groupng Herarchy for Appearance-based 3D Object Recognton. Computer Vson and Image Understandng. 76(): AAAI-5 / 967