Analysis of a Biologically-Inspired System for Real-time Object Recognition

Transcription

1 Cognitive Science Online, Vol.3.,.-4, 5 htt://cogsci-online.ucsd.edu Analysis of a Biologically-Insired Syste for Real-tie Object Recognition Erik Murhy-Chutorian,*, Sarah Aboutalib & Jochen Triesch,3 Det. of Electrical and Couter Engineering, and Det. of Cognitive Science University of California, San Diego 95 Gilan Drive La Jolla, CA Frankfurt Institute for Advanced Studies Frankfurt, Gerany *Corresonding author. E-ail address: erikc@ucsd.edu Abstract We resent a biologically-insired syste for real-tie, feed-forward object recognition in cluttered scenes. Our syste utilizes a vocabulary of very sarse features that are shared between and within different object odels. To detect objects in a novel scene, these features are located in the iage, and each detected feature votes for all objects that are consistent with its resence. Due to the sharing of features between object odels our aroach is ore scalable to large object databases than traditional ethods. To deonstrate the utility of this aroach, we train our syste to recognize any of 5 objects in everyday cluttered scenes with substantial occlusion. Without further otiization we also deonstrate near-erfect recognition on a standard 3-D recognition roble. Our syste has an interretation as a sarsely connected feed-forward neural network, aking it a viable odel for fast, feed-forward object recognition in the riate visual syste. Introduction Efficient detection of ultile objects in real-world scenes is a challenging roble for object recognition systes. Natural scenes can contain background clutter, occlusion, and object transforations which ake reliable recognition very difficult. In this work we develo a syste that efficiently and accurately recognizes artially occluded objects desite osition, scale, and lighting changes in cluttered real-world scenes. Most odern recognition aroaches reresent secific views of objects as constellations of localized iage features. The Scale Invariant Feature Transfor, SIFT, is a well-known exale (Lowe, 4). In this aroach, gradient histogra- Authors occasionally ake a distinction between recognition (what is this object?), detection (e.g., is a face soewhere in this iage?), and ultile object detection (is any of a set of known objects in this iage?). In this aer, we use the generic ter recognition to refer to all of these robles.

2 Cognitive Science Online, Vol.3, 5 based SIFT descritors are couted at Difference-of-Gaussian keyoints and stored along with a record of the key-oint s D location, scale, and orientation relative to the training iage. To detect an object in a new iage, an aroxiate nearest neighbor search atches SIFT descritors extracted fro the iage, and a Hough Transfor detects and roughly localizes the object. To irove erforance for ultile objects, siilar aroaches have eloyed a quantized feature vocabulary, such that the set of features is shared across different object odels (Murhy-Chutorian & Triesch, 5). In this aroach, every extracted local feature is coared to a uch saller set of vocabulary features by a fast nearest neighbor search, and a reference is stored with the key-oint s D location relative to the location of the object. As the nuber of objects increases, the nuber of shared features need not grow roortionally. This benefit fro shared features has been corroborated in a boosting fraework (Torralba, Murhy, & Freean, 4). These authors deonstrated that by allowing only a fixed nuber of total features, using such shared features greatly outerfors a set of classifiers learned indeendently for each object class. Vocabulary-based recognition systes have also been roosed for single object recognition and iage retrieval (Agarwal, Awan, & Roth, 4; Leibe & Schiele, 4; Sivic & Zisseran, 3). This aer resents a novel fraework for sharing ultile feature tyes, such as texture and color features, within and between different object reresentations. We learn robabilistic weights for the associations between features and objects so that any feature, regardless of tye, can contribute to the recognition in a unified fraework. An interesting debate regarding the aforeentioned recognition aroaches is the question of how invariance to transforations (osition, scale, rotation in lane, rotation in deth) should be achieved. On one end of the sectru are aroaches that try to hard-wire such invariance into the syste by using invariant features. At the other end are aroaches that try to learn certain invariance directly fro training data. Our aroach takes an interediate stance, where osition invariance is built into the syste, and invariance to scale and ose are learned fro training data. Syste Overview In brief, our syste works as follows. During training, it creates a set of weighted associations between a learned set of vocabulary features and the set of objects to be recognized. During recognition, vocabulary features that are detected at interest oints in the iage cast weighted votes for the resence of all associated objects at corresonding locations, and the syste detects objects whenever this consensus exceeds a learned threshold. In the following sections we describe these stes in ore detail. Feature Vocabulary The recognition syste uses a vocabulary of local features that quantize a otentially high-diensional feature sace. Our ileentation uses color and texture feature The ter vocabulary is analogous to the feature dictionary used in revious work (Murhy-Chutorian & Triesch, 5).

3 Cognitive Science Online, Vol.3, 5 3 vocabularies. The color features are reresented as D hue saturation vectors, corresonding to the local average of 5x5 ixel windows. The Euclidean distance in olar hue-saturation coordinate sace rovides the basis for coaring color features. To learn the color feature vocabulary, we extract color features at the locations of objects in a large training set of iages and cluster the with a standard K-eans algorith to arrive at our 5 entry color feature vocabulary. The texture features are 4-diensional Gabor jets (Lades et al., 993) corised of the agnitude resonses of Gabor wavelets with 5 scales and 8 orientations, for details see (Murhy-Chutorian & Triesch, 5). For vertically or horizontally oriented Gabor jets, the necessary convolutions can be efficiently calculated with searable filters. For all other orientations, the iage can be first rotated and then rocessed with the sae filters. Our ileentation rocesses all 4 convolutions in aroxiately s on a.8ghz couter. To coare two Gabor jets, x and y, we use the noralized inner roduct, T x y S ( x, y), () x y which is robust to changes in brightness and contrast. By noralizing the vectors and couting only the inner roduct at runtie, the calculations are reduced. To learn the Gabor feature vocabulary we extract any Gabor jets at interest oint locations fro around the objects in a large set of training iages. As an interest oint oerator we choose the Harris corner oint detector which is highly stable over ultile views of an object (Harris & Stehens, 988). We use a odified K-eans clustering to coute a 4 entry Gabor jet vocabulary. The odification of the K-eans clustering consists of noralizing the jets to unit agnitude following each iteration of the algorith. Given either feature tye, finding the nearest vocabulary features that best reresent it requires a nearest neighbor search in a -, or 4-diensional sace, resectively. An aroxiate kd-tree algorith accolishes this efficiently (Mount & Arya, 5). We have found that the syste erfors otially if we use the six nearest Gabor jets and the single nearest color-jet for each resective vocabulary query. As a consequence, our initial encoding of the iage in ters of its features is extreely sarse with only 6 out of 4 Gabor features or out of 5 color features being activated at a given interest oint location. Transfor Sace A D-Hough transfor sace (Ballard, 98; Lowe, 4) artitions the iage sace into a set of regions or bins for each object. During recognition, the detected vocabulary features cast weighted votes for the resence of an object in a secific bin, storing the consensus for classification 3. The otial size of the bins will be discussed in its own section. 3 To avoid the roble of boundary effects fro the discrete Hough bins, each feature votes for a bin and its 8 neighboring bins.

4 Cognitive Science Online, Vol.3, 5 4 Feature Associations Initially, we develo a sarse set of associations between the features and objects. If an object and feature are both resent in a training iage, the syste creates an association between the two. This association is labeled with the distance vector between the location of the feature and the center of a anually drawn bounding box around the object, discretized at the level of the bin sacing. Dulicate associations, (i.e. sae feature, sae object, sae dislaceent) are disallowed. Once all of the training iages have been rocessed in this way, the syste begins a second ass through the training iages to learn a weight for each of the associations. Assuing conditional indeendence between the inuts given the oututs, Bayesian robability theory dictates the otiu weights are given by the log-likelihood ratios, w fd ( X Y ) ln P( X Y ), ln P () f d where X f is a Bernoulli rando variable describing the resence (X f ) or absence (X f ) of feature f in the scene, and Y d is another Bernoulli rando variable indicating the resence or absence of object at a discretized satial offset d fro feature f 4. Figure shows the distribution of the log-likelihood weights for the color and the texture features. Not surrisingly, the higher-diensional texture features tend to be ore discriinative as they have a higher average log-likelihood weight. f d.6.5 Gaborjet weights Colorjet weights.4 Density Loglikelihood Weight Figure. Distribution of Log-Likelihood Weights for each feature tye Otiu Detection Thresholds During recognition, all of the detected features cast weighted votes to deterine the resence of the objects. If any Hough transfor bin receives enough activation, this suggests the resence of the object. To deterine a detection criterion, we develo otiu thresholds fro the axiu a osteriori (MAP) estiator under Gaussian 4 It ay see at this oint that a naive Bayes rule exansion could be alied with these log-likelihood ratios and known riors to obtain the osterior robability than an object is resent, but the underlying conditional indeendence assution is highly erroneous in our case and leads to rather oor erforance.

5 Cognitive Science Online, Vol.3, 5 5 assutions. Let Y be a Bernoulli rando variable describing the resence of the object and let t be a continuous rando variable corresonding to the axiu bin value in the Hough araeter sace of. The MAP estiator,, describes the ost likely value for y given the value of t : ( ) ( ) y y Y y Y t y Ρ arg ax ˆ (3) ( ) ( ) ( ) ( ) ( ) ( ), P() P : P() P : < t t t t (4) where P(y ) is the rior robability that Y y, and (t y ) is the conditional df of t given Y y. We then define the otiu threshold,, as the value of t which satisfies ( ) ( ) ( ) ( ) P P Y Y t Y Y t (5) For t > it is ore robable that the object is resent in the scene, and for t < it is ore robable that the object is absent. Assuing that (t y ) is a Gaussian distribution, we can fully deterine (t Y y ) knowing only the first and second order oents, l and l, where l if the object is resent. We estiate the oents fro the training data and find by solving the quadratic equation: ) ( ) ( ex ex µ θ µ θ π π (6) where P(Y ). Assuing > and l > as is always the case for our data, the solution is given as, 4 a ac b b θ (7) with ( ) ( ). ln + c b a µ µ µ µ Exerients and Results

6 Cognitive Science Online, Vol.3, 5 6 The CSCLAB cluttered scenes database was used to test the erforance of our syste (Murhy-Chutorian & Triesch, 5). It consists of 5 scenes of 5 everyday objects against cluttered, real-world backgrounds with significant occlusion. Each scene contains 3 to 7 objects as shown in Figure. The objects are resented at roughly the sae viewoint in every scene, but there reains differences in deth, osition, rotation, and lighting. The deth changes cause considerable scale variation aong the object classes, which vary by a factor of two on the average. The syste learns scale-invariant reresentations by building a congloerate set of associations fro training iages of objects at reresentative scales. Alternatively, it could be trained with fewer scenes, exlicitly resented at ultile scales (Burt & Adelson, 983). In addition, the database contains scenes of all ten backgrounds without objects, as well as scenes of every background with each object by itself. All of the scenes have associated XML files that store the anually-labeled bounding boxes and naes of the objects for suervised training and evaluation. The dataset was slit into three sets. The first set contained ultile object scenes which were used to create the feature dictionary. The second set contained additional ultile-object scenes and all of the individual object scenes. This set rovided the training data for learning associations between vocabulary features and objects and the corresonding weights. The third set, containing the reaining ultile-object scenes, was resented to the syste for recognition. Figure. Labeled Exale Scene fro the CSCLAB dataset Feature Sharing Figure 3 deonstrates the aount of feature sharing in the learned reresentations for the 5 objects fro the CSCLAB data base. In Figure 3(a) we show histogras of how frequently a feature is shared between reresentations of different objects. Interestingly, there is a sizable fraction of features that are shared by any objects, and only few features are not shared at all, i.e. they are secific to one object only. Figure 3(b) shows how often a feature is shared within one or ultile views of a single object. Noting that there are no dulicate associations, this denotes the nuber of associations between this feature and the object with different discretized

7 Cognitive Science Online, Vol.3, 5 7 dislaceents. One can see that this intra-object sharing is haening less often than the inter-object sharing, but this is a eaningless ratio, since it directly deends on our choice of the Hough bin size and nuber of objects. Frequency Frequency Nuber of Objects 3 4 Nuber of Objects (a) Histogra of inter-object sharing, showing the nuber of objects that connect to each feature for color-jets (left) and Gabor jets (right). Frequency Nuber of Connections Frequency Nuber of Connections (b) Histogra of intra-object sharing, showing the nuber of ties a feature connects to the sae object for color-jets (left) and Gabor jets (right). Figure 3. Feature Sharing Otial Bin Size The otial size of the Hough transfor bins is deterined by a trade-off between two coeting factors. If the bin size is too sall, votes fro the sae object ay fall into different bins because of variations in object aearance such as scale or rotation. Larger bins, however, increase the risk of a surious accuulation of votes fro background clutter or unrelated objects into a single bin, which can lead to a false ositive detection. Because of this trade-off, there exists an interediate bin size that yields otial erforance (Aboutalib, 5). We investigated this effect by

8 Cognitive Science Online, Vol.3, 5 8 systeatically varying the bin size 5. Figure 4 shows the result. The tradeoff favoring interediate bin sizes is clearly visible. Based on this result, we use 6x6 ixel Hough transfor bins to axiize recognition..96 Area under ROC curve Bin Size (Side length in ixels) Figure 4. Area under the averaged ROC curves for various bin sizes Recognition Perforance Figure 5 and Figure 6 show histogras of the detection rates and false ositive rates for the 5 objects in the CSCLAB dataset. The detection rate is defined as the fraction of objects that were successfully detected, and the false ositive rate is the fraction of iages in which an object is incorrectly detected. In this alication, the syste is able to detect ost of the objects ore than 8% of the tie while aintaining less than a 5% false ositive rate. The syste has the ost difficulty with the objects that lack sufficient texture, or have significant transarencies. Perforance exales are shown as ROC curves for the best, edian, and worst individual ROC curves are given in Figure 7. Figure 8 shows an average of the sline-interolated ROC curves for all of the objects. In the course of the exerient, of the 5 objects were erfectly recognized with a % detection rate and no false ositives. Figure 9 rovides exales of the syste s recognition ability. On a.8ghz ersonal couter, our syste requires aroxiately one second to recognize all of the objects in a 64x48 ixel iage. 5 In this exerient we ket the bin size fixed for every object, but an object secific selection of the bin size ay further irove erforance.

9 Cognitive Science Online, Vol.3, 5 9 Nuber of objects Detection rate Figure 5. Histogra of the individual object detection rates at otiu thresholds.8 Detection rate False ositive rate Figure 6. Histogra of the individual object er-iage false ositive rates at otiu thresholds Neural Network Interretation and Relation to Models of Biological Object Recognition It is frequently argued that the rearkable seed of riate object recognition suggests a rocessing architecture that is essentially feed-forward in nature, and roinent odels of biological object recognition are feed-forward rocesses (Fukushia, Miyake, & Ito, 983; Riesenhuber & Poggio, 999). Feed-forward

10 Cognitive Science Online, Vol.3, 5 odels are unlikely to be able to account for all asects of riate object recognition, but they ay be a reasonable aroxiation in any situations. We can interret our syste as a sile feed-forward neural network. In this case, the inut layer consists of the vocabulary features at every ossible discretized location. The outut layer consists of the objects at every ossible Hough bin. The activation of an outut node, y j y( j,q j ), is given by the linear suation, y i w j ij i x x w w x, ij ( fi, i ), fi j ( q j i ) i, (9) and the weights of the network are the log-likelihood ratios of the features entioned earlier. In this context, x i is the ith binary inut node that fires whenever the shared vocabulary feature, f i, is found anywhere inside the unit s recetive field, and wij is the weighted connection between y j and xi. i is the location of inut x i, and q j is the location of y j. We assue w ij whenever yj and x i are not connected. Although the weighted connections are learned fro the relative dislaceent between the inut and outut nodes, this can be interreted as weight sharing in a neural network with connections based on absolute dislaceents.

11 Cognitive Science Online, Vol.3, 5 Figure 7. ROC curves and estiated conditional dfs (black: object absent, gray: object resent) for individual object exales

12 Cognitive Science Online, Vol.3, 5 Figure 8. Averaged ROC curve for all 5 objects The feed-forward neural network interretation of our syste suggests that one could view it as an abstract odel of riate object recognition. In fact the introduction of Gabor wavelet features into couter vision systes was insired by biological findings. In this context, our shared Gabor jet features loosely corresond to shae selective cells in area V4. Coared to the other odels entioned above, the binning oeration inherent in the Hough transfor echanis corresonds to nonlinear oerations that introduce a degree of shift invariance in the above odels. The sarseness of connections fro these features to object detectors (corresonding to oulations of cells in inferoteoral cortex) is also in line with biological considerations, but in stark contrast to any revious odels of biological object recognition, we obtain excellent erforance on a difficult real-world recognition roble. To do so in real-tie aves the way for the develoent of ore elaborate odels of visual cognition that odel object recognition and learning in the context of ongoing behavior. Discussion We resented a new fraework for ultile-object detection with a vocabulary of shared features. Using ultile feature tyes and sarse, weighted associations between vocabulary features and objects, we deonstrated object detection in cluttered real-world scenes desite significant scale variation and occlusion in realtie. Since the syste can be interreted as a feed-forward neural network, it ay be viewed as an abstract odel of object recognition in the riate visual syste, although this was not the ain focus of this research. In a full 3-D recognition task on the otherwise uch siler COIL database, our syste showed excellent erforance. Evaluating our syste on a full 3-D recognition roble that also includes clutter, occlusions, and lighting variations reains a toic for future research. At resent, there are no available benchark databases of this kind. Perforance gains could be achieved by the addition of other feature tyes. Transarent objects and objects lacking unique texture and color were the ost difficult to detect, and this could be reedied by the addition of features that could detect these objects by their characteristic shae. The fraework resented in this aer easily accoodates additional features. A further avenue for future

13 Cognitive Science Online, Vol.3, 5 3 research is the incororation of stereo inforation and the exlicit odeling of object occlusions (Eckes, Triesch, & Malsburg, 5). We would also like to investigate the ability to learn objects with only inial suervision, since hand-labeled training data as we have used here is tedious to create. Recent ilot work has deonstrated this syste s otential for learning object reresentations in a sei-autonoous fashion through online deonstration, where objects are sily shown to the syste for an extended eriod of tie as they undergo scale and ose changes and the syste detects, tracks, segents, and learns to recognize these objects without additional huan intervention (Murhy-Chutorian, Ki, Chen, & Triesch, 5). Figure 9. Exale Recognition Results (squares indicate the estiated object center)

14 Cognitive Science Online, Vol.3, 5 4 Acknowledgents This work was suorted by the National Science Foundation under grants IIS- 845 and IIS We would like to thank Hyundo Ki, Huei-Ju Chen, Eily Ashbaugh and Nathan Rosencrans for their hel in creating the cluttered scenes database. We would also like to thank Boris Lau, Alan Robinson and To Sullivan for their helful suggestions and rograing contributions. References Aboutalib, S. (5). Position invariance in a view-based object recognition syste. Unublished honor s thesis, University of California, San Diego. Agarwal, S., Awan, A., & Roth, D. (4, Noveber). Learning to detect objects in iages via a sarse, art-based reresentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6 (), Ballard, D. (98). Generalizing the Hough transfor to detect arbitrary shaes. Pattern Recognition, 3 (),. Burt, P. J., & Adelson, E. H. (983). The lalacian yraid as a coact iage code. IEEE Transactions on Counications, COM-3,4, Eckes, C., Triesch, J., & Malsburg, C. von der. (in ress). Analysis of cluttered scenes using an elastic atching aroach for stereo iages. Neural Coutation. Fukushia, K., Miyake, S., & Ito, T. (983). Neocognitron: A neural network odel for a echanis of visual attern recognition. IEEE Transactions on Systes, Man, and Cybernetics, SMC 3 (5), Harris, C., & Stehens, M. (988). A cobined corner and edge detector. In Proceedings of the Alvey Vision Conference (. 47 5). Manchester. Lades, M., Vorbr uggen, J. C., Buhann, J., Lange, J., Malsburg, C. von der, W urtz, R. P., et al. (993). Distortion invariant object recognition in the dynaic link architecture. IEEE Transactions on Couters, 4, 3 3. Leibe, B., & Schiele, B. (4). Scale invariant object categorization using a scaleadative ean-shift search. In Proc. deutsche arbeitsge- einschaft fr ustererkennung attern recognition syosiu. Tuebingen, Gerany. Lowe, D. (4). Distinctive iage features fro scale-invariant keyoints. International Journal of Couter Vision, 6 (), 9. Mount, D., & Arya, S. (5, May). ANN: A library for aroxiate near- est neighbor searching, version.. (htt:// ANN/). Murhy-Chutorian, E., Ki, H., Chen, H.-J., & Triesch, J. (5). Scalable object recognition and object learning with an anthrooorhic robot head. In

15 Cognitive Science Online, Vol.3, 5 5 Proceedings of the IEEE Conference on Couter Vision and Pattern Recognition. San Diego, CA. Murhy-Chutorian, E., & Triesch, J. (5, January). Shared features for scalable aearance-based object recognition. In Proceedings of the IEEE Worksho on Alications of Couter Vision. Breckenridge, CO, USA. Nene, S., Nayar, S., & Murase, H. (996, February). Colubia object iage library (COIL-) (Tech. Re.). Colubia University. Riesenhuber, M., & Poggio, T. (999). Hierarchical odels of object recognition in cortex. Nature Neuroscience,, 9 5. Sivic, J., & Zisseran, A. (3, October). Video google: A text retrieval aroach to object atching in videos. In Proceedings of the IEEE International Conference on Couter Vision. Nice, France. Torralba, A., Murhy, K., & Freean, W. (4). Sharing features: efficient boosting rocedures for ulticlass object detection. In Proceedings of the IEEE Conference on Couter Vision and Pattern Recognition.