Developing Dually Optimal LCA Features in Sensory and Action Spaces for Classification

Transcription

1 Developing Dually Optimal LCA Features in Sensory and Ation Spaes for Classifiation Nikita Wagle and Juyang Weng Department of Computer Siene & Engineering Mihigan State University East Lansing, Mihigan Abstrat Appearane based methods have utilized a variety of tehniques suh as Linear Disriminant Analysis (LDA), Support Vetor Mahine (SVM), k-means lustering, and sparse autoenoders to extrat training-data dependent features from stati images by stati images, we mean, foreground objets and bakgrounds are present in stati images as snapshots. The Developmental Networks (DN) use Lobe Component Analysis (LCA) features developed not only from the image spae X but also the ation spae Z. Sine, Z an be taught to represent a set of trainer speified meanings (e.g., type and loation), a DN treats these meanings in a unified way for both detetion and reognition for objets in dynami luttered bakgrounds. However, the DN method has not been applied to publily available datasets and ompared with well-known major tehniques. In this work, we fill this void. We desribe how the Z information enables the features to be more sensitive to trainer speified output meanings (e.g., type and loation). The reported experiments fall into two extensively studied ategories global template based objet reognition and loal template based sene lassifiation. For the datasets used, the performane of the DN method is better or omparable to some major loal template based methods but the DNs also provide statistis-based loation information about the objet in a luttered sene. I. INTRODUCTION Tehniques for appearane-based pattern reognition an be ategorized into two types global template based methods and loal template based methods. The former type assumes that the objet of interest has already been deteted and ropped, then shifted, saled, and rotated in a standard way. The later type uses multiple loal templates at arbitrary image loations on senes where features of objets of interest an arise anywhere in the sene. If the loation of objet is not reported, the later type is also alled sene lassifiation (i.e., features indiate a sene type). A. Existing Major Methods One of the major ontributions of this work is the soures from whih features are extrated (or developed). The traditional methods disussed here, suh as LDA, SVM, spae auto-enoders, sparse RBM, k-means lustering, et. typially extrat features from sensory (i.e. image) spae only, not inluding the ation (i.e. motor) spae, and the feature spae (i.e. feature response ompetition) itself. The sparse method suh as k-means lustering, finds top-k lusters; however, learning is not based on top-k mehanism, but on the nearest luster instead in a bath fashion. In this work, the network TABLE I COMPARISON OF CHARACTERISTICS OF GLOBAL AND LOCAL TEMPLATE BASED METHODS. SPARSE METHODS INCLUDE SPARSE AUTO-ENCODERS, SPARSE RBMS, K-MEANS CLUSTERING, ETC. IN THIS WORK. Charateristis LDA SVM Sparse Methods DN Environmental openness Possible No No Yes High dimensional sensors Yes Possible Yes Yes Completeness Yes No No Yes Real-valued ations No No No Yes Real-time training No No No Yes Inremental learning No Yes No Yes Perform while learning No No No Yes Input having bakground No No Yes Yes struture of the DN is at least as important as the feature lustering tehniques. The LDA tehnique [1], [2] finds a linear feature spae in whih the ratio of the between-lass satter over the with-lass satter is maximized. The SVM tehnique [3], [4] onstruts a hyperplane in a high-dimensional kernel feature spae, so that the margin to the nearest data points is maximized. The loal template based methods use features derived from loal pathes of images (rather than from the whole image) by applying well-known unsupervised feature learning algorithm suh as sparse auto-enoders [5], sparse RBM [6], k-means lustering, et. [7], [8], [9]. B. Charateristis of DN The DN (Developmental Network) framework [10] has used global templates from image datasets [10], as well as loal templates from luttered senes [11]. The novel harateristis of a DN over some well-known pattern reognition methods are disussed below and a summary is provided in Table I. 1) Environmental openness: The DN learns and improves from human taught with motor-supervised experiene of ations through realisti senes, though its performane is imperfet due to limitations in omputational resoure and learning experiene. The DN does not assume that only a few objets (e.g., fae) are of interest, so the environment is open. 2) Low and high dimensional sensors: The DN is appliable to low- and high-resolution video, but should work reasonably well regardless of the resolution, sine the internal operations of DN are based on normalized inner produt. 3) Completeness: All the features in DN emerge as optimal

2 representations learned from the different amounts of teahing. The DN does not use, and not limited by, handrafted features (e.g., SIFT, or oriented edges whih may fail if edges are sparse or absent in an input). 4) Traditionally, only lass labels are used as output, but DN framework enables the system to learn multiple onepts so that type is just one among other possible onepts suh as loation and sale (not experimented here). 5) The DN framework uses top-down input in the Z-area representing loation and type information, as well as the traditional bottom-up inputs in the X-area to develop optimal features. Suh new kind of features is able to disregard input dimensions that are irrelevant to top-down inputs [10]. 6) The DN finds optimal lusters in the spae of X Z, (instead of only image spae X), whih are sensorimotor features, where Z is the effetor spae. By optimal, we mean maximal likelihood (ML) in the representation of the spae of X Z is based on limitation in omputational resoure amount of teahing in DN. 7) Real-valued ations: The DN aepts and proesses real-valued sensory and motor information, instead of human supplied disrete lass labels (e.g., eah neuron in Z represents a lass), making it more general, sine ations an be taught or reated for the physial world without a human predefined lass. 8) Real-time training: During training, the sensory and memory refreshing rate must be high enough so that eah physial event an be temporally sampled and proessed in real-time. The lass labels of objets appearing in the real sene require a human to supply, restriting the possibility of real-time training. The DN an take ations diretly, regardless of whether they represent a real-valued ation or a lass label. 9) Inremental learning: The DN supports bath learning. Additionally, aquired skills must be used to assist in the aquisition of new skills, as a form of saffolding. Eah new observation x X and z Z must be used to update the urrent DN and the urrent (x, z) must be disarded before the next (x, z) an be aquired. The amount of data in the sensory and motor stream is virtually unbounded in real time, making inremental learning neessary. 10) Perform while learning: At any time, the human teaher an teah the DN by imposing an ation on its motor port Z (motor-supervised learning). As soon as the human lets Z free, the DN generates z Z as its best predition or best ation at this time. 11) Input having bakground: The DN an deal with objet of interest in any position in unknown luttered bakgrounds. When DN is trained on loal pathes derived from image senes, they add votes to a single supervised loation in the Loation Motor (LM) of Z, although different loal pathes are at different loations in the input image. C. Novelty This work has two major novelties First, the DN algorithm is originally meant for a more general problem of spatiotemporal event detetion and reognition in omplex bakground, and is not restrited to the global and loal template based image mathing problem. However, it is desirable that we apply it to the spae-only problems here so that it an be ompared with major pattern reognition tehniques. The ways in whih motor spae Z is used, are different between DN method and others ompared. Seond, this work is the first to use the DN for problems where multiple loal features ontribute to the lassifiation (type) and loalization (loation). Multiple firing feature detetors (Y neurons) vote for the orresponding single neuron in the Type Motor (TM) area and the orresponding single neuron in the Loation Motor (LM) area, the two subareas of area Z. DN an be used for shallow learning, deep learning, and a mixed shallow and deep learning. Here, we onentrate on shallow learning. Experimentally, we show that the shallow learning (DN) is at least omparable to the deep learning methods (sparse auto-enoders, k-means lustering, et.). In a loal feature based DN, the where information from LM is applied as supervised input to the DN along with the what information during the training phase. While Z is free during testing phase, through multiple updates of DN, the loation and type information fed from Y to Z and then from Z bak to Y reinfore eah other and suppress inonsistent features the objet loation must be onsistent with type and vie versa. Thus, shallow learning method DN has faster relaxation between Y and Z than in deep learning methods. The remainder of the paper is organized as follows. In setion II, we review previous work. In setion III, we explain DN. The experimental results are presented in setion IV, and setion V gives onluding remarks. II. PREVIOUS WORK In this setion, we briefly disuss the most widely applied algorithms for global and loal template based image mathing. A. Global Template Based Methods Of the many methods for global template mathing (Wavelets, Neural Networks, Correlation, PCA, LDA, SVM), we disuss two types of methods LDA and SVM. 1) LDA algorithm: The LDA algorithm, previously devised in [2] is based on optimal subspae generation using two projetions - a Karhunen Loéve projetion to produe a set of MEF (most expressive features) features followed by a disriminant analysis projetion to produe a set of MDF (most disriminant features) features. The MEF projetion is sensitive to lighting variations. The MDF overomes this limitation by using information in the lass labels. 2) SVM algorithm: The SVM algorithm [3], [4], maps a set of n-dimensional data points from a finite-dimensional spae to a high- dimensional spae, onstruting a (n-1)-dimensional maximum- margin hyperplane or set of hyperplanes in a highdimensional spae, so that a funtional margin, that has the largest distane to the nearest training data points of any lass, is maximized. The testing data points are then mapped to the same spae and are predited to belong to a lass based on

3 SVM LCA MDF vetor LDA Truth The feature representation, y (i,j) is split into four equally sized quadrants and the sum in eah quadrant is omputed to yield a redued K-dimensional representation of eah quadrant for a total of 4K features on whih a linear lassifiation algorithm is applied to identify new data instanes [9]. III. THE DEVELOPMENTAL NETWORK MODEL In this work, we onsider the DN model to have a general purpose brain area Y, whih is onneted with the sensory area X and the motor area Z, as illustrated in Figure 2, in whih the order of areas from low to high is X, Y, Z. Muh of the material in this setion in extrated from Weng and Luiw 2011 [10]. Fig. 1. The two-lass lustering of data points when subjet to linear separator in MDF spae in LDA algorithm, RBF kernel based non-linear SVM lassifiation versus the separation of data points into voronoi regions in DN algorithm whih side of the margin they fall on. When relation between data points and lass labels is non-linear, the Gaussian RBF kernel (K(x i, x j ) = exp( γ x i x j 2 ), γ > 0) is hosen over the linear kernel. The effetiveness of an RBF kernel depend on a good hoie of two parameters C and γ, where C > 0 is the penalty parameter of degree of mislassifiation and γ = 1 2σ, and σ determines the area of influene of the support vetor over the new data instane. Figure 1 shows the two-lass lustering of data points the LDA boundary is linear, whereas the SVM boundary is haraterized by the non-linear kernel funtion hosen. The DN algorithm separates data points into voronoi regions, suh that the boundary separating the two regions is at an equal distane from both points. B. Loal Template Based Methods A simple feature learning framework that inorporates a loal feature-learning algorithm like sparse auto-enoders [5], sparse RBMs [6], K-means lustering, and Gaussian mixture models [7], [8], as a blak box module [9] within has been studied to disover loal features from unlabeled data instanes. A set of random pathes is extrated from unlabeled training data instanes, suh that eah path has dimension w w and has d-hannels, where w is the size of the reeptive field; eah w w path an be represented as a vetor in R N of pixel intensity values, suh that N = w w d. A dataset of randomly sampled pathes X = x (1),..., x m is then onstruted, where x i R N. Then, eah path x (i) is normalized, and the dataset is optionally whitened. An unsupervised learning algorithm viewed as a blak box module takes the dataset X and outputs a funtion f : R N R K and maps input vetor x (i) to a new feature vetor y = f(x) R K of K features, where K is a parameter of the unsupervised learning algorithm used. A (n w 1) (n w 1) representation with K hannels is defined for eah w w subpath of the training data; y (i,j) is referred to as the K-dimensional feature representation extrated from loation i, j of the training data. A. Area Funtion At the first time instanes, during the prenatal learning phase, the first neurons of A in {X, Y, Z} initialize their synapti vetors V = (v 1, v 2,..., v ), where eah synapti vetor v i is initialized using the input pair p i = (b i, t i ), where the bottom up input is b i and the top-down input is t i, i = 1, 2,...,, and initialize the firing ages A = (a 1, a 2,..., a ), suh that eah firing age a i is initialized to be zero, i = 1, 2,...,. After the birth phase, at eah time instant, eah area A omputes its response r from its input pair p = (b, t) based on its adaptive part N = (V, A) and its urrent response r. The urrent response r is regulated by the attention vetor b a. Eah area A updates its adaptive part N to N as follows: (r, N ) = f(b, r, t, r a, N) where f is the area funtion. The attention supervision vetor r a is used to softly avoid the area A from exessively learning bakground; it suppresses all the A neurons to zeros exept 3 3 = 9 ones entered at the orret objet loation. The area A, whether X, Y, orz ompute and update in a unified way as desribed above. But X area does not have bottom-up input and Z area does not have top-down input, sine they are nerve terminals. B. Area Computation We introdue that the reeptive field (RF) of a neuron should ontain three parts sensory RF (SRF), motor RF (MRF) and lateral RF (LRF), respetively. The effetive field (EF) of eah neuron should also inlude three parts: sensory EF (SEF), motor EF (MEF), and lateral EF (LEF), respetively. The SRF, MRF, LRF, SEF, MEF, LEF together form the hextuple fields of a neuron as shown in Figure 4(a). The lateral onnetions onnetions with neurons in the same area that are strongly orrelated or antiorrelated, the highly orrelated ells are onneted by exitatory onnetions to generate a smooth map that globally overs a rough terrain but gradually beomes seletive and loal to fit the details of the terrain, whereas, highly anti-orrelated ells are onneted by inhibitory onnetions, so that the neurons in the same ortial area an respond to different features, and allow only top ranked neurons to fire and update, ensuring that other weakly responding neurons do not fire so that they do not learn

4 irrelevant information and also keep their long-term memory intat. Consider an area A in {X, Y, Z}. Eah neuron in A has a weight vetor v = (v b, v t ), orresponding to the two area inputs (b, t) if both inputs are assoiated with them, else the part of input that is not assoiated with the area is not inluded in the notation. The sum of two normalized inner produts gives the pre-ation of the neuron: Z Motor Area r(v b, b, v t, t) = v ṗ, where v is the unit vetor of the normalized synapti vetor, v = ( v b, v t ), and ṗ is a unit vetor of the normalized input vetor, p = (ḃ, ṫ). The inner produt measures the degree of math between the two diretions v and ṗ, beause r(v b, b, v t, t) = os(θ) where θ is the angle between two unit vetors v and ṗ. In our DN based LCA algorithm [10], we use top-k mehanism to show that lateral onnetions within neurons in eah area enable them to sort top winner neurons within eah time step t n, n = 1, 2, 3,... If k = 1, only one winner neuron fires with a response value y j = 1 and the rest of the neurons in the area do not fire, so y i = 0 for i j. Thus, in general, for k > 1, a dynami saling funtion shifts and sales the pre-ation potential of eah neuron r i in a dynami manner, so that the winner has a response value y = 1 and the (k 1)-th and weaker neurons have a response value zero. C. Dual Optimality of Lobe Component Analysis The Lobe Component Analysis (LCA) [10] is a model for long-term memory retention, and uses a dually optimal (optimal in spae and time) framework that asts long-term and short-term memory together by the optimal distribution of the limited number of neurons of eah area in the input spae X Z optimal Hebbian learning, as shown in Figure 3. In LCA framework, the Y area ats as a bridge between the sensory area X and the motor area Z, as shown in 2. The twoway loal onnetions in green represent neuronal input and in pink represent neuronal output. In the same area, near neurons are onneted by exitatory onnetions for smoothness of representation, and far neurons are onneted by inhibitory onnetions, hene, ompetition between neurons result in detetion of different features by different neurons. The meaning of the dual (spatiotemporal) optimality of LCA is shown in Figure 3. The upper area is a 2-D representation of the positions for the neurons in several staked layers within Y area. The top-3 firing neurons in the Y area top-1 neuron in dark blue shade at the enter, top-2 in light blue shade, and top-3 in vertial blue n white stripes, are ontextdependent working memory for the urrent ontext and the ones that do not fire are ontext dependent long-term memory for the urrent ontext. A 2-D representation of a very high dimensional input spae P = X Z of the ortial area Y is shown by the lower area, in whih eah neuron in Y plane is linked with its synapti weight vetor by a urved ar; the synapti weight vetors of Y represented in P as small dots define a Voronoi diagram in P. The spatial optimality of LCA means that the target tiling by the Voronoi diagram in Y X Fig. 2. Top-3 firing neurons for urrent input Y Correspondene between a neuron and its feature Current input Bridge Sensory Area An illustration of LCA network model. Neuronal layer (2-D version) Best mathed neuron Move to tile the manifold using optimal diretions X xz and optimal step sizes The input spae of neuronal layer Fig. 3. The meaning of dual optimality in LCA network. the manifold is optimal to minimize the representation error for P = X Z, whereas its temporal optimality means that the neuronal weight of firing neurons must move toward their unknown best target the quikest throughout the development proess. Not only the diretion but also every step size of eah neuronal update is nearly optimal, as the statistial effiieny theory for neuronal weight update ensures minimum error in eah age-dependent update. D. Where-What Network The Where-What Network (WWN), a DN embodiment, shown in Figure 4(b) shows the three areas of a simple WWN retina X, simple brain Y, and motor Z. The Z area has two onept areas type or what motor TM and loation or

5 Sensory X Top-down SEF Internal Y Top-down MRF Motor Z Exitatory Inhibitory a a SRF Bottom-up Y LRF Lateral X MEF Bottom-up LEF (a) r LM Fig. 5. area. Human Type Animal output Bus Top-down Airplane ontext Car B X Y b (a) (b) Xi where motor LM. The onneting wires indiate that the presynapti and post-synapti neurons have o-fired; a two-way arrow indiates two one-way onnetions whose two synapses are generally not same. The weight is the frequeny of presynapti o-firing when the post-synapti neuron fires. Within eah ortial area, eah neuron onnets with highly orrelated neurons using exitatory onnetions but with highly antiorrelated neurons using inhibitory onnetions. Every Y neuron is loation-speifi and type-speifi, orresponding to an objet type (marked by its olor pattern) and to a loation blok (2 2 eah). Eah LM neuron is loation-speifi and type-invariant, and eah TM neuron is type-speifi and loation-invariant. Eah Z neuron pulls all appliable ases from Y area neurons as well as boosts all appliable ases in Y as top-down ontext. A WWN does not treat features in Y as a bag-of-features to redue the number of training samples, beause of the inner-produt-based neuronal response for Z. The loation of eah element in a vetor x affet the outome of the inner produt. In ase of shortage of Y neurons, eah Y neuron deals with the misalignment between an objet and its reeptive field, simulating a more realisti resoure situation. This shows that, a WWN enables un-modeled onepts, in this ase, loation and type, to be learned interatively and inrementally as ations, serve as attended spatiotemporal equivalent topdown ontexts to diret pereption, reognition and behavior emergene [10]. E. Properties of DN Here, we disuss two important properties of DN distane-sensitive property and top-down representational property relevant to our work. First, the expression for neuronal learning an be rewritten as vj vj w(nj )(yp vj ). Thus, the amount of vetor Top-down Z Top-down Z TM Fig. 4. Where-What networks. (a) Eah neuron has 6 fields. (b) Multiple firing features ontribute to loation and type. () The square-like tiling property of the self-organization in a ortial 8 square tiles with synapti enters ant lev e r Ir nt va le Re Z (b) b Z Loation output Top-down ontext Neurons ompete to fire b A A X Retina a Xr δr δi Bottom-up X (a) ant Xi nt a lev lev Irre Xi Xr Bottom-up X (b) Irre δ r δ i Re lev an t Xr Bottom-up X () Fig. 6. The top-down representational effet: the Xr subspae is sampled denser than Xi. hange w(nj )(yp vj ) is proportional to the vetor differene yp vj = p vj when y = 1, alled the distane-sensitive property. With this property, [10] has established the squarelike tiling property: Suppose that the learning rule in a selforganization sheme has the distane-sensitive property. Then the neurons in the area move toward a uniformly distribution (tiling) in the spae of areal input p if its probability density is uniform. The square-like tiling property of DN is illustrated in Figure 5. In a uniform input spae, neurons in an layer self-organize until their Voronoi regions are nearly isotropi (square-like to nearly hexagons in 2-D). The Voronoi region of neuron is very anisotropi elongated horizontally) resulting horizontal pulling is statistially stronger as shown in Figure 5(a). A horizontal perturbation leads to ontinued expeted pulling in the same diretion as shown in Figure 5(b). Through many updates, the Voronoi regions are nearly isotropi, as illustrated in Figure 5(). Seond, as shown in Figure 6, learning using top-down inputs sensitizes neurons to ation-relevant bottom-up input (e.g., foreground pixels) and desensitize to irrelevant omponents (e.g., leaked-in bakground pixels), when top-down input is unavailable during free-viewing, alled the top-down representational effet, established in [10]: Given a fixed number of neurons in a selforganization sheme that satisfies the distane sensitivity property, adding top-down input from motor Z in addition to bottom-up input X enables the quantization errors for ation-relevant subspae Xr to be smaller than the ation-irrelevant subspae Xi,

6 where X = X r X i. The top-down inputs sensitize the response for relevant bottom omponents although whih are relevant is unknown. Without top-down input, square Voronoi tiles in the bottomup spae give the same quantization width for irrelevant omponent X i and the relevant omponent X r : δ i = δ r. All samples in eah tile is quantized as the point (synapti vetor) at the enter as illustrated in Figure 5(a). With top-down inputs during learning, square tiles over the observed input manifold, indiating the loal relationships between X r and Z as shown in Figure 5(b). When top-down Z is not available during freeviewing, eah tile is narrower along diretion X r than along X i : δ r < δ i, meaning that the average quantization error for relevant X r is smaller than that for irrelevant X i as shown in Figure 5(). The above theorem gives two onsequenes (due to δ i > δ r): First, ation-relevant bottom-up inputs are salient (e.g., toys and other Gestalt effets). Thus, we need to reonsider the onventional thinking that bottom-up salieny is and probably totally innate. Seond, relatively higher variation through a synapse gives information for ellular synapti pruning in all neurons, to delete their links to irrelevant omponents. This was used in synapse maintenane [11], but this apability has been turned off beause the publi image datasets, that we use here, do not present variation of bakground pixels like a dynami physial world would. IV. EXPERIMENTAL RESULTS In this setion, we ondut two experiments, the DN algorithm is applied to: (a) A global template, in whih the patterns to be reognized have been shifted, rotated, and saled so that the entire input image ontains mainly the pattern of interest (explained briefly). (b) A set of loal templates, in whih eah input image ontains a luttered bakground, whih means that the detetion and pre-normalization in (a) of objet of interest has not been done. The experimental results are ompared to other widely-used major algorithms in the pattern reognition ommunity. A. Global Template Based Methods In the first experiment, a set of well-framed feature images from two datasets Weizmann and FERET are trained on LDA, SVM and LCA-net algorithms. In Weizmann dataset, image size is 88 64, from 28 humans, eah having 30 images with all possible ombinations of 2 expressions, 3 lighting onditions, 5 orientations, of whih 812 images are used for training and 28 images are for testing. The FERET fae dataset [12] is a set of 1762 images, of size 88 64, from 1010 humans, of whih 1624 images are used for training and 138 images are used for testing. The LDA lassifier gains 0.6% error on the Weizmann dataset after 30 ross-validation tests (a different image from eah lass is used for testing in eah test) and 17.4% error on FERET data. SVM algorithm gives 15.3% error on FERET data using RBF kernel (C=2; γ=0.0313) and 9.7% error on Weizmann data using linear kernel. LCA algorithm performs better than LDA and SVM Reognition rate NORB dataset CIFAR 10 dataset Number of epohs Fig. 7. The variation number of epohs in training phase to gain maximum attainable reognition rate on NORB and CIFAR-10 datasets Loation error (in pixel) Number of epohs Fig. 9. The number of epohs in training phase versus the loation error (in pixels) in CIFAR-10 dataset. using 625 neurons, it gains no error on Weizmann dataset and 8.7% error on FERET dataset. B. Loal Template Based Methods In the seond experiment, DN is applied to a set of loal templates is derived from two objet reognition datasets, NORB and CIFAR-10 using the idea of loal path extration from foreground in [11] and the reognition rate is ompared to some major loal-feature learning algorithms [5], [6], [8], [9]. TABLE II RECOGNITION RATE GAINED FROM VARIATION OF TOP-K FIRING NEURONS AND PATCH SIZE ON NORB DATASET Path size\k % 59.8% 85.1% 68.0% 54.1% % 60.1% 93.8% 72.9% 63.5% % 63.5% 93.8% 76.0% 63.5% % 63.5% 94.0% 81.6% 71.9% % 72.6% 94.2% 83.2% 71.9% % 72.9% 94.2% 83.7% 74.5% % 74.5% 95.0% 85.2% 74.5% % 71.3% 94.2% 81.6% 74.2% % 69.8% 94.2% 81.6% 66.3% % 68.8% 94.0% 79.2% 45.0% % 65.2% 93.5% 77.5% 32.1% % 65.2% 81.2% 51.3% 24.7% % 74.9% 34.7% % 74.9% 27.8% -

7 (a) Bottom- up Y (b) Top- down Y (TM) () Top- down Y (LM) Ship neuron Automobile neuron Row 5 Col 5 neuron Row 6 Col 5 neuron (d) Bottom- up Z Fig. 8. Visualization of weights in one depth (of 30 depths) in Y area of CIFAR-10 (a) bottom-up weights (23 23 Y neurons in whih eah ell has dimension 10 10) (b) top-down weights (TM) (23 23 TM neurons in whih eah ell has dimension 10 10) () top-down weights (LM) (23 23 LM neurons in whih eah ell has dimension 23 23) (d) Visualization of the bottom-up weights for Z area - two type neurons in one depth (of 30 depths) in TM (1 10 TM neurons in whih eah ell has dimension 23 23) and two loation neurons in LM (23 23 LM neurons in whih eah ell has dimension 23 23). TABLE III RECOGNITION RATE GAINED FROM VARIATION OF TOP-K FIRING NEURONS AND PATCH SIZE ON CIFAR-10 DATASET Path size\k % % 39.4% % 51.5% 23.2% % 30.3% 64.6% 38.3% % 44.4% 72.7% 52.5% 29.3% % 44.4% 80.8% 63.6% 43.4% % 42.4% 79.8% 61.6% 42.4% % 44.4% 76.7% 59.6% 43.4% % 44.4% 71.7% 56.5% 43.4% % 43.4% 68.6% 53.5% 40.4% % 43.4% 67.6% 51.5% 40.4% % 40.4% 64.6% 49.5% 40.4% The NORB dataset with elimination of omplex bakground [13] is a set of images of 50 toys, of size 96 96, belonging to 5 lasses namely, four-legged animals, human figures, airplanes, truks and ars, imaged by two ameras under 6 lighting onditions, 9 elevations and 18 azimuths, of whih images were used for training and testing eah. The CIFAR-10 dataset [14] is a set of olor images, of size 32 32, belonging to 10 lasses, with 6000 images per lass, of whih images are used for training images are used for testing. Eah image in the training set of NORB images is rotated at every angle of 20, so that 18 rotated instanes of eah image in training dataset are obtained, whih are added to the original training set so that a test image is orretly lassified regardless of the elevation angle. In the CIFAR-10 dataset, eah path extrated from the training image is trained at every loation within the image, for instane, a path of size 10 is trained at (32 101) (32 101) i.e different loations within the image and eah of the shifted image instanes are added to the training set, so that a test image is orretly lassified regardless of its position within the image. TABLE IV LOCAL TEMPLATE BASED METHOD COMPARISON ON NORB DATASET Algorithm Reognition rate Conv. Neural Network 93.4% Deep Boltzmann Mahine 92.8% Deep Belief Network 95.0% Deep Neural Network 97.1% Sparse Auto-enoder 96.9% Sparse RBM 96.2% K-means (Hard) 96.9% K-means (Triangle) 97.0% K-means (Triangle, 4000 features) 97.2% WWN 95.0% per frame training time 1.3s per image testing time 0.8s TABLE V LOCAL TEMPLATE BASED METHOD COMPARISON ON CIFAR-10 DATASET Algorithm Re rate Lo Err (pixels) Raw pixels 37.3% - 3-Way Fatored RBM (3 layers) 65.3% - Mean-ovariane RBM (3 layers) 71.0% - Improved Loal Coord. Coding 74.5% - Conv. Deep Belief Net (2 layers) 78.9% - Sparse Auto-enoder 73.4% - Sparse RBM 72.4% - K-means (Hard) 68.6% - K-means (Triangle) 77.9% - K-means (Triangle, 4000 features) 79.6% - WWN 80.8% 1.7 per frame training time 1.5s per image testing time 1.2s

8 The loal feature templates derived from NORB and CIFAR-10 images are trained on the WWN algorithm [11] A reognition rate of 95.0% was obtained from NORB dataset for a path of size 22 22, the thikness of the network being 10, when top-3 neurons fire as shown in Table II; whereas a reognition rate of 80.8% was obtained from CIFAR-10 images for a path of size 10 10, the thikness of the network being 30, when top-3 neurons fire as shown in Table III. The number of training epohs are varied from 0 to 15 and the reognition rate at eah epoh is plotted as shown in Figure 7. Eah epoh performs 3 iterations for reinforement learning of LM and TM input by WWN. The internal representation of Y area and Z area after the training phase of the WWN algorithm are visualized in Figure 8. Eah Y neuron (in all depths) detets a type as in Figure 8(b) in a speifi loation as in Figure 8 (). Due to limited neuronal resoures in Y area, some neurons deal with multiple objet types at multiple pixel loations. The bottom-up weights (TM and LM) of two of Z neurons as in Figure 8 (d) are normalized to the range 0 to 255 suh that the pixel value indiates the strength of the onnetion between the orresponding Y neuron (the same (row,ol) loation as the pixel) and the Z neuron. The Figure 9 shows the loation error (in pixels) over 20 epohs for CIFAR- 10; the loation error remains onstant after 10 epohs. The reognition rate, thus obtained, is ompared to the reognition rate obtained by some major loal-feature learning algorithms as shown in Table IV and Table V to show that a WWN performs omparable to them. In NORB dataset, the objet of interest is already entered, thus, where information in DN gets no room for improvement. However, in the CIFAR-10 dataset, the objet of interest appears at different loations within a sene. Thus, the where information from LM ensures that the objet is present in the sene as a onfiguration (not neessarily rigid) that is onsistent with the training experiene, not as sattered broken parts, for instane, the head and the tail of an airplane is present with an observed onfiguration and not as separate disassembled parts, whih might be a reason for the reognition rate to be slightly better than other methods. V. CONCLUSIONS In this work, the omparison of DN algorithm major wellknown pattern reognition algorithms showed that the performane of the DN method is omparable to those major existing methods. In the loal template based problems, DN allows different firing feature neurons to vote for a single neuron in TM and LM, whih has reahed pixel-level loation auray for CIFAR-10 dataset. The work here indiates that the pay-off of finding the better spatial features seems to have diminished for stati image-based methods. The DN method used here, for spae-only problem was meant for a tight integration of spae and time information for visual events in natural luttered bakgrounds, where objets dynamially sweep aross dynami bakgrounds and other foregrounds. In the work of [15], [16], it has been shown that using temporal information for spatial reognition in DN has onsiderably improved its reognition rate without imposing rigid onstraints on objet appearane through time, whih is typial with model-based objet traking. A possible future diretion of researh is the detetion of objet ontours from rihly textured dynami bakground using a tehnique alled synapse maintenane [11]. The funtion of synapse maintenane in DN was not used for experiments here beause the image datasets available in the publi repositories onsist of snapshots of stati senes whih are very different from what human-eye sees from the dynami physial world, where the ontours of unknown objets manifest themselves when an objet moves relative with its luttered dynami bakgrounds. ACKNOWLEDGMENT The authors would like to thank Matthew Luiw and Yuekai Wang for providing some of their programs. REFERENCES [1] R. Fisher, The use of multiple measurements in taxonomi problems, Annals of Eugenis, vol. 7, pp , [2] D. L. Swets and J. Weng, Using disriminant eigenfeatures for image retrieval, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 18, no. 8, pp , [3] C. Cortes and V. Vapnik, Support-vetor networks. [4] T. Poggio and G. Federio, Networks for approximation and learning, Proeedings of The IEEE, vol. 78, no. 9, pp , [5] I. Goodfellow, Q. Le, A. Saxe, H. Lee, and A. Ng, Measuring invarianes in deep networks, NIPS, [6] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, Convolutional deep belief networks for salable unsupervised learning of hierarhial representations, in Pro. 26th Int l Conf. on Mahine Learning, Montreal, Canada, June , pp [7] B. A. Olshaushen and D. J. Field, Emergene of simple-ell reeptive field properties by learning a sparse ode for natural images, Nature, vol. 381, pp , June [8] J.Yang, K. Yu, Y. Gong, and T. Huang, Linear spatial pyramid mathing using sparse oding for image lassifiation, Computer Vision and Pattern Reognition, [9] A. Coates, H. Lee, and A. Ng, An analysis of single-layer networks in unsupervised feature learning, in Proeedings of the 14th International Conferene on Artifiial Intelligene and Statistis (AISTATS), vol. 15 of JMLR: W&CP 15, Fort Lauderdale, FL, USA, [10] J. Weng and M. Luiw, Brain-like emergent spatial proessing, IEEE Trans. Autonomous Mental Development, vol. 4, no. 2, pp , [11] Y. Wang, X. Wu, and J. Weng, Synapse maintenane in the where-what network, in Pro. Int l Joint Conferene on Neural Networks, San Jose, CA, July 31 - August , pp [12] P. J. Phillips, H. Moon, P. Rauss, and S. A. Rizvi, The FERET evaluation methodology for fae-reognition algorithms, in Pro. IEEE Conf. Computer Vision and Pattern Reognition, Puerto Rio, June 1997, pp [13] Y. LeCun, F. Huang, and L. Bottou, Learning methods for generi objet reognition with invariane to pose and lighting, IEEE Computer Soiety Conferene on Computer Vision and Pattern Reognition (CVPR), [14] A. Krizhevsky, Learning multiple layers of features from tiny images, Department of Computer Siene, University of Toronto: Master s thesis, [15] M. Luiw and J. Weng, Where What Network 3: Developmental topdown attention with multiple meaningful foregrounds, in Pro. IEEE Int l Joint Conferene on Neural Networks, Barelona, Spain, July , pp [16] J. Weng, Why have we passed neural networks do not abstrat well? Natural Intelligene: the INNS Magazine, vol. 1, no. 1, pp , 2011.