Deep Rule-Based Classifier with Human-level Performance and Characteristics

Transcription

1 Deep Rule-Based Classifier with Human-level Performane and Charateristis Plamen P. Angelov 1,2 and Xiaowei Gu 1* 1 Shool of Computing and Communiations, Lanaster University, Lanaster, LA1 4WA, UK 2 Tehnial University, Sofia, 1000, Bulgaria (Honorary Professor) {p.angelov, x.gu3}@lanaster.a.uk Abstrat- In this paper, a new type of multilayer rule-based lassifier is proposed and applied to image lassifiation problems. The proposed approah is entirely data-driven and fully automati. It is generi and an be applied to various lassifiation and predition problems, but in this paper we fous on image proessing, in partiular. The ore of the lassifier is a fully interpretable, understandable, self-organised set of IF THEN fuzzy rules based on the prototypes autonomously identified by using a one-pass type training proess. The lassifier an self-evolve and be updated ontinuously without a full retraining. Due to the prototype-based nature, it is non-parametri; its training proess is non-iterative, highly parallelizable and omputationally effiient. At the same time, the proposed approah is able to ahieve very high lassifiation auray on various benhmark datasets surpassing most of the published methods, be omparable with the human abilities. In addition, it an start lassifiation from the first image of eah lass in the same way as humans do, whih makes the proposed lassifier suitable for real-time appliations. Numerial examples of benhmark image proessing demonstrate the merits of the proposed approah. Keywords- fuzzy rule based lassifiers, deep learning, non-parametri, non-iterative, self-evolving struture 1. Introdution Nowadays, deep learning has gained a lot of popularity in both the aademi irles and the general publi thanks to the very quik advane in omputational resoures (both hardware and software) [20], [26]. A number of publiations have demonstrated that deep onvolutional neural networks (DCNNs) an produe highly aurate results in various image proessing problems inluding, but not limited to, handwritten digits reognition [12], [13], [21], [40], objet reognition [18], [23], [42], human ation reognition [10], [41], human fae reognition [19], [33], [46], remote sensing image lassifiation [44], [50], et. Some publiations suggest that the DCNNs an math the human performane on handwritten digits reognition problems [12], [13]. Indeed, DCNN is a powerful tehnique that provides high lassifiation rates. There are also reently introdued approahes exploiting deep models for image understanding [31], [32] by learning informative hidden representations from visual features of images through DCNNs. However, DCNNs have a number of defiienies and shortomings. For example, they require a huge amount of training data, are usually offline, lak transpareny and their internal parameters annot be easily interpreted; they involve ad ho deisions onerning the internal struture; they have no proven guaranteed onvergene; they have limited parallelization ability. It is also well-known that DCNN-based approahes are not able to deal with unertainty. They perform lassifiation quite well when the validation images share similar feature properties with the training images, however, they require a full retraining for images from unseen lasses as well as for images with feature properties different from that of the training images. On the other hand, traditional fuzzy rule-based (FRB) systems are well known for being an effiient approah to deal with unertainties. FRB systems have been suessfully used for lassifiation [8], [24] offering transparent and interpretable struture. Their design also traditionally requires handrafting membership funtions, assumptions to be made and parameters to be seleted. More reently, very effiient data-driven FRB lassifiers were proposed whih an learn autonomously from the data (streams) [2], [8], and self-evolve, however even they ould not reah the levels of performane ahieved by deep learning lassifiers mainly beause of their quite simple and small internal struture. In this paper, we offer a prinipally new approah, whih ombines the advantages of both, the reently introdued self-organising non-parametri FRB systems [2], [7],, applied to lassifiation problem [3] with the onept of a massively parallel multi-layer struture that deep learning benefits from. This results in a prinipally new type of a multi-layer neuro-fuzzy arhiteture, whih we all Deep Rule-Based (DRB) system and demonstrate its performane on various image lassifiation problems. The proposed DRB approah *Corresponding Author

2 employs a massively parallel set of 0-order fuzzy rules [3], [7], [8] as the learning engine, whih self-organizes a transparent and human understandable IF THEN FRB system struture. Eah IF THEN fuzzy rule of the DRB system onsists of a (large) number of prototypes, whih are not pre-determined, but are identified through a fully autonomous, online, non-iterative, non-parametri training proess. These prototypes are the most representative atual data samples (images) at whih the data density obtains loal maxima (the most typial loally images); they are used to automatially form data louds (luster-like groupings of data with similar properties) by attrating the other data samples (images) to them [7]. The training proess of the DRB system an start from srath, and more importantly, it an start lassifiation from the first image of eah lass in the same way as humans do, and is able to onsistently self-evolve and self-update its struture and meta-parameters with newly observed training images, whih makes the proposed lassifier suitable for real-time appliations. The proposed DRB approah is more generi, but in this paper we limit our study only to image lassifiation. We use only the very fundamental image transformation tehniques suh as normalization, rotation, saling and segmentation. In this way, the generalization ability of the well-known (low and high level) feature desriptors from the field of omputer vision, whih we use (desribed in the next setion) is further improved. These pre-proessing steps are ommon for the omputer vision literature, but we do not use one speifi pre-proessing tehnique whih is often used (elasti deformation [12], [13]) beause of its low reproduibility and somewhat ontroversial nature. The DRB lassifier has a general arhiteture and is simpler, entirely data-driven and fully automati in omparison to than the DCNN-based approahes, but it is able to perform highly aurate lassifiation on various benhmark problems surpassing the state-of-the-art methods, inluding mainstream deep learning. Its prototype-based nature also allows the training proess to be non-parametri, non-iterative and highly parallelizable sine it onerns only the visual similarity between the identified prototypes and the unlabelled samples. As a result, it is faster by several orders of magnitude, does not require aelerated hardware suh as GPU, HPC and an be ported on hip and still be ontinuously learning. Moreover, thanks to the fat that only the general priniples are involved in the proposed approah, the DRB system an be easily modified and extended to various lassifiation and predition problems. In summary, if ompared with the state-of-the-art approahes, the proposed DRB lassifier has the following unique properties: i) it is free from prior assumptions and user- and problem- speifi parameters; ii) it offers a human-interpretable and self-evolving struture; iii) its training proess is fully online, transparent, non-iterative, non-parametri (it is prototype-based); iv) its training proess an start from srath ; v) its training proess is highly parallelizable; Numerial experiments based on various benhmark image lassifiation datasets (handwritten digits reognition, remote sensing image reognition and objet reognition) demonstrate its exellent performane. The remainder of this paper is organized as follows. Setion 2 introdues the general multi-layer arhiteture of the proposed approah. Setion 3 briefly desribes the feature desriptors involved in the DRB lassifier. The training proess and validation proess of the proposed DRB lassifier are presented in Setion 4. Numerial examples are given in Setion 5, and this paper is onluded by Setion General Arhiteture of the DRB Classifier The general arhiteture of the proposed DRB lassifier is depited in Fig. 1. One an see from the figure that the proposed DRB approah onsists of the following layers:

3 1. Pre-proessing blok; 2. Feature extration layer; Fig.1. General arhiteture of the DRB lassifier. 3. Massively parallel ensemble of highly interpretable IF THEN rules; 4. Deision-maker. The pre-proessing blok of the proposed DRB lassifier involves only the most fundamental and widely used pre-proessing tehniques, namely: i) normalization, iii) saling, ii) rotation and iv) image segmentation. Thus, it is, in fat, omposed of a number of sublayers serving for various purposes. It is well-known that normalization is the proess of linear transformation of the original value range of 0,255 into the range [0,1] [9]. Saling is the proess of resampling and resizing of a digital image [29]. Rotation is a tehnique usually applied to images rotated at a ertain angle around the entre point [9]. Saling and rotation tehniques are two types of affine distortion, and they an signifiantly improve the generalization ability and derease the overfitting [12], [13]. Segmentation is the proess of partitioning an image into smaller piees to extrat loal information or disard the less informative part of the images [9]. The main purpose of the pre-proessing blok within the proposed DRB lassifier is two-fold, namely: i) to improve the generalization ability of the lassifier, and ii) to inrease the effiieny of the feature desriptors in harvesting information from the image. The substrutures of the pre-proessing blok and the usages of the pre-proessing tehniques will be desribed in detail in Setion 5. A more detailed desription of the pre-proessing tehniques we used an also be found in [4]. For the feature extration layer, the proposed DRB lassifier may employ various different kinds of feature desriptors that are used in the field of omputer vision. Different feature desriptors have different advantages and defiienies [44]. In this paper we used two low-level feature desriptors (GIST [37] and HOG [14]) and one high-level feature desriptor (a pre-trained VGG-VD-16 [42]). The details of feature extration are further disussed in Setion 3. As it is demonstrated in setion 5 all three feature desriptors allow the DRB lassifier to ahieve very ompetitive lassifiation rate for various benhmark problems. The third layer of the proposed DRB lassifier is a massively parallel ensemble of IF THEN rules, whih will be desribed in more detail in Setion 4. This is the engine of the DRB lassifier and is based on the autonomously self-developing fuzzy rule-based models of the so-alled AnYa type [7] with singletons in the onsequent part (0-order models; also desribed in [3]). AnYa represents a set of non-parametri IF THEN fuzzy rules that do not require the membership funtion to be pre-defined. Instead, they emerge from the data pattern automatially following the Empirial Data Analytis [6] onept. The struture of a partiular AnYa type fuzzy rule is depited on Fig. 1 as well. As one an see, eah fuzzy rule used in this paper itself is a disjuntion (logial OR operators) of a (potentially, large) number of fuzzy sets formed based on data louds assoiated with the respetive prototypes. The prototypes themselves are being identified using a one-pass type training proess, whih an be massively parallelized if one onsiders eah data loud/prototype as a separate fuzzy rule. The loal deision-maker is a winner-takes-all operator.

4 The final layer is the deision-maker, whih deides the winning lass label based on the partial suggestions of the massively parallel loal/sub-deision makers per IF THEN rule/per lass. This layer is only used during the validation stage and it applies the winner-takes-all priniple as well. As a result, one an see that the proposed DRB lassifier atually uses a two-stage deision-making struture. The validation proess is desribed in Setion 4. For larity, we summarize the key notations of this paper and the respetive definitions in Table I. Notations C d k I x N k, Table I. Definitions of the Key Notations Used in This Paper Definitions The number of lasses of the dataset The dimensionality of the feature vetor The number of the observed training images/urrent time instane A single image The orresponding feature vetor of I The number of identified prototypes of the th lass The global mean of feature vetors of the training images of the th lass I The k th training image of the th lass x k, The orresponding feature vetor of k, P i, The i th prototype of the th lass p i, The mean of feature vetors of the training images assoiated with P i, S i, The number of training images assoiated with P i, r i, The radius of the area of influene of the data loud assoiated with P i, Sg i I The sore of onfidene given by the loal deision-maker of the th fuzzy rule The i th segment of I 3. Feature Extration In this setion, we will briefly desribe the feature desriptors that are employed in the DRB lassifier to make it self-ontained. Feature extration an be viewed as a projetion from the original images to a feature spae that makes the images from different lasses separable, namely, I x. Current feature desriptors an be divided into three ategories based on their desriptive abilities [44], namely: low-level, medium-level and high-level. Different feature desriptors have different advantages. In general, low-level feature desriptors work very well on problems where low-level visual features, e.g., spetral, texture, and struture, play the dominant role. In ontrast, high-level feature desriptors work better on lassifying images with highdiversity and nonhomogeneous spatial distributions beause they an learn more abstrat and disriminative semanti features. In this paper, two low-level feature desriptors (GIST and HOG) are employed, and we further reate a ombination of both to improve their desriptive ability. However, as the low-level feature desriptors are not enough to handle effiiently omplex, large-sale problems, we also use one of the most widely used high-level feature desriptors (a pre-trained VGG-VD-16 [42]). It has to be stressed that the high-level feature desriptor is diretly used without further tuning and is a part of the pre-proessing layer. As there is no interdependene of different images within the feature extration stage, it an be parallelized massively to further redue the proessing time. One the global features (either low- or high-level) of the image are extrated and stored, there is no need to repeat the same proess again. We also have to stress that this paper desribes a general DRB approah and the feature desriptors are not neessarily limited to GIST or HOG or the pre-trained VGG-VD-16 only. Alterative feature desriptors an be used, i.e. CaffeNet [22], SIFT [34], et., and further ombinations of different visual features an also be onsidered as well. One may further onsider to refine the ommonly used visual features into more informative representations by unovering an appropriate latent subspae [30]. However, seleting the most suitable feature desriptor(s) for a partiular problem requires prior knowledge about the problem, and this is out of the sope of this paper.

5 3.1. Employed Low-Level Feature Desriptors A. GIST Desriptor GIST feature desriptor gives an impoverished and oarse version of the prinipal ontours and textures of an image [38]. In the proposed DRB lassifier, we use the same GIST desriptor as desribed in [38] without any modifiation, whih extrats a dimensional feature vetor denoted by g I g 1I, g2i,..., g512 I. B. HOG Desriptor HOG desriptor [14] has been proven to be very suessful in various omputer vision tasks, suh as objet detetion, texture analysis and image lassifiation. In the DRB lassifier, although the size of the images varies for different problems, we used the default blok size of 2 2 and hanged the ell size to fix the dimensionality of the HOG features to be 1 576, denoted by h I = h 1 I, h2 I,..., h576 I. To improve the distintiveness of the HOG feature vetors of images between different lasses, we expand the value range of the HOG vetors by the following nonlinear nonparametri funtion [4],[5]: 2 sgn(1 ) exp 1sgn(1 ) 1 exp 1 x x x x (1) 1, x 0 where sgn( x) 0, x 0 1, x 0 C. Combined GIST-HOG Features, and the nonlinearly mapped HOG feature vetor of I is denoted by I h. To further improve the desriptive ability of the GIST and HOG feature desriptors, in this paper, we further ombine the GIST and HOG feature vetors to reate a new, more desriptive integrated vetor as follows: f I g g I I, h I hi (2) where denotes the norm Employed High-Level Feature Desriptor The VGG-VD-16 [42] is urrently one of the best performing pre-trained DCNN feature desriptors widely used in different works. It has a simpler struture, but is able to provide better performane on various problems. We use the pre-trained VGG-VD-16 model as the high-level feature desriptor without any tuning to enhane the ability of the proposed DRB lassifier in handling omplex, large-sale, high-density image lassifiation problems. Following the ommon pratie, the dimensional ativations from the first fully onneted layer are extrated as the feature vetor of the image I, denoted by v I v1 I, v2 I,..., v4096 I. However, as the pre-trained model requires the input image to be the size of pixels, it is, in fat, not good in handling problems with small-size images with simple semanti ontents. 4. Massively Parallel Fuzzy Rule Base In the DRB lassifier, we employ a non-parametri rule-base formed of 0-order AnYa type fuzzy rules [3],[7], whih makes the proposed lassifier interpretable and transparent for human understanding (even to a non-expert) unlike the mainstream deep learning [10],[12],[13],[18],[20],[21],[26],[42]. Beause of the prototype-based nature, the DRB lassifier is free from prior assumptions about the type of distribution as well as the random or deterministi nature of data [6], the requirements of setting ad ho model struture, handrafting membership funtions, et. Meanwhile, the prototype-based nature further allows the DRB lassifier a non-parametri, non-iterative, self-organising, self-evolving and highly parallel underlying struture

6 [2],[7], [8]. Thus, the training of the proposed DRB lassifiers is fully autonomous, signifiantly faster and an start from srath [2]. As desribed in more detail later in this setion as well as in [3] and [6], the system automatially identifies prototypes from the empirially observed data (images) and forms data louds (luster-like groups of points with no predetermined shape) resembling Voronoi tessellation [37] per lass. Thus, for a training dataset, whih onsists of C lasses, C independent 0-order IF THEN FRB subsystems are generated (one per lass) in parallel. One the training proess is finished, eah subsystem generalizes/learns one 0-order AnYa type fuzzy rule orresponding to its own lass based on the identified prototypes: I P,1 I ~ P, IF ~ OR OR THEN lass (3) N where ~ denotes similarity, whih an also be seen as a fuzzy degree of satisfation/membership [7] or typiality [6]; I is a partiular image and x is its orresponding feature vetor; x an be f I or I g I, I h, v ; P is the j th visual prototype of the th lass; p, j, jis the orresponding feature vetor of P, j and has the same dimensionality as x ; j 1, 2,..., N ; N is the number of prototypes of the th lass. 1,2,..., C. Examples of AnYa type fuzzy rules generalized from the popular handwritten digits reognition problem, MNIST dataset [27] for digits 2, 3, 5 and 8 are visualized in Table II. As we an see, AnYa type fuzzy rules in the table provide a very intuitive representation of the mehanism. Moreover, eah of the AnYa type fuzzy rules an be interpreted as a number of simpler fuzzy rules with single prototype onneted by OR operator. As a result, a massive parallelization is possible. Table II. Illustrative Example of AnYa Fuzzy Rules with MNIST Dataset Fuzzy Rules IF (I~ ) OR (I ~ ) OR (I ~ ) OR (I ~ ) OR OR (I ~ ) OR (I ~ ) THEN (digit 2) IF (I ~ ) OR (I ~ ) OR (I ~ ) OR (I ~ ) OR OR (I ~ ) OR (I ~ ) THEN (digit 3) IF (I ~ ) OR (I ~ ) OR (I ~ ) OR (I ~ ) OR OR (I ~ ) OR (I ~ ) THEN (digit 5) IF (I ~ ) OR (I ~ ) OR (I ~ ) OR (I ~ ) OR OR (I ~ ) OR (I ~ ) THEN (digit 8) In the remainder of this setion, we will desribe the training and validation proesses as well as the deision-making mehanism of the proposed DRB lassifier Training of the DRB System Due to the highly parallel struture of the proposed system, in this subsetion, we summarize the main th proedure of the training proess of a single FRB subsystem, namely the one. Stage 0: System Initialization The th FRB subsystem is initialized by the first image of the th lass, I,1. We firstly apply the vetor normalization to the global feature vetor of I,1, denoted by x,1 ( x,1 x,1,1, x,1,2,..., x,1, dimensionality):,1,1,1 d, d is the x x x (4) With the vetor normalization, the Eulidean distane between two normalized data samples z y y an be onverted to osine dissimilarity as follows: 2 1 os zy, z and z z y y, where z, y is the angle between z and y. The vetor normalization operation helps to overome the so-alled urse of dimensionality [1].

7 Then, the meta-parameters of the system are initialized as follows: k 1; x ; N 1; P I ; p x ; S 1; r r ; (5),1, N,1, N,1, N, N o th is the global mean of all the observed data samples of the lass; p is the mean of feature vetors of the images assoiated with the first data loud with the visual prototype where k is the urrent time instane; N, P ; N, S is the number of images assoiated with the data loud; N, r is the radius of the area of the data N, loud; r o is a small value to stabilize the initial status of the newly formed data louds. Data louds are very muh like lusters, but are nonparametri and do not have a speifi pre-determined, regular shape. They diretly represent the loal ensemble properties of the observed data samples [7]. In this paper, we use 21 os(30 o o ) r to define the degree of similarity on the edge of the data loud. We need to stress that, r o is not a problem-speifi parameter and requires no prior knowledge to be determined Stage 1: Preparation For the newly arrived th k ( k k 1 ) training image that belongs to the firstly apply the vetor normalization (expression (4)) to its orresponding feature vetor: the global mean, is updated as follows: th lass, denoted by k, x k, x x k, k, I we. Then, k 1 1 x (6), k k k And we alulate the data densities of all the existing prototypes P i, ( i 1,2,..., N, where N is the number of identified prototypes) as detailed in [6]: D P 1 i, p, i as well as the data density of the new image I k, : (7a) D I 1 k, x, k (7b) where X 1 ; X is the average norm of the observed normalized data samples, whih is always equal to 1 due to the vetor normalization operation. Stage 2: System Update In this stage, we update the system struture and meta-parameters to aommodate the newly arrived image. Firstly, Condition 1 is heked to see whether I k, beomes a new prototype: Condition 1: I, max P, I, min P, k j1,2,..., N i k j1,2,..., N i THEN Ik, is a new prototype IF D D OR D D One Condition 1 is satisfied, I k, is set to be a new prototype and it initializes a new data loud: N N 1; P I ; p x ; S 1; r r ; (9), N, k, N, k, N, N o If Condition 1 is not met, we find the nearest prototype to I k,, denoted by P n,, using equation (10):, n, k, j j1,2,..., N P arg min x p (10) (8)

8 Before we assoiate I k, with the data loud of P n,, Condition 2 is heked to see whether I k, loates in the area of influene of P n, : Condition 2: IF, k, n r, N THEN I, k is assigned to P, n x p (11) If Condition 2 is met, I k, is assigned to the data loud formed around the prototype P n, and the metaparameters of this data loud are updated as follows: S S S 1; p p x ; r r ; (12) n, 2 2 2, n, n, n, n, k, n, n, n S, n S, n where 1 p., n, n 2 Otherwise, it means that I k, is out of the influene area of the nearest data loud, and, therefore, a new data loud is initialized by I k, with I k, as its prototype ( N N 1 ). The meta-parameters of the new data loud are, then, added using expression (9). Then, the next image is grabbed at Stage 1. After all the training samples have been proessed, the system goes to the final stage and generates the AnYa type fuzzy rule. Stage 3: Fuzzy Rules Generation One the training proess has been finished, the system will generate one AnYa fuzzy rule based on the identified prototypes: Rule : IF I P OR OR I P THEN lass ~ ~ (13),1, N If more training samples are available later, the FRB subsystem an ontinue the proessing yle from Stage 1 and update the fuzzy rules aordingly. The flowhart of the training proess of the FRB subsystem is depited in Fig. 2.

9 Fig. 2. Flowhart of the training proess of the FRB subsystem 4.2. Classifying with the Identified FRB System After the identifiation proedure, the FRB system generates C fuzzy rules in regards to the C lasses. For eah testing image I, eah one of the C fuzzy rules will generate a sore of onfidene I by its loal (per rule) deision-maker based on the feature vetor of I, denoted by x : I x p (14) 2 arg max exp, j j1,2,..., N

10 1, 2,..., C per image, whih are the inputs of the overall deision-maker of the DRB lassifier. As a result, one an get C sores of onfidene I I I I 4.3. Deision-Making Mehanism For a single FRB system, the overall deision-maker (the last layer in Fig. 1) deides the label of the validation image using the winner-takes-all priniple as follows: 1,2,..., C label I arg max I (15) In some appliations, i.e. fae reognition, remote sensing, objet reognitions, et., where loal information may play a more important role than the global information, one an onsider segmenting (both the training and validation) images to apture loal information. In suh ases, the 0-order FRB subsystems are trained with segments of training images instead of the full images. The overall label of a validation image is given as an integration of all the sores of onfidene that the FRB subsystems assoiated with its segments, denoted by Sg 1, Sg 2,, Sg T : T 1 label I arg max Sg i (16) 1,2,..., C T i1 If an FRB ensemble [23] is used, the label of the validation image is onsidered as the integration of all the sores of onfidene that the FRB systems given to the image [4]: K 1 label I arg max, i I max, i I (17) 1,2,..., C K i1,2,..., K i1 where K is the number of FRB systems in the ensemble. 5. Numerial Examples and Disussions In this setion, we study the performane of the proposed DRB lassifier. All the numerial examples are onduted using Matlab2017a on a PC with dual ore i7 proessor with lok frequeny 3.4GHz eah and 16GB RAM. To illustrate the proposed DRB lassifier, we onsider the following four well-known benhmark datasets overing three different hallenging problems: 1) MNIST dataset for handwritten digits reognition [27]; 2) Singapore dataset for remote sensing [17]; 3) UCMered dataset for remote sensing [47]; 4) Calteh101 dataset for objet reognition [16]. We then ompare the results with the state-of-the-art approahes. As the four benhmark datasets are very different from eah other, we will use four different, but same as in the publiations [17], [18], [27] experimental protools for eah dataset, respetively MNIST Dataset The MNIST dataset [27] is a famous benhmark database for handwritten digits reognitions that ontains grey images (60000 of them form the training set and are used for validation) of handwritten digits ( 0 to 9 ). The image size is for both the training and validation images. There is a large number of publiations reporting highly aurate results. However, due to the fat that the dataset itself has flaws, there are a number of validation images with unreognizable digits even for humans (see Fig. 3), the testing auray is below 100%, although losely approahing it, see table III.

11 Fig.3. The only 56 mistakes made by the DRB Ensemble out of validation images The detailed arhiteture of the proposed DRB for handwritten digits reognition for the training proess is shown in Fig.4. The arhiteture for the validation proess is given in Fig.5. Fig.4. Arhiteture for training (handwritten digits reognition) Fig.5. Arhiteture for validation (handwritten digits reognition) The pre-proessing blok of the proposed DRB lassifier for handwritten digits reognition onsists of the following layers, where we adopt the same rotation and saling operation as used in referenes [12], [13] but without using elasti distortion:

12 1. Normalization layer, whih applies linear normalization to fit the original pixel value range of 0, 255 0, 1. into the range of 2. Saling layer, whih resizes the validation images from their original size of into 7 (S=7) different sizes: i) 28 22, ii) 28 24, iii) 28 26, iv) 28 28, v) 28 30, vi) and vii) Rotation layer, whih rotates the images by 11 ( R 11) different angles i) -15 o, ii) -12 o, iii) -9 o, iv) -6 o, v) -3 o, vi) 0 o, vii) 3 o, viii) 6 o, ix) 9 o, x) 12 o and xi) 15 o. 4. Segmentation layer, whih extrats the entral area ( ) from the training images. It disards the borders that onsist mostly of white pixels with little or no information. The saling and rotation layers reate 77 ( SR 77 ) new training sets from the original one with respet to different saling sizes and rotation degrees [4]. As a result, we will train 77 DRB systems in regards to the 77 new training sets and later form an ensemble. Eah DRB system onsists of 10 AnYa type 0-order fuzzy rules with a large number of prototypes onneted with a disjuntion (Logial OR ) as shown in Table II, orresponding to digits 0 to 9. For eah validation image, we just apply the normalization and segmentation operations. Sine the images within the MNIST dataset are quite small and simple, high-level feature desriptors are not suitable for this problem. Therefore, the feature desriptor used by the DRB lassifier in this experiment is GIST, HOG or the ombined GIST and HOG (CGH) features. However, due to the different desriptive abilities of these features, the performane of the DRB lassifier is somewhat different. The reognition auray of the proposed DRB lassifier using different feature desriptors is tabulated in Table III. The orresponding average training times for the 10 fuzzy rules are tabulated in Table IV. Approahes Auray Training Time Table III. Comparison between the Proposed Approah and the State-of-the-Art Approahes DRB- GIST % DRB- HOG % DRB- CGH % DRB Ensem ble % Less than 2 minute for eah part DRB Casade [5] Large Convolution al Networks [40] Large Convolution al Networks [21] 99.55% 99.40% 99.47% Committee of 7 Convolution al Neural Networks [12] 99.73% 2% Committee of 35 Convolution al Neural Networks [13] 99.77% Almost 14 hours for eah one of the DNNs. No No PC-Parameters Core i (3.60GHz), 16 GB DDR3 Core i7-920 (2.66GHz), 12 Information Information GB DDR3 GPU Used None 2 GTX 480 & 2 GTX 580 Elasti Distortion No No No Yes Tuned Parameters No Yes Yes Yes Iteration No Yes Yes Yes Randomness No Yes Yes Yes Parallelization Yes No No No Evolving Ability Yes No No No By further reating a DRB ensemble onsisting of a DRB lassifier trained with GIST features and a DRB lassifier trained with HOG features, we ahieve a better reognition performane, whih is tabulated in Table III as well. In our previous work, we also proposed a DRB asade [5] that further improves the reognition auray by using a SVM for onflit resolution, whih is also presented in Table III. The onflit resolution only applies to a small number (about 5%) of the validation data for whih the two highest onfidene values are lose to eah other and thus there may be two possible winners with similar overall sores [5]. One of the

13 important advantages of the proposed DRB lassifier is that it provides in a lear and expliit form per rule/lass onfidene level. The only 56 images that are inorretly reognized by the proposed DRB ensemble are depited in Fig. 3 and the orresponding labels are given on top of these images. As we an see, none of these digits is written learly and the majority of them are far different from the normal handwriting styles. One of the most distintive advantages of the proposed DRB lassifier is its evolving ability, whih means that there is no need for omplete re-training of the lassifier when new data samples are available. To illustrate this advantage, we train the DRB lassifier with images in the form of an image stream (video). Meanwhile, the exeution time and the reognition auray are reorded during the proess. In this example, we use the original training set without resaling or rotation, whih speeds up the proess signifiantly. The relationship urves of the training time (the average for eah of the 10 fuzzy rules) and reognition auray with the growing amount of the training samples are depited in Fig. 6. Table IV. Computation Time for the Learning Proess per Sub-system (in seonds) Fuzzy Rule # Digital GIST Feature HOG CGH (a) Auray (b) Training time Fig.6.The relationship urve of training time and reognition auray with different amount of training samples In order to evaluate the performane of the proposed DRB lassifier, we also present the state-of-the-art approahes reporting the urrent best and the seond best published results (with and without elasti distortion) worldwide in Table III. As we an see, the approahes reported in [12], [13] using elasti distortion an ahieve slightly better results than the approahes [21], [40] as well as the proposed DRB lassifier. However, this omes at a prie of using elasti distortion. This kind of distortion exhibits a signifiant randomness that may turn an unreognizable digit into a reognizable one and vie versa, whih also asts doubt on the effetiveness of the approahes in real-world appliations. In addition, elasti distortion puts in question the ahieved results repeatability and requires a ross-validation that further obstruts online appliations and the reliability of the results [4]. Without using elasti distortion, the urrent published best result is 99.47% [21], whih is omparable with the proposed DRB ensemble, but worse than the DRB asade [5]. However, one needs to notie that the DCNNs require a large number of parameters (tens or hundreds of millions) to be optimized, hugely longer time and more omplex aelerated hardware, annot start from srath, annot evolve with the data stream and are not human-interpretable.

14 5.2. Singapore Dataset Singapore dataset was onstruted from a large satellite image of Singapore [17]. This dataset onsists of 1086 images with pixels size with 9 sene ategories: i) airplane, ii) forest, ii) harbor, iv) industry, v) meadow, vi) overpass, vii) residential, viii) river, and ix) runway. Examples of images of the 9 lasses are given in Fig.7. Fig.7. Examples of images from the Singapore dataset The arhiteture of the proposed DRB lassifier, as shown in Fig.8, onsists of the following layers: 1. Normalization layer; 2. Rotation layer, whih rotates the images by i) 0 o, ii) 90 o, iii) 180 o and iv) 270 o to improve the generalization ability of the lassifier. 3. Segmentation layer, whih splits eah image into smaller piees by a size sliding window with the step size of 32 pixels in both horizontal and vertial diretions. The segmentation layer uts one image into 49 piees. 4. Feature desriptor, whih extrats the ombined GIST and HOG features from eah segment. 5. FRB system, whih onsists of 9 fuzzy rules, eah of them is trained based on the segments of images of a partiular lass within the dataset. 6. Deision-maker, whih generates the labels using equation (16). Fig.8. Arhiteture for remote sensing (with low-level feature desriptors) Following the ommonly used experimental protool [17], we firstly transform the images into grey-level ones and train the proposed DRB lassifier with randomly seleted 20% of images of eah lass and use the remainder as a validation data set. The experiment is repeated 5 times and the average auray is reported in Table V. Visual examples of the extrated IF THEN rules per lass during experiments are given in Table VI. The performane of the proposed DRB is also ompared with the state-of-the-art approahes as follows: 1. Transfer Learning with Deep Representations (TLDP) [41]; 2. Two-Level Feature Representation (TLFP) [17];

15 3. Bag of Visual Words (BoVW) [47]; 4. Sale-Invariant Feature Transform with Sparse Coding (SIFTSC) [11]; 5. Spatial Pyramid Mathing Kernel (SPMK) [25]. and the reognition auraies of the omparative approahes are reported in Table V as well. One an see that, the proposed approah is able to produe a signifiantly better reognition result than the best urrent methods. Furthermore, by using a smaller step size, the DRB lassifier an grasp more details, and this leads to a better reognition performane. Table V. Comparison between the Proposed Approah and the State-of-the-Art Approahes Method Auray (%) TLDP [41] TLFP [17] BoVW [47] SIFTSC [11] SPMK [25] DRB-GCH DRB-VGG Table VI. Visual Examples of the AnYa Type Fuzzy Rules Fuzzy Rules IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Airplane) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Forest) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Harbour) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Industry) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Meadow) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Overpass) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Residential) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (River) IF (Sg ~ ) OR (Sg ~ ) OR (Sg ~ ) OR OR (Sg ~ ) THEN (Runway) To show the evolving ability of the proposed DRB lassifier, we randomly selet out 20% of the images of eah lass for validation and train the DRB lassifier with 10%, 20%, 30%, 40%, 50%, 60%, 70% and 80% of the dataset. The experiment is repeated five times and the average auray is tabulated in Table VII. The

16 average time for training is also reported, however, due to the unbalaned lasses, the training time as tabulated in Table VII is the overall training time of the nine fuzzy rules. As handwritten digits images in the MNIST dataset are muh simpler, the low-level feature desriptors are suffiient for problems of this type. In ontrast, remote sensing images have more fine details and a variety of semanti ontents. Therefore, we further introdue the high-level feature desriptor, namely, the pre-trained VGG-VD-16 model, into the DRB lassifier and use the original RGB remote sensing images for training. The arhiteture of the DRB lassifier is adjusted as depited in Fig. 9 to aommodate the high-level feature desriptor. As one an see, the adjusted DRB lassifier is different from the one using low-level feature desriptors in terms of the following layers: 1. Segmentation layer, whih splits eah image into smaller piees by a pixels size sliding window with the step size of 64 pixels in both horizontal and vertial diretions. The segmentation layer uts one image into 4 piees. 2. Saling layer, whih resizes the image segments into the size of pixels; 3. Feature desriptor, whih extrats a dimensional feature vetor from eah segment; And the rotation layer, FRB layer and deision makers are the same as shown in Fig. 8. Then, the experiments in Tables V and VII are repeated using the same experimental protool, and the new results are tabulated in the respetive Tables. Table VII. Results with Different Amount of Training Samples CGH VGG CGH VGG Ratio 10% 20% 30% 40% Auray (%) Time (in seonds) Auray (%) Time (in seonds) Ratio 50% 60% 70% 80% Auray (%) Time (in seonds) Auray (%) Time (in seonds) Fig.9. Arhiteture for remote sensing (with high-level feature desriptors) From the above experiments one an see that by using the high-level feature desriptor, both the reognition auray and the omputational effiieny of the DRB lassifier on the remote sensing problem are signifiantly boosted.

17 5.3. UCMered Dataset UCMered dataset [47] onsists of fine spatial resolution remote sensing images of 21 hallenging sene ategories (inluding airplane, beah, building, et.). Eah ategory ontains 100 images of the same image size ( pixels). The example images of the 21 lasses are shown in Fig.10. Following the ommonly used experimental protool [17], we randomly selet 80% of images of eah lass for training and use the remainder as a validation set. The experiment is repeated 5 times and the average auray is reported in Table VIII. In this experiment, we use the same arhiteture as depited in Fig. 9. The performane of the proposed DRB is also ompared with the state-of-the-art approahes as follows: 1. Two-Level Feature Representation (TLFP) [17]; 2. Bag of Visual Words (BoVW) [47]; 3. Sale-Invariant Feature Transform with Sparse Coding (SIFTSC) [11]; 4. Spatial Pyramid Mathing Kernel (SPMK) [25],[48]; 5. Multipath Unsupervised Feature Learning (MUFL) [15]; 6. Random Convolutional Network (RCNet) [50]; 7. Linear SVM with Pre-Trained CaffeNet (SVM+Caffe) [39]; 8. LIBLINEAR Classifier with the VGG-VD-16 Features (LIBL+VGG) [44]; 9. Linear SVM with the VGG-VD-16 Features (SVM+VGG). Fig.10. Example Images from the UCMered dataset Table VIII. Comparison between the Proposed Approah and the-state-of-the-art Approahes Approah Auray Approah Auray TLFR [17] 91.12% RCNet [50] 94.53% BoVW [47] 76.80% SVM+ Caffe [39] 93.42% SIFTSC [11] 81.67% LIBL+VGG [44] 95.21% SPMK [48] 74.00% SVM+VGG 94.48% MUFL [15] 88.08% DRB 96.14% From the omparison given in Table VIII one an see that, the proposed DRB lassifier, again, produed the best lassifiation performane. Similarly, we randomly selet out 20% of the images of eah lass for validation and train the DRB lassifier with 10%, 20%, 30%, 40%, 50%, 60% and 70% of the dataset. The experiment is repeated 5 times, and the average auray and time required for training (per rule) are tabulated in Table IX. One an see from Table X that the DRB lassifier an ahieve 95%+ lassifiation auray with less than 20 seonds for training eah fuzzy rule in addition to the highly interpretable struture and ability to ontinue to learn and evolve automatially.

18 Table IX. Results with Different Amount of Training Samples Ratio 10% 20% 30% 40% Auray (%) Time (in seonds) Ratio 50% 60% 70% 80% Auray (%) Time (in seonds) Calteh101 Dataset Calteh 101 dataset [16] ontains 9144 pitures of objets belonging to 101 ategories plus one bakground ategory. The number of images in eah lass varies from 33 to 800. The size of eah image is roughly pixels. This data set ontains both lasses orresponding to rigid objet (like bikes and ars) and lasses orresponding to non-rigid objet (like animals and flowers). Therefore, the shape variane is signifiant. The examples of this dataset are presented in Fig. 11. The arhiteture of the DRB lassifier for objet reognition is depited in Fig. 12, whih is the same as the latter part of the DRB lassifier for remote sensing problems as presented in Fig. 9. The images of the Calteh 101 dataset [16] are very uniform in presentation, aligned from left to right, and usually not oluded, therefore, the rotation and segmentation are not neessary. Following the ommonly used protool [18], we ondut the experiments by seleting 15 and 30 training images from eah lass and using the rest for validation. The experiment is repeated 5 times and the average auray is reported in Table X. We also ompare the DRB lassifier with the state-of-the-art approahes as follows: 1. Convolutional Deep Belief Network (CBDN) [28]; 2. Learning Convolutional Feature Hierarhies (CLFH) [23]; 3. Deonvolutional Networks (DECN) [49]; 4. Linear Spatial Pyramid Mathing (LSPM) [45]; 5. Loal-Constraint Linear Coding (LCLC) [43]; 6. DEFEATnet [18]; 7. Convolutional Sparse Autoenoders (CSAE) [35]; 8. Linear SVM with the VGG-VD-16 Features (SVM+VGG). Fig.11. Example images of the Calteh 101 dataset As one an see from Table X that the DRB lassifier easily outperforms all the omparative approahes in the objet reognition problem. Same as the previous example, we randomly selet out 1, 5, 10, 15, 20, 25, and 30 images of eah lass for training the DRB lassifier and use the rest for validation. The experiment is

19 repeated 5 times, and the average auray and time onsumption for training (per rule) are tabulated in Table XI, where we an see that, it only requires less than 2 seonds to train a single fuzzy rule. Fig.12. Arhiteture for objet reognition Table X. Comparison between the Proposed Approah and the State-of-the-Art Approahes Approah Auray (%) 15 Training 30 Training CBDN [28] CLFH [23] DECN [49] LSPM [45] LCLC [43] DEFEATnet [18] CSAE [35] SVM+VGG DRB Table XI. Results with Different Amount of Training Samples Training Number Auray (%) Time (in seonds) Conlusion and Future Work In this paper, a new powerful multilayer fuzzy rule-based (DRB) lassifier for image reognition problems is proposed. It is a neuro-fuzzy arhiteture, but we stress its IF THEN highly interpretable rule-base aspet. Thanks to its prototype-based nature, the proposed approah an self-organize a transparent and human understandable fuzzy rule-based (FRB) system struture in a highly effiient way starting from srath. Its one-pass type training proess is non-parametri, entirely data-driven and fully automati. Without any iteration, the DRB lassifier is able to offer extremely high lassifiation, omparable with human abilities and on par or surpassing best published mainstream deep learning alternatives. The proposed DRB lassifier is a general approah for various problems and serves as a strong alternative to the state-of-the-art approahes by providing a fully human-interpretable struture after a very fast (in orders of magnitude faster than the mainstream deep learning methods), transparent, nonparametri training proess. Numerial examples on four well-known benhmark datasets demonstrate the exellent performane and strong advantages of the proposed approah. As future work, we are partiularly interested in applying the DRB system on human fae reognition problems. We will also apply the DRB system to other image proessing problems and heterogeneous lassifiation problems where the data are oming in different form (images/video, text/natural language as well as signals/physial variables). The onvergene of the DRB systems will be studied as well. We are further interested to study ollaborative senarios whereby a set of distributed DRB lassifiers exhange prototypes. Finally, we will also study the loal optimality of the lassifier struture.

20 Referene [1] C. C. Aggarwal, A. Hinneburg, and D. A. Keim, On the surprising behavior of distane metris in high dimensional spae, in International Conferene on Database Theory, 2001, pp [2] P. Angelov, Autonomous learning systems: from data streams to knowledge in real time. John Wiley & Sons, Ltd., [3] P. P. Angelov and X. Gu, Autonomous learning multi-model lassifier of 0-order (ALMMo-0), in IEEE International Conferene on Evolving and Autonomous Intelligent Systems, 2017, pp [4] P. P. Angelov and X. Gu, MICE: Multi-layer multi-model images lassifier ensemble, in IEEE International Conferene on Cybernetis, 2017, pp [5] P. Angelov and X. Gu, A asade of deep learning fuzzy rule-based image lassifier and SVM, in International Conferene on Systems, Man and Cybernetis, 2017, pp [6] P. P. Angelov, X. Gu, and J. Prinipe, A generalized methodology for data analysis, IEEE Trans. Cybern., DOI: /TCYB , [7] P. Angelov and R. Yager, A new type of simplified fuzzy rule-based system, Int. J. Gen. Syst., vol. 41, no. 2, pp , [8] P. Angelov and X. Zhou, Evolving fuzzy-rule based lassifiers from data streams, IEEE Trans. Fuzzy Syst., vol. 16, no. 6, pp , [9] R. G. Casey, Moment Normalization of Handprinted Charaters, IBM J. Res. Dev., vol. 14, no. 5, pp , [10] K. Charalampous and A. Gasteratos, On-line deep learning method for ation reognition, Pattern Anal. Appl., vol. 19, no. 2, pp , [11] A. M. Cheriyadat, Unsupervised feature learning for aerial sene lassifiation, IEEE Trans. Geosi. Remote Sens., vol. 52, no. 1, pp , [12] D. Cireşan, U. Meier, L. M. Gambardella, and J. Shmidhuber, Convolutional neural network ommittees for handwritten harater lassifiation, in International Conferene on Doument Analysis and Reognition, 2011, vol. 10, pp [13] D. Ciresan, U. Meier, and J. Shmidhuber, Multi-olumn deep neural networks for image lassifiation, in Conferene on Computer Vision and Pattern Reognition, 2012, pp [14] N. Dalal and B. Triggs, Histograms of oriented gradients for human detetion, in IEEE Computer Soiety Conferene on Computer Vision and Pattern Reognition, 2005, pp [15] J. Fan, T. Chen, and S. Lu, Unsupervised feature learning for land-use sene reognition, IEEE Trans. Geosi. Remote Sens., vol. 55, no. 4, pp , [16] L. Fei-Fei, R. Fergus, and P. Perona, Learning generative visual models from few training examples: an inremental Bayesian approah tested on 101 objet ategories, Comput. Vis. Image Underst., vol. 106, no. 1, pp , [17] J. Gan, Q. Li, Z. Zhang, and J. Wang, Two-level feature representation for aerial sene lassifiation, IEEE Geosi. Remote Sens. Lett., vol. 13, no. 11, pp , [18] S. Gao, L. Duan, and I. W. Tsang, DEFEATnet A deep onventional image representation for image lassifiation, IEEE Trans. Ciruits Syst. Video Tehnol., vol. 26, no. 3, pp , [19] S. Gao, Y. Zhang, K. Jia, J. Lu, and Y. Zhang, Single Sample Fae Reognition via Learning Deep Supervised Auto-Enoders, IEEE Trans. Inf. Forensis Seur., vol. 6013, no., pp. 1 1, [20] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. Crambridge, MA: MIT Press, [21] K. Jarrett, K. Kavukuoglu, M. Ranzato, and Y. LeCun, What is the best multi-stage arhiteture for objet reognition?, in IEEE International Conferene on Computer Vision, 2009, pp [22] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshik, S. Guadarrama, and T. Darrell, Caffe: onvolutional arhiteture for fast feature embedding, in ACM International Conferene on Multimedia, 2014, pp [23] K. Kavukuoglu, P. Sermanet, Y.-L. Boureau, K. Gregor, M. Mathieu, and Y. LeCun, Learning onvolutional feature hierarhies for visual reognition, in Advanes in neural information proessing systems, 2010, pp [24] L. Kunheva, Combining pattern lassifiers: methods and algorithms. Hoboken, New Jersey: John Wiley & Sons, 2004.