742 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 13, NO. 6, DECEMBER Dong Zhang, Luo-Feng Deng, Kai-Yuan Cai, and Albert So

Transcription

1 742 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 Fuzzy Nonlinear Regression With Fuzzified Radial Basis Function Network Dong Zhang, Luo-Feng Deng, Kai-Yuan Cai, and Albert So Abstract A fuzzified radial basis function network (FRBFN) is a kind of fuzzy neural network that is obtained by direct fuzzification of the well known neural model RBFN A FRBFN contains fuzzy weights and can handle fuzzy-in fuzzy-out data This paper shows that a FRBFN can also be interpreted as a kind of fuzzy expert system Hence it owns the advantages of simple structure and clear physical meaning Some metrics for fuzzy numbers have been extended to the metrics for -dimensional fuzzy vectors, which are applicable to computations in FRBFNs The corresponding metric spaces for -dimensional fuzzy vectors are proved to be complete Further, FRBFNs are proved to be able to act as universal function approximators for any continuous fuzzy function defined on a compact set This paper applies the proposed FRBFN to nonparametric fuzzy nonlinear regression problems for multidimensional LR-type fuzzy data Fuzzy nonlinear regression with FRBFNs can be formulated as a nonlinear mathematical programming problem Two training algorithms are proposed to quickly solve the two types of problems under different criteria and constraint conditions, namely, the two-stage and BP (Back-Propagation) training algorithms Simulation studies are carried out to verify the feasibility and demonstrate the advantages of the proposed approaches Index Terms Fuzzified radial basis function network (FRBFN), fuzzy neural network, fuzzy number, fuzzy regression, universal approximation I INTRODUCTION REGRESSION analysis is a set of methods that are used in evaluating the functional relationship between the dependent and independent variables and also in determining the best-fit model for describing the relationship, by exploiting the knowledge from the given input output data pairs The deviations between the observed values (from the data set) and the estimated values (from a regression model) can occur due to the measurement errors and/or modeling errors With the modeling errors ignored, the deviations are supposed to be random in classical regression analysis Thus, statistical methods have been employed for estimation and evaluation in this case Fuzzy regression was first proposed by Tanaka et al for linear case in [1] In fuzzy regression analysis, the deviations are attributed to the imprecision of the observed values and/or the indefiniteness of model structure In this case, the observed values can differ from the estimated values to a certain degree of belief Thus fuzzy theory or possibility theory is used there Manuscript received May 9, 2003; revised May 19, 2004 and February 24, 2005 D Zhang, L-F Deng, and K-Y Cai are with the Department of Automatic Control, Beihang University, Beijing , China ( Zhangdongde@sohucom) A So is with the Department of Building and Construction, City University of Hong Kong, Hong Kong, China Digital Object Identifier /TFUZZ Fuzzy linear regression (FLR) assumes the linear fuzzy model for describing the functional relationship between data pairs In nonlinear case, this assumption may leadto large modeling errors Hence fuzzy nonlinear regression methods [6], [11], [16], [24] were suggested to overcome the deficiencies of FLR methods In practice, however, the nonlinear functional relationship between input output data pairs is frequently unknown There is no systematic approach to determine an appropriate regression model in advance For this reason we need some universal regression models The fuzzy nonlinear regression using universal regression models is also referred to as the nonparametric fuzzy nonlinear regression in the literature The key point here is to select a good universal regression model for applications By a good universal regression model we mean that it should C1) have strong approximation ability for nonlinear fuzzy functions; C2) be suitable for developing efficient optimization algorithms; C3) be simple and be easy to interpret intuitively Fuzzy neural networks can be adopted as universal regression models The existing fuzzy neural networks can be roughly classified into the following three categories: 1) fuzzy neural networks by direct fuzzification of the regular neural networks; 2) fuzzy neural networks by network realization of fuzzy expert systems; 3) fuzzy neural networks by incorporation of fuzzy operations, like t-norms or t-conorms The kind of fuzzy neural networks still process real number signals and have real number weights The fuzzy neural networks of the first category are especially called fuzzified neural networks in this paper Among them, the most often used one is the fuzzified multilayer feedforward neural network, which was obtained by fuzzification of the famous regular neural model multilayer perceptron (MLP) This kind of model was investigated in [11] [17] and is especially called fuzzified MLP in this paper The computation in such a model is based on fuzzy arithmetic by using the extension principle According to the survey of Buckley [20], many training algorithms have been suggested to determine the weights of fuzzified MLPs, including fuzzy back-propagation, -cut backpropagation, random search, genetic algorithms, etc Fuzzified MLPs were already applied to fuzzy nonlinear regression by Ishibuchi et al in [11] and [16] However, a fuzzified MLP is not a good universal regression model by examining the three criteria C1) C3) First, Buckley [21] indicated that the set of fuzzified MLPs can not act as a universal approximator for continuous fuzzy functions, which /$ IEEE

2 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 743 means that the approximation ability of fuzzified MLPs is not strong Secondly, the existing training algorithms for fuzzified MLPs are inclined to have slow convergence rate since it is difficult for these training algorithms to utilize the structure characteristics of a fuzzified MLP Thirdly, it is difficult to understand the physical meaning of a fuzzified MLP In contrast to many papers on fuzzified MLPs, much fewer papers were devoted to the fuzzified radial basis function network (FRBFN), which is obtained by direct fuzzification of the well-known neural model RBFN In [24], Cheng and Lee investigated fuzzy regression with fuzzy radial basis function network Their approach, however, is only able to handle crisp-in fuzzy-out data Only the output layer of the classical RBFN is fuzzified so that FLR method can apply to this layer In [25], Chi and Hsu developed a fuzzy radial basis function neural network for predicting multiple quality characteristics of plasma arc welding They proposed a two-stage training algorithm that comprises an unsupervised training stage and a supervised training stage However, their approach is only able to handle fuzzy-in crisp-out data Neither of the paper has carried out the rigorous theoretical analysis of the universal approximation ability of FRBFNs Many other papers are related to the so called fuzzy radial basis function network or fuzzified radial basis function network, such as [26], [27] However, those networks are mainly used to deal with crisp-in crisp-out data They are particularly called neuro-fuzzy networks in the terminology of fuzzy and neural fields In this paper, we investigate the nonparametric fuzzy nonlinear regression with FRBFNs as the universal regression models The proposed FRBFN is a fuzzy extension of the classical RBFN Thus, it falls into the first category of fuzzy neural networks In addition, FRBFN can also be interpreted as a fuzzy expert system Therefore it also belongs to the second category of fuzzy neural networks The contribution of this paper lies in the following aspects The proposed FRBFN can handle fuzzy-in fuzzy-out data as well as crisp data A FRBFN owns the advantage of simple structure and clear physical meaning Some metrics for fuzzy numbers have been extended to the metrics for -dimensional fuzzy vectors, which are applicable to computations in a FRBFN It has been indicated that a FRBFN can also be interpreted as a fuzzy expert system This provides a fuzzy linguistic interpretation of the network The set of FRBFNs has been proven to be a universal approximator for any continuous fuzzy functions defined on a compact set The proposed FRBFN has been applied to two types of nonparametric fuzzy nonlinear regression problems for LR-type fuzzy data It has been shown that the proposed FRBFN is a good universal regression model by examining the three criteria C1) C3) Two types of training algorithms, namely, the two-stage algorithm and back-propagation (BP) algorithm, have been proposed to quickly solve the two types of fuzzy nonlinear regression problems These algorithms are efficient because the structure characteristics of a FRBFN can be incorporated in these algorithms The remaining part of our paper is organized as follows In Section II, we recall some basic concepts and results concerning fuzzy numbers and fuzzy regression In Section III, we introduce the structure of a FRBFN, interpret the network in the fuzzy sense, and prove the universal approximation ability of it Section IV presents two types of fuzzy regression problems with FRBFN and proposes two types of training algorithms to solve them Simulation studies are carried out in Section V Section VI concludes the whole paper and discusses the future work Several mathematical proofs and formulas are contained in Appendix II PRELIMINARIES A Fuzzy Number Fuzzy numbers are used for representing fuzzy data in onedimensional space We will recall some basic concepts of fuzzy numbers in the following In [34] and [52], Dubois and Prade introduced the definition of fuzzy numbers and established some of the basic properties of this family of fuzzy sets Later, the definition was slightly altered by Goetschel et al in [53], and equivalent definitions to the new one were proposed in [35], [49], [51] In this paper, we adopt the definition given by Diamond [47], [51] and rewrite it as follows Definition 1: (Fuzzy Number): A fuzzy set of, say, is called a fuzzy number if its membership function satisfying the following properties: 1) is normal, ie, there exists an such that ; 2) is fuzzy convex, ie, for any and ; 3) is upper semicontinuous; 4) is compact For each denote the -level set of Then from (1) (4) it follows that the -level set is a nonempty compact convex subset of for all Denote the set of all fuzzy numbers Denote the set of fuzzy numbers with their supports bounded by an nonempty compact subset of, that is, Then we have The definition of LR-type fuzzy numbers was first suggested by Dubois and Prade [52] This definition was slightly revised by Zimmermann in [33] In this paper, we shall slightly alter the definitions in [33] and [34] for practical purposes A function defined on, usually denoted by or,isareference function of fuzzy numbers iff 1) is a strictly decreasing upper-continuous function on ; 2) and ; 3) for any A singleton fuzzy membership function on is defined by otherwise (1)

3 744 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 A function defined on, denoted by, is said to be the -companion function of a reference function iff if if Definition 2 (LR-Type Fuzzy Number): A fuzzy number said to be an LR-type fuzzy number iff for for is the left and the right reference function is -companion function of, and is the -companion function of is called the mean (or center) value of and are called left and right spreads, respectively When the spreads are zero, we see from the above is a crisp number As the spreads increase, becomes fuzzier and fuzzier An LR-type fuzzy number is usually characterized by a triple Denote the set of all the LR-type fuzzy numbers Then for any,wehave A triangular fuzzy number is a particular LR-type fuzzy number with its reference functions defined as follows: otherwise Then a triangular fuzzy number can be characterized by a triple A triangular fuzzy number is called a symmetric triangular fuzzy number if its left and right spreads are equal, that is, This kind of fuzzy number can be denoted by a binary In practice, a triangular fuzzy number can also be characterized by an ordered triple with, such that and Obviously,,, and For any If the in of Definition 2 is not a real number but an interval, then the fuzzy number is called an LR-type flat fuzzy number [34], also known as an LR-type fuzzy interval [33], or an LR-type fuzzy number with a flat [46] The following definition was obtained by slightly altering that in [34] Definition 3 (LR-Type Flat Fuzzy Number): A fuzzy number is said to be an LR-type flat fuzzy number iff (2) is (3) (4) (5) (6) of AnLR-type flat fuzzy number can be characterized by a quadruple Denote the set of all the LR-type flat fuzzy numbers Obviously, For any,wehave A trapezoidal fuzzy number is a particular LR-type flat fuzzy number, with and defined by (5) A trapezoidal fuzzy number can be characterized by a quadruple It can also be characterized by an ordered quadruple, with, such that and Then for any,wehave B Two Operations of Fuzzy Numbers Operations on fuzzy numbers are based on the extension principle introduced by Zadeh, which is one of the most basic ideas of fuzzy set theory In this paper we are mainly interested in the addition and scalar multiplication of fuzzy numbers Definition 4: Addition and scalar multiplication of fuzzy numbers in are defined as follows 1) (Addition), for 2) (Scalar multiplication), for and For any, and each, the above two operations have been proved to have the following properties [see Proposition 6111, 47] 1) (Addition) 2) (Scalar multiplication) In particular, these two operations can be simplified for LR-type and flat fuzzy numbers Let and be two LR-type fuzzy numbers with and, we have [34, pp 54 55] (8) (9) (10) (11) Similarly, for two LR-type flat fuzzy numbers and, we have for for for (7) (12) (13) is the left and the right reference function is -companion function of, and is the -companion function Equations (10) (13) can be easily verified by using the previous properties of the two operations and (4) (8)

4 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 745 C Fuzzy Vector In the real world, the fuzzy data being handled are often multidimensional The following definitions of fuzzy vectors are given for representing these fuzzy data Definition 5 (Fuzzy Vector): An -dimensional fuzzy vector is an -tuple of fuzzy numbers with Denote the th-order Cartesian product of, that is, Then, is the set of all -dimensional fuzzy vectors Similarly, we can also define An -dimensional fuzzy vector is called an -dimensional LR-type fuzzy vector if, and is called an -dimensional LR-type flat fuzzy vector if Denote and the set of all -dimensional LR-type fuzzy vectors and the set of all -dimensional LR-type flat fuzzy vectors, respectively D Metric Spaces of Fuzzy Numbers Distance, one of the most basic concepts in mathematics, is used to describe the difference between any two elements of a set Definition 6: is a nonempty set, and a distance measure is defined on The distance measure is said to be a metric if for any it satisfies the following properties: 1) ; 2) if and only if ; 3) ; 4) The set is called a metric space Here, we will introduce three most commonly used metric definitions of fuzzy numbers in the literature Definition 7: The supremum metric on is defined by (14) for all, where is the Hausdorff metric on nonempty compact convex subsets of defined by (15) for any two subsets Lemma 1: is a complete metric space The proof was first given by Puri and Ralescu [49] and also given in the proof of [47, Prop 723] Definition 8: The -metrics on is a class of metrics for defined by (16) for all Lemma 2: is a metric space, and is a complete metric space for any nonempty compact subset of Readers are referred to [47, Props 712 and 724] for the proof of this lemma Fig 1 Structure of a fuzzified radial basis function network For any, let, and Then we have the following definition Definition 9: The -metrics on is a class of metrics for defined by (17) for all Lemma 3: is a metric space, and is a complete metric space for any nonempty compact subset of Readers are referred to [47, Prop 726, pp 52 53] for the proof of this lemma The first recorded use of the metric appears to be Heilpern [48] Puri and Ralescu [35], [49], [50] exploited its metric properties in a metric space context Klement et al [54] used the metric for a strong law of large numbers, and make reference to more general metrics The metrics were proposed by Diamond and Kloeden [51] who showed their equivalence to metrics and investigated their topological characteristics Readers are referred to [47] for more details Besides, the metric space is not a separable metric space, whereas the metric spaces and are separable [47, pp 62 64] The metric proposed by Diamond and Kloeden [51] was originally defined on fuzzy sets of, and has a very complex form which involves the concept of support functions and a Lebesgue integration over a unit sphere in However, when, the expression of the metric can be reduced to a simple one, as given in (17) III FUZZIFIED RADIAL BASIS FUNCTIONAL NETWORK A Network Structure The proposed FRBFN, which has a single hidden layer with hidden neural nodes, is obtained by direct fuzzification of the classical RBFN that was proposed by Moody and Darken [36] The network structure of a FRBFN is illustrated in Fig 1 The input to the FRBFN is a fuzzy vector with s being fuzzy numbers The activation function in each hidden node is of Gaussian form (18)

5 746 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 where, is the center of the th hidden node with s being fuzzy numbers, is the distance between the input vector and the center, which will be discussed in the following part, and is the width or deviation of the th node Note that,, and are all real numbers Let Equation (18) can be equivalently written as (19) where is called the slope of the th node The output of the network is a fuzzy linear combination of the hidden layer outputs (20) where is the connection weight between the output node and the th hidden node, which is also a fuzzy number, and (21) Theorem 1: A FRBFN with its fuzzy weights taking values in and, maps a fuzzy vector in to a fuzzy number in Proof: Since the input variable and hidden-layer weights of the FRBFN are fuzzy vectors in, the outputs of the hidden layer of the FRBFN are positive real numbers, as seen from (18) Since the weights in the output layer are fuzzy numbers in, the output layer of the network performs only two kinds of operations for fuzzy numbers in, ie, the scalar multiplication with positive scalars and addition We see from [47, Prop 6110] that is closed under these two operations Thus, the output of the network is a fuzzy number in Theorem 2: A FRBFN with its fuzzy weights taking values in and, maps a fuzzy vector in to a fuzzy number in Proof: Since the input variable and hidden-layer weights of the FRBFN are fuzzy vectors in, the outputs of the hidden layer of the FRBFN are positive real numbers, as seen from (18) Since the weights in the output layer are fuzzy numbers in, the output layer of the network performs only two kinds of operations for fuzzy numbers in, ie, the scalar multiplication with positive scalars and addition We see from (10) (11) that is closed under these two operations Thus, the output of the network is a fuzzy number in Theorem 3: A FRBFN with its fuzzy weights taking values in and, maps a fuzzy vector in to a fuzzy number in The proof is similar to that of Theorem 2 and omitted here for simplicity Remark 1: Theorem 2 and 3 indicate that if the fuzzy inputs and weights of a FRBFN are all of LR-type, the outputs of the network are still of LR-type It is a nice property for a FRBFN to deal with LR-type fuzzy data In contrast, a fuzzified MLP does not definitely maps an LR-type input to an LR-type output B Metric Spaces of Fuzzy Vectors We see from Part A of this section that a distance between two fuzzy vectors needs to be calculated in a FRBFN Hence we have to define a distance measure for fuzzy vectors Obviously, the metrics defined for fuzzy numbers can be easily extended for fuzzy vectors Suppose a metric is well defined on a subset of and, thus, is a metric space The metric on the subset of is defined as (22) where,,, and Then, we have the following theorem Theorem 4: If is a metric space, is also a metric space In addition, the completeness of implies the completeness of The proof of this theorem is found in Appendix By using (22), the metrics, and for fuzzy numbers can be easily extended to, and for -dimensional fuzzy vectors, respectively Then we have the following corollaries Corollary 1: is a complete metric space This corollary is easily verified by applying Lemma 1 and Theorem 4 Corollary 2: is a metric space, and is a complete metric space This corollary is easily verified by applying Lemma 2 and Theorem 4 Corollary 3: is a metric space, and is a complete metric space This corollary is easily verified by applying Lemma 3 and Theorem 4 C A Fuzzy Interpretation of Fuzzified RBFNs It was shown in [38] that a classical RBFN is functionally equivalent to a kind of simplified T-S fuzzy inference system (or fuzzy controller) In other words, a classical RBFN can be viewed as a fuzzy inference system, which provides one way of interpreting the network In the following we shall show that a fuzzified RBFN can be easily interpreted as a fuzzy expert system Given a FRBFN as formulated in (18) (21), we note that each hidden node of it is corresponding to a fuzzy IF THEN rule as follows: IF is THEN is (23) The computation in the th hidden node can be viewed as a matching operation between the input and th fuzzy rule In this case, the matching degree (or compatibility grade) of with the fuzzy IF THEN rule is (24) where can be viewed as a parameterized similarity measure between two fuzzy vectors, which means that the smaller

6 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 747 their distance is, the more similar they are to each other The computation in the output layer of the FRBFN can be viewed as a fuzzy inference process: the fuzzy outputs of every fuzzy rule and the matching degrees of with every fuzzy rule are aggregated to yield a final output as follows: (25) The aforementioned fuzzy reasoning method was called the modified simplified fuzzy reasoning method, which was proposed by Ishibuchi et al in [17] From the above analysis we can see that a FRBFN is actually a network realization of a kind of fuzzy expert system Remark 2: From the viewpoint of the possibility theory, the function is actually a Gaussian fuzzy membership function to evaluate how much an input variable is similar to the antecedent If the metric in (19) is extended from a metric of fuzzy numbers by using (22), (24) can be further decomposed into (26) where with being the similarity measure on the th-dimensional fuzzy numbers Then we are able to interpret the given FRBFN as a fuzzy expert system with the following traditional fuzzy IF THEN rules IF is and is and and is THEN is (27) In this fuzzy expert system, the modified simplified fuzzy reasoning method in [17] is still adopted, the product operator is used for the fuzzy connective and, and the matching degree of with each rule is calculated by (26) The previous analysis reveals that we can interpret the physical meaning of a FRBFN based on fuzzy linguistics and fuzzy inference mechanism D Universal Approximation It is well known that the set of classical RBFNs is a universal approximator for continuous real functions Is the set of their fuzzified versions still a universal approximator for continuous fuzzy functions? We give the following universal approximation theorems Lemma 4: Suppose is a metric defined on a subset of, and is a compact subset of denotes the set of all continuous fuzzy functions from to, and denote the set of all fuzzy functions from to that are determined by FRBFNs For any given fuzzy function and an arbitrary real number, there exists a fuzzy function, such that for any fuzzy vector, iff is closed under the scalar multiplication with positive scalars and addition, and the metric satisfies the following two conditions 1) for any and 2) for any,, and The proof of this Lemma can be found in Part B of Appendix Theorem 5: Let be a compact set, denote the set of all continuous fuzzy functions from to, and denote the set of all fuzzy functions from to that are determined by FRBFNs For any given fuzzy function and an arbitrary real number, there exists a fuzzy function, such that for any fuzzy vector The proof of this theorem can be found in Part B of Appendix, and can also be found in [43] Theorem 6: Let be a compact set, denote the set of all continuous fuzzy functions from to, and denote the set of all fuzzy functions from to that are determined by FRBFNs For any given fuzzy function and an arbitrary real number, there exists a fuzzy function, such that for any fuzzy vector Theorem 7: Let be a compact set, denote the set of all continuous fuzzy functions from to, and denote the set of all fuzzy functions from to that are determined by FRBFNs For any given fuzzy function and an arbitrary real number, there exists a fuzzy function, such that for any fuzzy vector The proofs of the previous two theorems are also found in Part B of the Appendix Remark 3: The above theorems show that the set of FRBFNs is a powerful function approximator for continuous fuzzy functions, and thus FRBFN is an ideal candidate for the universal regression model in fuzzy nonlinear regression The proofs for these theorems also reveal the interpolation characteristic of the proposed FRBFN, as can be seen in Appendix IV FUZZY REGRESSION WITH FUZZIFIED RBFN A Background In the work of [1], Tanaka et al considered the fuzzy regression problems with the following linear fuzzy function as the regression model (28) where is a fuzzy output,,isan -dimensional nonfuzzy input vector, s are symmetric triangular fuzzy numbers, denoted by The coefficients s in (28) are determined from the given nonfuzzy input output data pairs, The goal of

7 748 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 the FLR is to determine by minimizing the system vagueness subject to the following inclusion condition: for (29) where is the -level set of the fuzzy output Let The FLR problem can be formulated as the following linear programming problem: subject to (30) (31) The criterion function means the system vagueness or fuzziness In this case, is defined as the total spread of the fuzzy outputs The constraint condition (31) means that the -level set of the output should include the target output, and the spreads of the fuzzy coefficients should be nonnegative The first FLR method was improved by using other types of fuzzy coefficients, including quadratic fuzzy coefficients [2], exponential fuzzy coefficients [3], nonsymmetric triangular fuzzy coefficients [9], [16], trapezoidal fuzzy coefficients [16], general LR-type fuzzy coefficients [8], etc Also, FLR methods were proposed for fuzzy-in fuzzy-out data [28] Depending on the criterion function, the existing FLR methods can be roughly classified into the following two categories: F1) FLR methods using minimum fuzziness criteria [1] [3], [8], [9], [16], [19]; F2) FLR methods using fuzzy least-squares criteria [4], [5], [7], [28], [41] Fuzzy nonlinear regression was first considered by Celminšin [6] Ishibuchi et al applied the fuzzified MLP to fuzzy nonlinear regression problems [11], [16] Cheng and Lee suggested the application of RBFNs to fuzzy nonlinear regression for crisp-in fuzzy-out data pairs [24] B Problem Descriptions We first present a general description for a multi-input singleoutput fuzzy regression problem Suppose a set of input output data is given is an -dimensional fuzzy vectors, the corresponding output is a fuzzy number, and a crisp number (vector) can be viewed as a particular fuzzy number (vector) Assume the underlying fuzzy function between the input and output data to be, and is called the fuzzy regression model represents the unknown fuzzy parameter vector of is a real-valued criterion function The goal of fuzzy regression is to find by minimizing, subject to some constraint conditions of the parameter vector, such that (32) where the error is due to the observed error and/or model error, is some operator to make the above equality hold Obviously, a fuzzy regression problem can be formulated as a mathematical programming problem (33) subject to the constraint conditions Very often, the underlying function between the input output fuzzy data is unknown and its functional structure is hardly identified To establish a practical model for applications, we need a universal fuzzy regression model to approximately represent the unknown functional relationship As seen in the last section, the proposed FRBFN has the strong approximation ability and fuzzy linguistic interpretation Thus it can serve as such a universal fuzzy regression model Hereinafter we shall confine fuzzy data to LR-type ones As a result, the proposed FRBFN will be used to approximate the nonlinear fuzzy functions that map an LR-type fuzzy input to an LR-type fuzzy output A simple example of such kind of function is given as However, the class of these functions is very limited For example, define as the fuzzy function extended from the real function by the use of extension principle Obviously, does not necessarily map an LR-type fuzzy input to an LR-type fuzzy output Since such simple fuzzy functions as are not included in the class, what are the advantages of restricting the type of fuzzy data to LR type? The reasons why we choose to develop fuzzy regression approaches for LR-type fuzzy data are explained in the following 1) To make fuzzy numbers computable in digital computers From Section II we see that fuzzy numbers are expressed and determined by their membership functions It is difficult, if not impossible, to directly use these membership functions to compute fuzzy numbers, since each membership value has to be calculated to get the membership function of a fuzzy number, especially when the number of elements in the universe of discourse is infinite There are usually three ways to make fuzzy numbers computable in the literature The first way is to fix the shape of membership functions and only use several shape parameters to represent a fuzzy number Therefore computations of fuzzy numbers are performed with these parameters, as have been done for LR-type fuzzy numbers The second way is to discreterize the universe of discourse to a set with limited number of elements, and thus computations of fuzzy numbers can be easily carried out The third way is to use a limited number of level-sets (closed intervals) to represent a fuzzy number, and perform computations of these intervals to approximate computations of fuzzy numbers The first way is chosen in this paper 2) To meet the requirements of engineering practice In engineering practice, fuzzy data are often expressed by LR-type fuzzy numbers (vectors), such as triangular-type fuzzy numbers To express fuzzy data in this way is often enough for practical problems, and can greatly reduce the computational complexity With LR-type input output fuzzy data given, it is naturally assumed that there should

8 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 749 the following two types of nonlinear mathematical problems to do fuzzy regression with FRBFN 2) Problem 1: Fig 2 Coving area of a LR-type fuzzy number subject to the constraint conditions (37) (39) 2) Problem 2: (40) be an underlying fuzzy function which maps an LR-type fuzzy input to an LR-type fuzzy output Therefore it has practical significance to model this kind of fuzzy function Select a FRBFN with hidden nodes as the fuzzy regression model, and suppose its fuzzy weights are all LR-type The following two types of criterion functions are adopted in this paper 1) F1) Minimum Fuzziness Criterion Function The system fuzziness (or system vagueness) in this paper is defined as the total covering area of LR-type fuzzy outputs, that is (34) where and The covering area of an LR-type fuzzy number is explained in Fig 2 It is the first time that the type of criterion function is introduced, to the best of our knowledge By (10), (11), and (20), (34) can be rewritten as (35) 2) F2) Fuzzy Least-Squares Criterion Function The fuzzy least-squares criterion function in this paper is chosen as (41) subject to the constraint conditions (38) and (39) The constrained optimization techniques, such as the penaltyfunction method, and many kinds of optimization algorithms, such as the gradient descent method and Genetic algorithm, can be used to solve the above nonlinear programming problems However, a nonlinear programming process is computationally expensive In practice we are more concerned with how to determine the weights as rapidly and simply as possible The following parts of this section are devoted to this end C The Parameterized Metrics We see from Section II that the metrics, and are defined levelwisely on In practice, it is very difficult to directly use these definitions to compute the distance between two fuzzy numbers Since the LR-type (flat) fuzzy numbers can be characterized by several parameters, it will be nice to find a parameterized metric for them In this part, we shall show that the metric can be parameterized for LR-type and flat fuzzy numbers In this paper, the metric on is defined by (36) where is a metric defined on The metric can be either selected from the metrics, and,or defined independently With a FRBFN selected as the regression model, the constraint conditions on LR-type fuzzy weights of the FRBFN may be made up of two parts: an inclusion condition and other constraint conditions In this paper the following inclusion condition (suggested in [16]) is used: for (37) where a membership grade is assigned to each input output data pair by human experts The other constraint conditions are (38) (39) With the criterion function and constraint conditions selected, the determination of unknown weights of the FRBFN for the given dataset, can be formulated as a nonlinear mathematical programming problem as shown above In this paper, we present (42) for all, with and By (4), we have (43), as shown at the bottom of the next page, where,, and Similarly, we can also define the metric on with the metric as follows: (44)

9 750 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 for all, From this, we see that the metric has been parameterized to attain the metrics and Obviously, has the same form as when, and Further, we can easily define and by using (22) on LR-type and flat fuzzy vectors, respectively In particular, we have and for triangular and trapezoidal fuzzy numbers D Two-Stage Training Algorithm In this part, we propose a two-stage algorithm to solve the Problem 1 approximately The algorithm comprises two stages In the first stage, which is called the unsupervised training stage, a fuzzy -means (FCM) clustering algorithm for LR-type fuzzy vectors is used in determining the fuzzy weights in the hidden layer In the second stage, which is called the supervised training stage, a linear mathematical programming problem is solved to determine the fuzzy weights in the output layer We explain the two-stage algorithm in details as follows 1) FCM Clustering for LR-Type Fuzzy Vectors: The FCM clustering algorithm is a popular clustering algorithm introduced by Dunn [42] for clustering real number vectors Yang and Ko modified the algorithm for LR-type fuzzy numbers [29] and conical fuzzy vectors [30] Here, we further develop it to handle -dimensional LR-type fuzzy vectors, by the use of the metric introduced in Part C of this section Let be a set of fuzzy vectors in, with and,, We are interested in clustering into classes Similar to the concept of FCM, we propose the following objective function: (45) where is an index of fuzziness, is a matrix with the membership of to the th cluster, s are called fuzzy -prototypes The fuzzy clustering of is the minimization of with respect to and s, subject to for all Let with,, By the use of Lagrange method, the necessary condition for a minimizer of is shown in (46) (48) at the bottom of the page, and (49) We give the FCM clustering algorithm for LR-type fuzzy vectors as follows S1): Fix ; fix ; and fix Initialize the matrix randomly such that, and initialize the -prototypes (ie, set the values of ) by selecting fuzzy vectors from randomly Set the maximum number of iterations (43) (46) (47) (48)

10 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 751 S2): Calculate by using, and (46) (48) S3): If, ; else if, S4): Calculate by using and (49) S5): Compare with in a convenient matrix norm If the norm is less than or the predetermined maximum number of iterations is reached, STOP; else, go to next step S6): Set and, return to step S2) Then, the fuzzy center in the th hidden node of the FRBFN, can be chosen as the prototype of the th cluster after the FCM clustering of input data is over The width (or slope) in the th hidden node, can be determined with the -nearest neighbor heuristics as follows or (50) FLR method which is similar to the one suggested by Savic and Pedrycz [41] to determine the fuzzy weights in the output layer The method comprises two steps a) Determine the by solving a conventional linear regression problem The linear model s inputs are s, target outputs are s, and its coefficients are s Conventional least-squares method is used to solve this problem b) With fixed in the last step, determine the s and s by solving the previous linear programming problem (52) (53) E BP Algorithm In this part, we propose a BP algorithm to solve the Problem 2 In this case, the fuzzy weights of the fuzzified RBFN are determined in the sense of fuzzy least-squares Note that the spreads,,, and must be kept nonnegative during the training progress This is realized by letting (56) and tuning,,, and instead The update formulas in the th epoch are shown as follows: (51) where denotes the set of -nearest neighbors of the th center 2) FLR for LR-Type Fuzzy Data: With hidden weights fixed in the first stage, Problem 1 is reformulated as follows: where (57) subject to (52) where (53) (54) For triangular fuzzy numbers, the constraint condition (53) can be further written as (55) The aforementioned mathematical programming is linear and thus can be easily solved In this paper, we adopt the following (58) with positive real numbers, and being learning rates The partial derivatives in the above formulae are calculated in the back-propagation fashion, as shown in Part C of the Appendix The procedure of the BP algorithm is presented as follows S1): Initialization From the given data set, data pairs are selected randomly Set the fuzzy centers s as the selected s, and the fuzzy weights s as s corresponding to the s The s are determined by the use of -nearest neighbor heuristic as shown in (51)

11 752 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 Fix, and Fix the training accuracy and the maximum number of training epochs S2): For the given data set, calculate the criterion function forward and the partial derivatives of with the adjustable parameters backward S3): If or the maximum number of training epochs is reached, STOP; else update the adjustable parameters by the use of (57) (58) and return to step S2 Remark 4: It is well known that the selection of the initial values can greatly influence the convergence rate of the BP algorithm The initial weights of the FRBFN can be conveniently set from the dataset because the physical meaning of the fuzzy weights in the network is very clear In contrast, the physical meaning of the fuzzified MLP is difficult to understand so that it is not easy to initialize its weights Remark 5: The two-stage algorithm proposed in the previous part can also be used to initialize the fuzzy weights for the BP algorithm Even the clustering algorithm itself can also be employed to initialize the weights by clustering the -dimensional data in the input output space Remark 6: In the above algorithm, all the weights in the FRBFN are tuned However we can only tune the weights in the output layer while leaving the weights in the hidden layer fixed after the FCM clustering procedure In the above, we have developed fuzzy regression approaches with FRBFN for LR-type fuzzy data Since a LR-type fuzzy number can be characterized by only several parameters, readers may argue that traditional neural networks, such as MLP, can also be used to handle fuzzy data of this type So, what are the advantages of using the proposed FRBFN? We think the advantages are lying in the following aspects 1) In theory, the structure of FRBFN can handle fuzzy data of any type, while traditional neural networks can only handle fuzzy data that are coded by discrete parameters 2) FRBFNs have been proved to have the universal approximation property for fuzzy functions, while traditional neural networks have not the property 3) The proposed FRBFN has a fuzzy linguistic interpretation, which is helpful for users to understand the physical meaning of a trained FRBFN, combine field experts knowledge into training process, and extract knowledge from fuzzy training data, while traditional neural networks have not the linguistic meaning when they are used to handle fuzzy data 4) The proposed FRBFN can always ensure the computational validity of fuzzy numbers, while traditional neural networks can not always do To explain this point more clearly, we propose the following small example Suppose we have a fuzzy data pair, where and To model the functional relationship between and, we use a traditional neural network with only one neural node, as shown in Fig 3 Let,,,, and Then we see that the network exactly map to Nowwe provide the network with a new fuzzy input, and the network output is about, which is apparently Fig 3 Structure of a traditional neural network with only one node TABLE I TRAINING DATASET OF EXAMPLE 1 not a valid LR-type fuzzy number This example shows that a traditional neural network can not always ensure the computational validity of fuzzy numbers In contrast, from the structure of FRBFN and Theorem 1 3, we see that a FRBFN can always ensure the computational validity of fuzzy numbers V SIMULATION STUDIES In this section, two simulation examples are presented to demonstrate the proposed approaches Example 1: The fuzzy-in fuzzy-out dataset of Example 1 is given in Table I The inputs s and outputs s are both triangular fuzzy numbers Fig 4 illustrates the distribution of the data in the input output space As shown in Fig 4, each rectangle corresponds to a 0-level set of a fuzzy data pair The dataset is generated as follows Thirty pairs of input output centers are collected from by using the following real-valued function: (59)

12 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 753 TABLE II SIMULATION RESULTS OF EXAMPLE 1 Fig 4 0-level sets of the given input output fuzzy data in Example 1 which is a second-order polynomial function The left and right spreads of s are generated as follows: for for (60) The left spreads and the right spreads of follows: s are generated as (61) The fuzzy regressions with FRBFN have been done on the dataset The used FRBFN contains 15 hidden nodes The proposed two training algorithms have been programmed on Matlab 61 and carried out on a PIII 866 PC The design parameters in our programs for this example are summarized as follows 1) Set the design parameters of the two-stage algorithm For the FCM algorithm we set,, and choose the maximum iteration number as 100 For the FLR algorithm we set 2) Set the design parameters of the BP algorithm We set,,, and choose the maximum epoch number as The fuzzified MLP proposed by Ishibuchi et al [14] has also been used to do fuzzy regression Let denote the output of the fuzzified MLP for a fuzzy input In [14] they defined the following fuzzy least-squares criterion function: where (62) Fig 5 Training results of BP algorithm for the FRBFN in Example 1 denotes the 02-level sets of inputs and target outputs - denotes the 02-level sets of inputs and network outputs Fig 6 Training results of BP algorithm for the fuzzified MLP in Example 1 denotes the 02-level sets of inputs and target outputs - denotes the 02-level sets of inputs and network outputs with and denoting the lower bound and upper bound of the -level cut set of a fuzzy number, respectively In this paper, Based on this criterion function, they developed a BP algorithm with a momentum item for their proposed fuzzified MLP We also realized their algorithm on Matlab 61 The used network in this example is a three-layer fuzzified MLP consisting of 15 hidden nodes The learning rates for the weights in the hidden layer and the output layer are both 05 The constant of the momentum item is selected as 09 The algorithm has been run for epochs for this example The simulation results are summarized in Table II We compute the

13 754 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 TABLE III TRAINING DATASET OF EXAMPLE 2 Fig 7 0-level sets of the given two dimensional fuzzy input data in Example 2 criterion functions for all the algorithms Hereinafter, we shall use to compare the training results of our proposed approaches with those of [14] Fig 8 Surface shaped by the given input output centers in Example 2 Note that the computing time in Table II is related to personal programming style, programming language and computer settings

14 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 755 TABLE IV SIMULATION RESULTS OF EXAMPLE 2 From Table II we see that the proposed FRBFN with its algorithms yields much smaller value of in less training time than the fuzzified MLP This is due to the following two reasons Firstly, the characteristics of FRBFN are well understood and can be incorporated in the training algorithms On the contrary, it is difficult to make use of the characteristics of fuzzified MLP in the training algorithms Secondly, FRBFN has the universal approximation ability, whereas the fuzzified MLP has not the ability We also note that in this example the two-stage algorithm spends much less training time but yields smaller value of than the BP algorithm However, such result may not appear in other cases, because the two-stage algorithm is not aimed at the minimization of For each input data, we compute the fuzzy outputs of the FRBFN trained by BP algorithm and the trained fuzzified MLP, respectively The 02-level sets of inputs and outputs are shown in Figs 5 and 6 Example 2: The fuzzy-in fuzzy-out dataset of Example 2 is given in Table III The input data s are twodimensional triangular fuzzy vectors and the output data s are triangular fuzzy numbers Fig 7 illustrates the distribution of input data As shown in Fig 7, each rectangle corresponds to a 0-level set of an input data Fig 8 illustrates the centers of the fuzzy input output data pairs in three-dimensional space The centers of the fuzzy output data are generated by using the following real-valued function: (63) which is a second-order polynomial function With the above function, 49 pairs of input output centers are generated from The left spreads and the right spreads of the input fuzzy numbers are generated randomly from the interval The left spreads and the right spreads of s are generated as follows: (64) The proposed two training algorithms have been carried out on the dataset The used FRBFN in this example contains 15 hidden nodes The design parameters in our programs for this example are summarized as follows 1) Set the design parameters of the two-stage algorithm For the FCM algorithm we set,, and choose the maximum iteration number as 100 For the FLR algorithm we set 2) Set the design parameters of the BP algorithm We set,,, We also use a three-layer fuzzified MLP with 15 hidden nodes to do fuzzy regression The BP algorithm proposed by Ishibuchi et al has been run on the dataset for epochs The simulation results are summarized in Table IV Note that the computing time in Table IV is related to personal programming style, programming language and computer settings We can see from Table IV that the proposed FRBFN and its algorithms yield much smaller value of in less training time than fuzzified MLP We can also see that the initialization of parameters with the two-stage algorithm can greatly accelerate the converging rate of the BP algorithm, and thus the total training time can be greatly reduced This fact also demonstrates the advantage of the proposed FRBFN VI CONCLUSION In this paper, a fuzzy neural network named FRBFN is applied to the nonparametric fuzzy nonlinear regression The proposed FRBFN is a fuzzy extension of the classical RBFN We summarize the work done in this paper as follows At first we present the structure of a FRBFN Next we show that a FRBFN can be interpreted as a kind of fuzzy expert system Then, we extend some metrics of fuzzy numbers to the metrics for fuzzy vectors and proved that the yielded metric spaces are complete After that we prove FRBFNs can approximate any continuous fuzzy function defined on a compact subset to any degree of accuracy Then two types of fuzzy regression problems with FRBFN are presented for LR-type fuzzy data, and two types of training algorithms are proposed for solving them correspondingly A parameterized metric is employed in these two problems and their training algorithms At last, two simulation examples are presented to demonstrate our approach We show in our analysis and simulations that the FRBFN is more appropriate to serve as a universal regression model than the fuzzified MLP One limitation of this paper is that the proposed fuzzy regression approaches were only developed for LR-type fuzzy data The future work in this direction involves using FRBFNs to handle more general types of fuzzy data, developing more efficient training algorithms, studying the robust fuzzy regression analysis with FRBFN, and applying the proposed approaches to practical problems

15 756 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 APPENDIX A The Proof for Theorem 4 Proof: Let, and be any three fuzzy vectors in with,, and If is a metric space, we can easily verify that the metric satisfies the following properties: 1) ; 2) if and only if ; 3) (Symmetry) ; 4) (Triangular inequality) Now, we proceed to verify that satisfies the property 4) B The Proofs for Lemma 4 and Theorems 5 7 The proof for Lemma 4 is presented as follows Proof: Since is continuous on the compact set, is uniformly continuous on, which means that,, such that whenever Let For each let This produces an open cover of compact Therefore, there is a finite subcover, From condition 2), we can further obtain (B1) for any, in, with an arbitrary positive integer Now, we are ready to construct a FRBFN The FRBFN contains hidden nodes, with as the center of the th node, as the fuzzy weight associated with the th node in the output layer, and The determination of the value of will be explained later The function determined by the constructed network, denoted by, is shown in the following: (B2) By using the Cauchy Schwarz inequality with Thus, property 4) holds for By1) 4), is a metric space Suppose the metric space is complete Next we shall show that the metric space is also complete Let be an arbitrary Cauchy sequence in, that is,,as Thus, as So is a Cauchy sequence in Since is complete, there exists, such that as Let Obviously Then, we have It follows that Since is closed under the scalar multiplication with positive scalars and addition, the output of the function is a fuzzy number in Thus, To end the proof, we only need to determine the value of such that for any, By Condition (1) and, wehave as that is,, as Thus, is a complete metric space (B3)

16 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 757 By (B1) and condition 1), we further have Let Note that is independent on We have as Therefore, there exists such that Set So far all the parameters in the FRBFN have been determined At last, we have (B4) Denote and Wehave (B5) Since,wehave for Since is continuous on the compact set, is also a compact set with denoting the set of image Let The proof for Theorem 5 is given here Proof: We see from the [47, Prop 6110] that is closed under the scalar multiplication with positive scalars and addition To end the proof, we need only to prove that the metric as defined in (14), owns the following two properties: 1) for any and ; 2) for any,,, and The proof of the property 1) can be found in [45] and are reformulated here for completeness Some preliminary notions and results for the proof are also referred to [45] or [47] Let,,, and From the definitions of and scalar multiplication, we have (B6) Then, we have By the definition of, we further have (B8) (B9) Thus, property 1) holds for Now, we proceed to verify property 2) For all, wehave (B7) (B10)

17 758 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 This leads to By the Minkowski inequality By the integral Minkowski inequality (B11) Thus, property 2) holds for By Lemma 4, Theorem 5 is proved The proof for Theorem 6 is similar to that for Theorem 5 and omitted here for simplicity In the following, we give the proof for Theorem 7 Proof: We see from the [47, Prop 6110] that is closed under the scalar multiplication with positive scalars and addition To end the proof, we need only to prove that the metric as defined in (17), owns the following two properties: 1) for any and ; 2) for any,,, and Let,,, and We see from the definition of that Thus, property (2) holds for proved (B13) By Lemma 4, Theorem 7 is C Formulas for BP Algorithm The prediction error for the th data is defined as (C1) Let,, By (C1), we obtain the following formulas: (C2) (C3) (C4) Since Now we proceed to verify prop- Thus, property 1) holds for erty 2) (B12) we have (C5) (C6) (C7) (C8) Let By (C2) (C5), we further get (C9)

18 ZHANG et al: FUZZY NONLINEAR REGRESSION WITH FUZZIFIED RADIAL BASIS FUNCTION NETWORK 759 Let where By (C9), (19), and (21), we have (C10) S3) Calculate by (C9); S4) Calculate by (C10); S5) Calculate, and by (C12) (C14); S6) Calculate, and by (C6) (C8); S7) Calculate,,, and by (C15) (C18); where,, Let denote the squared distance between input and the center of th hidden node, that is By (C11), we get the following formulas: By (C10) (C14) and (19), we further have (C11) (C12) (C13) (C14) (C15) (C16) (C17) (C18) So, by (C1) (C18), the derivatives in (58) are calculated in the following back-propagation fashion: S1): Provide the network with a data and Calculate by (C1); S2) Calculate,, and by (C2) (C4); ACKNOWLEDGMENT The authors would like to thank the anonymous referees for their valuable comments on this paper REFERENCES [1] H Tanaka, S Vejima, and K Asai, Linear regression analysis with fuzzy model, IEEE Trans Syst, Man Cybern, vol SMC-12, no 6, pp , Nov/Dec 1982 [2] H Tanaka and H Ishibuchi, Identification of possibilistic linear systems by quadratic membership functions of fuzzy parameters, Fuzzy Sets Syst, vol 41, pp , 1991 [3] H Tanaka, H Ishibuchi, and S Yoshikawa, Exponential possibilistic regression analysis, Fuzzy Sets Syst, vol 69, pp , 1995 [4] A Celminš, Least squares model fitting to fuzzy vector data, Fuzzy Sets Syst, vol 22, pp , 1987 [5], Multidimensional least-squares model fitting of fuzzy models, Math Model, vol 9, pp , 1987 [6], A practical approach to nonlinear fuzzy regression, SIAM J Sci State Comput, vol 12, no 3, pp , 1991 [7] P Diamond, Fuzzy least squares, Inform Sci, vol 46, pp , 1988 [8] A Bárdossy, Note on fuzzy regression, Fuzzy Sets Syst, vol 37, pp 65 75, 1990 [9] K K Yen, S Ghoshray, and G Roig, A linear regression model using triangular fuzzy number coefficients, Fuzzy Sets Syst, vol 106, pp , 1999 [10] Y H O Chang and B M Ayyub, Fuzzy regression methods-a comparative assessment, Fuzzy Sets Syst, vol 119, pp , 2001 [11] H Ishibuchi, R Fujioka, and H Tanaka, Fuzzy regression analysis using neural networks, Fuzzy Sets Syst, vol 50, pp , 1992 [12], Neural networks that learn from fuzzy if-then rules, IEEE Trans Fuzzy Syst, vol 1, no 1, pp 85 87, Feb 1993 [13] H Ishibuchi, H Tanaka, and H Okada, Interpolation of fuzzy if-then rules by neural networks, Int J Approx Reason, vol 10, no 1, pp 3 27, 1994 [14] H Ishibuchi, K Kwon, and H Tanaka, A learning algorithm for fuzzy neural networks with triangular fuzzy weights, Fuzzy Sets Syst, vol 71, pp , 1995 [15] H Ishibuchi, K Morioka, and I B Turksen, Learning by fuzzified neural networks, Int J Approx Reason, vol 13, no 4, pp , 1995 [16] H Ishibuchi and M Nii, Fuzzy regression using asymmetric fuzzy coefficients and fuzzified neural networks, Fuzzy Sets Syst, vol 119, pp , 2001 [17] H Ishibuchi, M Nii, and C H Oh, Approximate realization of fuzzy mappings by regression models, neural networks and rule-based systems, in Proc IEEE Int Fuzzy Syst Conf, vol II, Seoul, Korea, Aug 22 25, 1999, pp [18] H Ishibuchi, T Nakashima, and T Murata, Performance evaluation of fuzzy classifier systems for multi-dimensional pattern classification problems, IEEE Trans Syst Man Cybern, B, Cybern, vol 29, no 5, pp , Oct 1999 [19] J P Dunyak and D Wunsch, Fuzzy regression by fuzzy number neural networks, Fuzzy Sets Syst, vol 112, pp , 2000 [20] J J Buckley and Y Hayashi, Fuzzy neural networks: A survey, Fuzzy Sets Syst, vol 66, pp 1 13, 1994 [21], Can fuzzy neural nets approximate continuous fuzzy functions?, Fuzzy Sets Syst, vol 61, pp 43 52, 1993

19 760 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL 13, NO 6, DECEMBER 2005 [22], Fuzzy input output controllers are universal approximators, Fuzzy Sets Syst, vol 58, pp , 1993 [23] S Abe and R Thawonmas, A fuzzy classifier with ellipsoidal regions, IEEE Trans Fuzzy Syst, vol 5, no 3, pp , Aug 1997 [24] C B Cheng and E S Lee, Fuzzy regression with radial basis function network, Fuzzy Sets Syst, vol 119, pp , 2001 [25] S C Chi and L C Hsu, A fuzzy radial basis functional neural network for predicting multiple quality characteristics of plasma arc welding, in Proc Joint 9th IFSA World Congr 20th NAFIPS Int Conf, vol 5, Vancouver, BC, Canada, Jul 25 28, 2001, pp [26] L-X Wang, Fuzzy basis functions, universal approximation, and orthogonal least-squares learning, IEEE Trans Neural Networks, vol 3, no 5, pp , Sep 1992 [27] J Nie and D A Linkens, Learning control using fuzzified self-organizing radial basis function network, IEEE Trans Fuzzy Syst, vol 1, no 4, pp , Nov 1993 [28] M S Yang and T S Lin, Fuzzy least-squares linear regression analysis for fuzzy input output data, Fuzzy Sets Syst, vol 126, pp , 2002 [29] M S Yang and C H Ko, On a class of fuzzy c-numbers clustering procedures for fuzzy data, Fuzzy Sets Syst, vol 84, pp 49 60, 1996 [30], Fuzzy clustering procedures for conical fuzzy vector data, Fuzzy Sets Syst, vol 106, pp , 1999 [31], On cluster-wise fuzzy regression analysis, IEEE Trans Syst, Man Cybern, B, Cybern, vol 27, no 1, pp 1 13, Feb 1997 [32] H Bandemer and W Nather, Fuzzy Data Analysis Dordrecht, The Netherlands: Kluwer, 1992 [33] H J Zimmermann, Fuzzy Set Theory and its Applications Dordrecht, The Netherlands: Kluwer, 1991 [34] D Dubois and H Prade, Fuzzy Sets and Systems: Theory and Applications New York: Academic, 1980 [35] M L Puri and D A Ralescu, Fuzzy random variables, J Math Anal Appl, vol 114, pp , 1986 [36] J Moody and C J Darken, Fast learning in networks of locally-tuned processing units, Neural Comput, vol 1, pp , 1989 [37] J-S Jang, ANFIS: Adaptive-network-based fuzzy inference system, IEEE Trans Syst, Man Cybern, vol 23, no 3, pp , May/Jun 1993 [38] J-S Jang and C-T Sun, Functional equivalence between radial basis function networks and fuzzy inference systems, IEEE Trans Neural Networks, vol 4, no 1, pp , Jan 1993 [39] R N Dave, Characterization and detection of noise in clustering, Pattern Recogn Lett, vol 12, pp , 1991 [40] H Demuth and B Mark, Neural Network Toolbox User s Guide (Version 30) Natick, MA: The MathWorks, Inc, 1998 [41] D Savic and W Pedrycz, Evaluation of fuzzy regression models, Fuzzy Sets Syst, vol 64, pp , 1994 [42] J C Dunn, A fuzzy relative of the ISODATA process and its use in detecting compact, well-separated clusters, J Cybern, vol 3, pp 32 57, 1974 [43] D Zhang and K Y Cai, Fuzzified RBFN s can arbitrarily approximate any continuous fuzzy functions (in Chinese), Acta Automatica Sinica, submitted for publication [44] H Ishibuchi, Fuzzy regression analysis, Jpn J Fuzzy Theory Syst, vol 4, pp , 1992 [45] O Kaleva, Fuzzy differential equations, Fuzzy Sets Syst, vol 24, pp , 1987 [46] A Kaufmann, Introduction to Fuzzy Arithmetic: Theory and Applications New York: Van Nostrand Reinhold, 1985 [47] P Diamond and P Kloeden, Metric Spaces of Fuzzy Sets: Theory and Applications Singapore: World Scientific, 1994 [48] S Heilpern, Fuzzy mappings and fixed point theorem, J Math Anal Appl, vol 85, pp , 1981 [49] M L Puri and D A Ralescu, Differentials of fuzzy functions, J Math Anal Appl, vol 91, pp , 1983 [50], The concept of normality for fuzzy random variables, Ann Probab, vol 13, pp , 1985 [51] P Diamond and P Kloeden, Metric spaces of fuzzy sets, Fuzzy Sets Syst, vol 35, pp , 1990 [52] D Dubois and H Prade, Operations on fuzzy numbers, Int J Syst Sci, vol 9, pp , 1978 [53] R Goetschel Jr and W Voxman, Topological properties of fuzzy numbers, Fuzzy Sets Syst, vol 10, pp 87 99, 1983 [54] E P Klement, M L Puri, and D A Ralescu, Limit theorems for fuzzy random variables, Proc Royal Soc, London A, vol 407, pp , 1986 Dong Zhang received the BS degree from Tsinghua University, Beijing, China, in 1998, and the PhD degree from Beihang University (Beijing University of Aeronautics and Astronautics), Beijing, China, in 2004 He was a Research Assistant at City University of Hong Kong in 2002 Since October 2004, he has been with Han Wang Ltd, Beijing, China As a candidate for the PhD degree, he published about 10 papers in international journals, Chinese journals, and international conferences His main research interests include fuzzy systems, neural networks, and computer vision Luo-Feng Deng received the BS and MS degrees from the School of Automation Science and Electrical Engineering, Beihang University (Beijing University of Aeronautics and Astronautics), Beijing, China, in 2000 and 2003, respectively Since April 2003, she has been with the Beijing Institute of Tracking and Telecommunications Technology (BITTT), Beijing, China Her current research interests include neural networks and complex system simulation Kai-Yuan Cai was born in April 1965 He entered Beihang University as an undergraduate student in 1980, received the BS, MS, and PhD degrees all from Beihang University (Beijing University of Aeronautics and Astronautics), Beijing, China, in 1984, 1987, and 1991, respectively He was a Cheung Kong Scholar (Chair Professor), jointly appointed by the Ministry of Education of China and the Li Ka Shing Foundation of Hong Kong in 1999 He has been a Full Professor at Beihang University since 1995 He was a Research Fellow at the Centre for Software Reliability, City University, London, UK, and a Visiting Scholar at City University of Hong Kong, Swinburge University of Technology (Australia), University of Technology, Sydney, Australia, and Purdue University, West Lafayette, IN He has published over 70 research papers and is the author of three books: Software Defect and Operational Profile Modeling (Boston, MA: Kluwer, 1998); Introduction to Fuzzy Reliability (Boston, MA: Kluwer, 1996); Elements of Software Reliability Engineering (Beijing, China: Tshinghua Univ Press, 1995, in Chinese) His main research interests include software reliability and testing, intelligent systems and control, and software cybernetics Dr Cai serves on the Editorial Board of the international journal Fuzzy Sets and Systems and is the Editor of the Kluwer International Series on Asian Studies in Computer and Information Science ( He served as program committee co-chair for the Fifth International Conference on Quality Software (Melbourne, Australia, September 2005), the First International Workshop on Software Cybernetics (Hong Kong, September 2004), and the Second International Workshop on Software Cybernetics (Edinburgh, UK, July 2005) He also served (will serve) as guest editor for Fuzzy Sets and Systems (1996), the International Journal of Software Engineering and Knowledge Engineering (2006), and the Journal of Systems and Software (2006) Albert So received the BSc, MPhil, and PhD degrees from the Department of Electrical and Electronic Engineering, University of Hong Kong, in 1983, 1988, and 1995, respectively Before he joined City University of Hong Kong, he had been with the Electrical and Mechanical Services Department of the Hong Kong Government He is now with the Department of Building and Construction, City University of Hong Kong, as an Adjunct Professor and Managing Director of CityUOS Ltd, an associated company of the CityU Enterprise Ltd Dr So is a Chartered Engineer of UK, a Registered Professional Engineer of HK, FCIBSE, MIEE, SMIEEE, MHKIE, and MASHRAE