Pattern Recognition Using Discriminative Feature Extraction

Transcription

1 500 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 2, FEBRUARY 1997 IV. EXPERIMENTAL RESULTS The threshold selection method of Section III-B was implemented and tested on a variety of 2-D signals corrupted by additive white Gaussian noise. Figs. 1 3 present the results of our algorithm and compare them with the SureShrink and VisuShrink algorithms of Donoho and Johnstone [2], [14]. Daubechies six-coefficient orthonormal wavelet [15] was used as the wavelet basis for the examples in Figs. 1 and 3; the Haar wavelet was used in Fig. 2. In all cases, only the three finest scales of the wavelet decomposition were denoised, as the wavelet coefficients at the coarser scales are comprised mostly of signal, and denoising these scales may actually be detrimental [14]. The standard deviation of the noise was estimated by ^, as described in Section III-A. SureShrink [2] is a data analytic technique that chooses a threshold for each detail signal by minimizing the Stein unbiased estimate of risk [16] for threshold estimates. SureShrink automatically adapts to the underlying smoothness of the signal being estimated in a way that is near-optimal in the minimax mean-squared-error (MSE) sense. In an attempt to minimize MSE, SureShrink allows some noise to remain in the reconstruction, as is apparent from Fig. 2. The VisuShrink algorithm [14], on the other hand, eschews minimizing MSE in an attempt to achieve improved visual quality by removing essentially all of the empirical wavelet coefficients that could (statistically) be attributed solely to noise. This is done by uniformly applying a threshold of = 2log(n), where n is the sample size. By varying the level of significance, the data analytic method presented in Section III-B can behave either like SureShrink or like VisuShrink or anywhere in between. This is illustrated in Figs. 1 3: in Fig. 1, is large (0.999), and our method behaves like SureShrink; in Fig. 2, is small (0.05), and our method behaves like VisuShrink; Fig. 3 shows the effect increasing has on our method. When the two Donoho Johnstone threshold selectors are considered in the context of statistical test of hypotheses, it is apparent that the VisuShrink method makes it relatively difficult for a coefficient to be declared significant, and the SureShrink method makes is relatively easy. The ability to use an easily interpretable user-supplied value like to control the smoothness of the reconstructed signal makes this procedure quite versatile for a wide range of applications, such as functional MRI signal analysis [17], image enhancement (Fig. 1), and function estimation (Figs. 2 and 3). REFERENCES [1] D. L. Donoho, Nonlinear wavelet methods for recovery of signals, densities, and spectra from indirect and noisy data, in Proc. Symposia Appl. Math., 1993, pp , vol. 47. [2] D. L. Donoho and I. M. Johnstone, Adapting to unknown smoothness via wavelet shrinkage, J. Amer. Stat. Assoc., vol. 90, pp , [3] R. T. Ogden and E. Parzen, Data dependent wavelet thresholding in nonparametric regression with change-point applications, Comput. Stat. Data Anal., vol. 22, pp , [4] G. P. Nason, Wavelet regression by cross-validation, Dept. Stat., Stanford Univ., Stanford, CA, Tech. Rep. 447, [5] N. Weyrich and G. T. Warhola, De-noising using wavelets and crossvalidation, Dept. Math. Stat., Air Force Inst. Technol., AFIT/ENC, 2950 P. St., Wright-Patterson Air Force Base, OH, Tech. Rep. AFIT/EN/TR/94-01, [6] B. Vidakovic, Nonlinear wavelet shrinkage with Bayes rules and Bayes factors, Duke Univ., Durham, NC, Tech. Rep., [7] S. G. Mallat, A theory for multiresolution signal decomposition: The wavelet representation, IEEE Trans. Pattern Anal. Machine Intell., vol. 11, pp , [8] D. L. Donoho, De-noising via soft-thresholding, IEEE Trans. Inform. Theory, vol. 41, no. 3, pp , May [9] R. A. DeVore and B. J. Lucier, Fast wavelet techniques for near-optimal processing, in IEEE Military Comm. Conf., 1992, pp [10] J. B. Weaver, X. Yansun, D. M. Healy, and L. D. Cromwell, Filtering noise from images with wavelet transforms, Magnetic Resonance Med., vol. 21, no. 2, pp , [11] R. T. Ogden and E. Parzen, Change-point approach to data analytic wavelet thresholding, J. Stat. Comput., vol. 6, pp , [12] S. M. Ross, Stochastic Processes. New York: Wiley, [13] G. R. Shorack and J. A. Wellner, Empirical Processes with Applications to Statistics. New York: Wiley, [14] D. L. Donoho and I. M. Johnstone, Ideal spatial adaptation via wavelet shrinkage, Biometrika, vol. 81, pp , [15] I. Daubechies, Orthonormal bases of compactly supported wavelets, Commun. Pure Appl. Math., vol. 41, pp , [16] C. Stein, Estimation of the mean of a multivariate normal distribution, Ann. Stat., vol. 10, pp , [17] M. L. Hilton, T. Ogden, D. Hattery, G. Eden, and B. Jawerth, Wavelet denoising of functional MRI data, in Wavelets in Medicine and Biology, A. Aldroubi and M. Unser, Eds. Boca Raton, FL: CRC, 1996, pp , ch. 4. Pattern Recognition Using Discriminative Feature Extraction Alain Biem, Shigeru Katagiri, and Biing-Hwang Juang Abstract We propose a new design method, called discriminative feature extraction (DFE) for practical modular pattern recognizers. A key concept of DFE is the design of an overall recognizer in a manner consistent with recognition error minimization. The utility of the method is demonstrated in a Japanese vowel recognition task. I. INTRODUCTION Most practical recognizers (recognition systems) are modular systems consisting of a feature extractor (feature extraction module) and a classifier (classification module). In this correspondence, we propose a novel, general design method called discriminative feature extraction (DFE) for achieving high-recognition accuracy by using such modular systems. The formulation of DFE mainly relies on the minimum classification error/generalized probabilistic descent (MCE/GPD) method [2], [3]. We shall describe the details of the method and show its utility through experiments in designing speech pattern recognizers having a liftering feature extractor. II. DISCRIMINATIVE FEATURE EXTRACTION We consider a modular pattern recognizer, which is shown in Fig. 1, for an M-class recognition task (C j ; j = 1; 111;M): Let Manuscript received September 15, 1995; revised July 20, The associate editor coordinating the review of this paper and approving it for publication was Dr. Michael Zervakis. A. Biem is with ATR Human Information Processing Research Laboratories, Kyoto, Japan. S. Katagiri is with ATR Interpreting Telecommunications Research Laboratories, Kyoto, Japan. B.-H. Juang is with Bell Laboratories-Lucent Technologies, Murray Hill, NJ USA. Publisher Item Identifier S X(97) X/97$ IEEE

2 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 2, FEBRUARY x be a measured pattern such as a segment of a waveform or a sequence of spectral parameters. The feature extractor with parameter 2 is a function that maps x to its corresponding feature pattern y, i.e., f 2 (x) =y: The feature pattern y is the input accepted by the classifier C(1) that operates under the following decision rule: C(y) =C i ifi = arg max g j(y;3) (1) j where g j(y;3)is a discriminant function, which indicates the degree to which y belongs to C j and is defined by the parameter (set) 3: Traditionally, the feature extractor f 2 is determined by empirical means independent of the design of the discriminant function set fg j (y;3)g: Although this conventional approach has led to successful results to some extent, it is obvious that there is still plenty of room for improvement: Due to the lack of interaction in design between the feature extractor and the classifier, the empirical method of design does not guarantee that the resulting feature representation is the best for the classification process. A key point of our DFE formulation is the overcoming of the above problem by incorporating the right interaction in the design. The first step of the formulation is to integrate the feature extraction function with the classifier function. Obviously, by substituting y with f 2 (x) in (1), we would have an equivalent integrated recognizer design framework defined by a set of discriminant functions fg j(f 2(1); 3)g: The next step is to provide an optimization mechanism for adjusting all of the parameters 8=(2;3) of this discriminant function set in a direct manner to the final recognition accuracy. To achieve this two-step formulation, we use MCE/GPD in a somewhat expanded fashion. The above substitution results in g j (f 2 (x); 3) ) g j (x;8) with equivalent parameter set 8: This reminds us that working directly with a recognizer defined by a super parameter set 8 could have been more effective than assuming separability 8=(2; 3): Nevertheless, assuming a modular design, which implies a separation of the parametric function g j (f 2(1); 3), leads to the opportunity to focus on some parametric family of feature extraction functions and to study further characteristics of the features from different perspectives, such as perception research. Taking this point into account, we shall maintain the assumption of a modular design, i.e., 8 = (2;3) throughout our formulation. The formulation of DFE, mainly relying on MCE/GPD, is defined as follows. Assume that a design sample x t (2 C k ) is given at the training time index t: The feature extractor maps x t to y t : We emulate a recognition decision for this sample with a misclassification measure defined over the set of the discriminant functions fg j (y;3)g: In the case of using the decision rule of (1), a typical example of the misclassification measure is d k (f 2(x t); 3) = d k (y t;3) =0g k (y t;3)+ 1 M 01 j6=k fg j(y t;3)g where is a positive number that controls the relative contribution of the classes considered. We shall use this measure throughout this correspondence. We evaluate a resulting classification decision by using a loss (cost). In accordance with the MCE concept, we use the smooth binary (0-1 step) loss defined as 1 `k(x t ;8)= (3) 1 + expf0ad k (f 2(x t); 3)g where a is a positive constant. Then, we reach the DFE adjustment rule 1= (2) 8 t+1 =8 t 0 t Ur 8`k(x t ;8 t ) (4) Fig. 1. Structure of a modular pattern recognizer. which is based on the probabilistic descent theorem [1], [3], where U positive-definite matrix t small, monotonically decreasing positive number at t (called a learning factor) 8 t status of 8 at t: Note here that the chain rule of differential calculus is used for adjusting the feature extractor module. The theorem shows that an infinite repetition of the adjustment achieves local minimization of the expected loss L(8) = E x [`k(x;8)] = E x [`k(f 2 (x); 3)] resulting in a local optimal status of 8 (see [1]). However, in practice, since infinite training is obviously impossible, we repeat, in a practical implementation of DFE, a finite run of the adjustment (4) over a given finite set of training samples. Consequently, this practical DFE training reduces the empirical loss 1=T 6 t `(x t ;8); or the recognition error count that is approximated by this empirical loss, where T is the number of the given training samples. III. APPLICATION TO LIFTERING-BASED SPEECH RECOGNITION A. Task and Experimental Conditions A cepstrum vector, which is computed by applying the inverse DFT to a logarithmic power spectrum of a speech fragment, has been shown to have information useful for classifying phonemes, particularly in its low-quefrency components. Methods of designing a lifter, which is a quefrency-domain filter used to extract the useful low-quefrency components, have thus been extensively investigated (e.g., [4], [5]). However, most of them are inadequate due to a lack of consideration with regard to directly linking the lifter design with the design of post-end classifiers. To investigate the fundamental nature of DFE, we conducted experiments in designing a recognizer having a liftering-based feature extraction module in a five-class, Japanese vowel recognition task. Each input (to the feature extractor) pattern is represented as a fixed-dimensional cepstrum vector, which corresponds to a center fragment of a vowel sound. Liftering is usually done by multiplying an input cepstral value by a weight value. Taking this into account, we use the recognizer illustrated in Fig. 2. The recognizer consists of a liftering feature extractor and a three-layer (one hidden layer) perceptron neural network classifier. The lifter is implemented with a set of weights, each associated with a straight connection at one of the 128 (0th to 127th) quefrency positions (see later paragraphs). The classifier is a usual, fully connected multilayer perceptron network. The input layer of this classifier has 128 nodes, corresponding to the lifter structure. The top output layer has five nodes in accordance with the number of classes. The output functions one at each node are nonlinear (sigmoidal) only at the classifier hidden layer while being linear at the other layers. A cepstrum vector, i.e., a system input (corresponding to x), is first liftered by an innerproduct computation with the lifter weight vector at the feature extraction module, and then, the resulting weighted pattern (corresponding to y) is classified based on the standard network computation at the classification module. The node of the top classifier layer outputs the discriminant function value.

3 502 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 2, FEBRUARY 1997 Fig. 2. Recognizer structure consisting of a three-layer perceptron classifier with a liftering module. Fig. 4. Error rates versus the number of hidden nodes. Upper: baseline system without liftering (with allpass lifter). Bottom: uniformly unitialized. vowel) were used for design, and the other half (70 samples per speaker and per vowel) were used for testing. Fig. 3. Baseline rectangular lifters. The recognizer would appear to be a standard four-layer network. However, as its lowest straight connection is purposely selected for implementing liftering, the recognizer is obviously a modular system having differences in nature between the front-end and post-end modules. This modularity assumption, which is a fundamental concept of the DFE definition, will become clearer from later discussions about the system initialization for training. The input patterns were generated by segmenting vowel center fragments with a 21-ms Hamming time window, from 500 phonetically balanced sentences uttered by five speakers (three males and two females: 100 sentences/speaker) under noise-free conditions and using 256-point FFT. Speech data was digitized at a 12 khz sampling frequency and stored at 16 bits. The total number of generated input samples was 3500; half of them (70 samples per speaker and per B. Results and Discussions Considering the nature of the GPD s adaption, we ran several pairs of training and testing for each system selection, such as the selection of the classifier hidden nodes and the selection of training initialization, changing the training conditions, such as the setting of the learning factor and the order of design sample presentation. Since the variance among these conditions was minor, we shall use the best accuracy (lowest recognition error rate) for each system selection in the following. It is advisable that, similar to general cases of gradient-based optimization algorithms, the DFE-trained, or GPD-trained, recognizer is initialized in some reasonable way. However, it is rather unclear how one can reasonably initialize the back-end network classifier. Therefore, the neural network classifier was initialized randomly, as is usually done in neural network applications. As for the feature extractor, the following three initialization methods were investigated: A random initialization that set the lifter weights to small random values.

4 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 2, FEBRUARY TABLE I ERROR RATES FOR THE BASELINE SYSTEMS AND THE DFE-TRAINED SYSTEMS Liftering process Training Testing 3.5% 16.0% (no liftering (128-length fixed lifter)) (32-length fixed lifter) 13.4% 15.3% (16-length fixed lifter) 8.4% 14.5% (8-length fixed lifter) 9.7% 14.9% (16-length rectangle initialization) 8.4% 14.2% (random initialization) 7.0% 16.8% (uniform initialization) 3.2% 11.3% A uniform initialization that set the lifter weights to the identical value one (1). A rectangular initialization that set the low (quefrency)-pass rectangular lifter with some preset duration. For comparison purposes, we investigated a in which the feature extractor module was fixed, and only the classifier module was trained with MCE/GPD. In this, the lifter was set to the rectangular shape, which has been described in the third initialization case above; due to the lack of exact information on the advisable length of the lowpass lifter, which would lead to accurate vowel classification, we investigated four lifter duration settings of 8, 16, 32, and 128, as seen in Fig. 3. Note that the rectangular lifter with the duration of 128 works as an allpass lifter, corresponding to the situation of no liftering. Generally, the number of trainable parameters affects the achievable accuracy of the classifier. To investigate this point in our framework, we ran preliminary experiments, setting the number of classifier hidden nodes to 30, 40, 50, 60, and 70 for the MCE/GPDtrained having the 128-length allpass lifter and the uniformly initialized,. Fig. 4 shows the achieved error rates one for each number of hidden nodes: The upper figure is for the, and the bottom is for the DFE-trained system. The results show that using 60 hidden nodes is the most suitable for our experimental framework. In light of the above results, we used a classifier having 60 hidden nodes in the subsequent experiments. First, we compared the four lowpass lifter lengths for the. As summarized in Table I, the length of 16 produced the best performance over the testing data. The no-liftering situation achieved very high accuracy on the training set but resulted in lower accuracy on the testing set. The results may suggest that the training in the no-liftering case used the information that was specific to the training samples but irrelevant to finding the true class boundary, and an appropriate liftering is indispensable for alleviating this problem. We then compared the three initialization methods for the DFE training. Based on the above results, the length of the initial lifter was set to 16 in the rectangular initialization. The results for these three initialization cases are also shown in Table I. DFE successfully achieved 11.3% in the uniform initialization case. Here, and this is important, the DFE training provided this improvement over the testing data, whereas on the training data, it showed a similar accuracy to that of the no-liftering. This may allow us to argue that the DFE-trained lifter successfully extracted features that are general and more useful for the classification of vowels. The random initialization performed poorly. This was probably caused by the local optimality problem. To further understand the nature of DFE-trained feature extraction, Fig. 5. Typical DFE-designed lifter issued from the uniform initialization. Upper: the entire view. Bottom: the same lifter with a focus on the lower quefrency region. Fig. 6. Typical DFE-designed lifter issued from the 16-length rectangular initialization. Upper: the entire view. Bottom: the same lifter with a focus on the lower quefrency region. we observed the shape of the trained lifters. Fig. 5 shows a typical shape of the trained lifter in the uniform initialization case; Fig. 6 shows this for the rectangular initialization case. Each of these lifters de-emphasizes two quefrency regions, i.e., 1) the high quefrency region that corresponds to pitch harmonics and spectral minute structure and 2) the lower quefrency region (0-2 quefrency region) that is dominated by the bias and slant of the overall spectrum while enhancing the region of 3-20 quefrency that mainly corresponds to the spectral formant structure. This quite reasonable result might be good support for our argument cited above. According to a separate analysis of pitch frequencies of the design speech samples, it was found that female pitches ranged from 40 quefrency to 60 quefrency and male pitched was concentrated in the 80-quefrency region. The trained lifter in Fig. 5 successfully sup-

5 504 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 2, FEBRUARY 1997 Fig. 7. Normalized inverse variance over the training set. Normalization is performed such that the maximum value is equal to 1. presses these pitch components, although a few ripples remain in the corresponding regions. The pitch components are better suppressed in Fig. 6. It is obvious that these trained lifters are different from conventional ones based on a priori knowledge (e.g., see [4]). We compared the resulting lifters with those obtained from the data-driven approach proposed in [5]. Fig. 7 shows the inverse function curve of the average intraclass variance, computed according to the method in [5]. One should note that we used the DFT-based cepstrum, whereas [5] used an LPC-based one; therefore, it is difficult to compare both directly. However, it is clear that the curve successfully suppresses the lower-quefrency components but fails to suppresses the higherquefrency components. Again, the comparison clearly shows that DFE provides an important departure from the conventional design approach. IV. CONCLUSION In this correspondence, we have proposed a novel recognizer design method called discriminative feature extraction (DFE) and demonstrated its utility through the design of exemplar vowel fragment recognizers using liftering-based feature extraction. The proposed method enables one to link the feature extractor with the classifier to directly minimize misclassifications. For evaluation, we conducted a vowel fragment recognition experiment using a lifteringbased feature extractor. It was shown that a DFE-designed lifter successfully achieved fewer classification errors than the baseline MCE/GPD-trained network classifier represented by a rectangular lifter shape when adjusting a uniformly initialized lifter (i.e, no use of a priori knowledge in initializing the feature extractor parameter). The difference between DFE and classical use of MCE/GPD has a large significance, although a rather simple one. That is, the design scope extended by DFE will achieve more accurate recognition, given system resources (e.g., trainable parameters), by extracting more suitable features for classification; this property can conversely lead to system size reduction and faster recognition computation, relieving the burden on the classifier. Furthermore, by controlling the training conditions, such as the learning factor, the wider scope will realize a more flexible inter-module interaction. DFE substantially increases the number of adjustable system parameters. This fact would enlarge the ratio of the number of parameters to that of given training samples, hence, probably increasing the statistical bias of training results. However, in contrast with the usual case of increasing the parameters of a classifier that operates in a fixed feature space, the nature of such bias in the DFE training framework is unclear. This point is an important future research issue. REFERENCES [1] S. Amari, A theory of adaptive pattern classifiers, IEEE Trans. Energy Conv., vol. EC-16, no. 3, pp , [2] S. Katagiri, C.-H. Lee, and B.-H. Juang, A generalized probabilistic descent method, in Proc. Acoust. Soc. Jpn. Fall Conf., 1990, pp [3] B.-H. Juang and S. Katagiri, Discriminative learning for minimum error classification, IEEE Trans. Acoust., Speech, Signal Processing, vol. 40, pp , Dec [4] G. Ohyama, S. Katagiri, and K. Kido, A new method of cepstrum analysis using comb type quefrency window, J. Acoust. Soc. Jpn., vol. 2, no. 3, pp , [5] Y. Tohkura, A weighted cepstral distance measure for speech recognition, IEEE Trans. Acoust., Speech, Signal Processing, vol. 35, no. 10, pp , Oct