A Trimmed Translation-Invariant Denoising Estimator

Transcription

1 A Trimmed Translation-Invariant Denoising Estimator Eric Chicken and Jordan Cuevas Florida State University, Tallahassee, FL Abstract A popular wavelet method for estimating jumps in functions is through the use of the translation invariant (TI) estimator. The TI estimator addresses a particular problem, the susceptibility of wavelet estimates to the location of features in a function with respect to the support of the wavelet basis functions. The TI estimator reduces this reliance by cycling the data through a set of shifts, thus changing the relation between the wavelet support and the jump location. However, a drawback to TI is that it includes every shifted analysis in the reconstruction, even those that may reduce, rather than improve, the effectiveness of the method. In this paper, we propose a method that modifies the TI to improve jump reconstruction in terms of mean square error of the reconstructions and visual performance. Information from the set of shifted data sets is used to mimic the performance of an oracle which knows exactly which are the best TI shifts to retain in the reconstruction. The TI estimate is a special case of the proposed method. A simulation study comparing this proposed method to existing wavelet estimators and the oracle is provided. 1 Introduction Efficient estimation of jump-like features in a function is an important goal in signal analysis. These features of the signal are often the most difficult to extract from noisy data and may be the most interesting part of the function. For example, in statistical process control a process fault can often result in a spike in the data, or the nominal process signal may contain jumps in a fault-free situation. The tonnage signal of Jin and Shi (2001) is a clear example where it is vital to estimate such jumps accurately. Early methods of functional analysis did not solve this problem completely satisfactorily. Fourier analysis, for example, introduced unwanted artifacts near jumps in the signal. Thus, while a jump could be modeled, the surrounding smooth portions of the signal were adversely affected. Modifications to Fourier analysis to correct for this problem, referred to as Gibbs phenomenon (Walker and Wright (2002)), were developed, but typically at a price. For example, the σ-approximation method reduced the spread and magnitude of Gibbs phenomenon, but it also smoothed over the jumps. Wavelet methods have recently been applied to this problem. The multiresolution nature of wavelets enable an estimator to accurately model functions with jumps, but there is one significant drawback. The placement of such features in relation to the support of the wavelet basis functions affects the quality of the reconstruction. Wavelets, stationary, thresholding, nonparametric regression, translation- Key words and phrases. invariant, oracle. 1

2 This problem was addressed by the stationary wavelet transform method, also known as translation-invariant (TI) denoising, see Coifman and Donoho (1995) and Nason and Silverman (1995). The dependence of jump location on reconstruction accuracy of the signal was lessened by analyzing multiple location-shifted versions of the data and averaging the resulting individual estimates. TI-denoising has better jump reconstruction abilities than the discrete wavelet transform (DWT) of Mallat (1999) applied only to the original data. There is a drawback to the TI method. Since one cannot tell beforehand which translations of the original data will best estimate the unknown function and its jumps, all possible translations are considered. While an efficient algorithm exists that quickly calculates the wavelet coefficients for all the translations, using all these translations can introduce unwanted variability in the reconstruction of the unknown function. As an example, consider Figure 1. The observed sampled data follows y i = f(x i ) + ε i, i = 1, 2,..., n, (1) where ε i are independent and identically distributed errors with mean 0 and constant variance σ 2 and the sample points x i = i/n are equally spaced over an interval. Let f be the blocks function of (ref) and σ 2 chosen such that the signal-to-noise ratio (SNR) is 5. The true function f is shown as a dotted line, the TI estimate is solid. The dashed line represents the standard deviation of the estimates from each of the n possible translation estimates at each point x i. The TI estimate at a point x i is the average of these n point estimates at x i. The standard deviation was shifted down by 4 in the figure to avoid cluttering the plot. Note that the standard deviation of the point estimates is low over the smooth portions of f, but quite high at the points x i that are near the jumps in f. The large standard deviations of the n translation estimates at the jumps are a result of the dependency of the wavelet estimates on the placement of these jumps. Some of the translations result in poor jump estimates while others result in good ones. The TI estimate averages over all these estimates, both good and bad. It is hoped that the good estimates will overcome the deficiencies brought in by the bad ones. However, averaging over a set of point estimates with considerable variation often results in oversmoothing at that point. When the standard deviation at a point x i is low, all the translation point estimates are in close agreement. When it is high, they are not. In the first case, we note that the TI estimate in Figure 1 is accurate, while in the second case this is not so. The higher variability in the translation point estimates leads to reduced estimation accuracy. In particular, we observe oversmoothing of the jumps. This suggests a method to improve the TI estimate. At points of large standard deviation, we propose to reduce the set of translations used to create the TI estimate at that point. In particular, we will iteratively remove those translation estimates for the function at x i that are extreme with respect to the rest. This will have the effect of reducing the standard deviation at the estimated points x i. We then take the average of the translation estimates remaining after this trimming. The correct amount of trimming should result in an improved fit to the original, unknown function f in terms of mean squared error. Additionally, it should provide an improved visual reconstruction of any jumps in the f. Since we do not observe f, the appropriate level of trimming is determined by estimating a maximum allowed point estimate variability 2

3 True f TI estimate sd of TI Figure 1: A function f, its TI estimate and the variability of the n TI shift-unshift estimates at each point x i. Note that the variability (measured as sample standard deviation) is shifted down by 4 to unclutter the plot. α [0, ] that results in the best fit to the to the observed noisy data. If the trimming parameter α is, the proposed estimator is equivalent to the TI estimator. This paper is divided as follows. Section 2 provides a brief background on wavelets and provides details on the estimator. Section 3 contains a simulation comparison of the proposed estimator to the TI-denoising estimator, the VisuShrink estimator and an oracle estimator. This oracle is the proposed estimator which always chooses the correct value for α. 2 The Estimator Before describing the construction of the proposed estimator, a brief background on wavelets and wavelet notation is provided. 3

4 2.1 Wavelets Wavelets are an orthogonal series representation of functions in the space of square-integrable functions L 2 (R). Vidakovic (1999) and Ogden (1997) offer good introductions to wavelet methods and their properties. Let ϕ and ψ represent the father and mother wavelet functions, respectively. There are many choices for these two functions, see Daubechies (1992). Here, ϕ and ψ are chosen to be compactly supported and to generate an orthonormal basis. Let and ϕ jk (x) = 2 j/2 ϕ(2 j x k) ψ jk (x) = 2 j/2 ψ(2 j x k) be the translations and dilations of ϕ and ψ, respectively. For any fixed integer j 0, is an orthonormal basis for L 2 (R). Let and {ϕ j0 k, ψ jk j j 0, k an integer } ξ jk = f, ϕ jk θ jk = f, ψ jk be the usual inner product of a function f L 2 (R) with the wavelet basis functions. Then f can be expressed as an infinite series f(x) = k ξ j0 kϕ j0 k(x) + θ jk ψ jk (x). (2) The function f is not known and must be estimated. This is done using the discrete wavelet transform (DWT). If f is sampled as a vector of dyadic length n = 2 J for some positive integer J, then the DWT will provide a total of n estimated coefficients ξ j0 k and θ jk over the indices j = j 0, j 0 + 1,..., J 1 and for all appropriate k. The lowest level possible for j 0 is 0, the highest is J 1. The wavelet basis functions are easily periodized to a specified interval. In this paper, we use wavelets that have been periodized to the interval [0, 1]. In this case, the index k for resolution level j runs from 1 to 2 j in (2). Wavelets have the useful property that they can simultaneously analyze a function in both time and frequency. This is done by projecting the function to be analyzed into several subspaces or resolution levels. Each resolution level represents a different degree of smoothness of the function. The lowest resolution level, associated with the index j = j 0, represents the smoothest or coarsest part of the function. Increasing the index j corresponds to decreasing smoothness. The highest resolution levels j therefore represent the behavior of the function at the highest frequencies. Since the wavelet series (2) forms an orthogonal representation, the sum of the projections in these resolution levels is the original function f. Because of the compact support of the wavelet functions ϕ and ψ, wavelets also provide the ability to localize the analysis within each subspace. The higher the resolution, the greater the degree of localization. 4 j=j 0 k

5 By varying the resolution level j wavelets have the ability to zoom in or out onto the smooth or detailed structure of f. This is referred to as the multiresolution property of wavelets. Changing the index k allows wavelets to localize the analysis. These properties enable wavelets to model functions of very irregular types, as well as smooth functions. We use W to denote the n n DWT transformation matrix. Applying the DWT to the observed values y = (y 1, y 2,..., y n ) in (1) gives the estimated wavelet coefficients θ = { ξ j0 1, ξ j0 2..., ξ j0 2 j 0, θ j0 1, θ j0 2,... θ J 1,2 J 1} = W y, Applying the inverse DWT W 1 = W to these coefficients returns the original data, y = W W y. Most wavelet analysis uses some form of thresholding. These can be term-by-term methods, where each individual wavelet coefficient is modified individually, or block methods, where coefficients are modified in groups. A popular method of term-by-term thresholding is the VisuShrink method of Donoho and Johnstone (1998). For a single estimated coefficient θ jk, ˆθ jk = η( θ jk, λ) = sgn( θ jk )( θ jk λ) + is the thresholded coefficient, where λ is a threshold parameter. This type of thresholding is referred to as a soft threshold. It sets coefficients smaller in magnitude than λ to 0 and shrinks coefficients larger in magnitude than λ towards 0. The value of λ is chosen to give optimal results in terms of reconstruction, λ = σ 2 log(n) where σ and n is from (1). A thresholded estimate is then formed as ˆf = W η(w y, λ). Note that only select resolution levels are subjected to thresholding. Typically these are the highest m resolution levels, where m is specified by the user. The properties and advantages of differing type of thresholding methods is well documented in the literature. See Donoho and Johnstone (1994, 1998); Cai (1999); Cai and Silverman (2001); Chicken (2003, 2005). 2.2 Trimmed TI Estimator To construct the proposed trimmed TI estimator, to be referred to subsequently as TTI, we first examine the TI estimator. For a signal of length n as in (1), define the shift operator S as S(y, j) = (y n j+1, y n j+2,..., y n, y 1, y 2,..., y n j ). The operator S shifts the observed data y to the right j times. The TI estimator analyzes all n possible shifts of the data. Set ˆf T I,j = S 1 W η(w S(y, j), λ). Then ˆf T I,j is a wavelet thresholded estimate of the j-shifted data. The data is shifted (S), projected into the wavelet domain (W ), thresholded by some rule η with threshold λ, has 5

6 the inverse DWT W applied and finally the inverse shift S 1 applied. If there were any features in the original data y that were poorly positioned with respect to the wavelet basis, ˆf T I,j may no longer be hampered by this. The TI estimator performs this analysis over all n possible shifts and averages the result: ˆf T I (x) = 1 n ˆf T I,j (x). n In this paper, we use the Haar wavelet basis. This basis is chosen for its ability to model jumps. This ability derives from the fact that all the basis functions in the detail resolution levels contain a jump in the midpoint of their support. The function in the Haar basis at detail resolution level j and translation k is ψ jk (x) = j=1 2 j/2, x [k2 j, (k )2 j ) 2 j/2, x [(k )2 j, (k + 1)2 j ) 0, otherwise. Usually, the use of the Haar basis results in poor visual reconstruction of a noisy function. However, the averaging employed by the TI and TTI estimators negate this undesirable property. In these cases, we retain the jump detecting ability of the Haar basis without a decline in the reconstruction property. Other wavelet bases do not have the jump modeling ability to the extent that the Haar basis does. They tend to oversmooth jumps to an unacceptable degree. As seen in Figure 1, the TI estimate at some points x i have larger variability than others. TTI will modify TI by reducing the variability of the TI estimates at those points. Since the estimate of f on intervals of low variability appears quite good, we do not necessarily wish to reduce the variability at all points x i. Instead, we will modify the n TI shift-unshift estimates so that the maximum variability at the points x i is some α [0, ). Thus, for some points x i, the n TI estimates have been modified, for other points it has not. Let α i be the sample standard deviation of the TI shift-unshift estimates at the x i : α i = sd({ ˆf T I,j (x i ) j = 1, 2,..., n}), i = 1, 2,..., n, where sd refers to the usual sample standard deviation estimate. Let α T I = max 1 i n α i. Then for α α T I no modifications are made to TI. The effective range of α is therefore restricted to [0, α T I ] for a particular set of observed data y. For a specified value of α [0, α T I ], the variability at x i is reduced by iteratively removing extreme values of the initial n TI estimates ˆf T I,j (x i ). The median of the these n values is used as the center point. Estimates farthest from this center are removed one at a time until the first time the standard deviation is at or below α. Let ji α be the subsequence of length n i of the original n j-shifts values {1, 2,..., n} that corresponds to those shifts retained after this iterative process at x i. Then the TTI estimate using this particular α is ˆf T T I,α (x i ) = 1 ˆf T I,j (x i ). n i j j α i 6

7 Note that the same α is used for all i = 1, 2,..., n, but the set ji α then TTI and TI are equivalent. The TTI estimate is ˆf T T I = ˆf T T I,ˆα will vary with i. If α α T I where ˆα is an estimate of that α which provides the best estimate of f as measured by the mean squared error (MSE). Ideally, this is α o = arg min α MSE(f, ˆf T T I,α ) (3) but this cannot be found since f is unknown. Instead, we replace f with the data, ˆα = arg min α MSE(y, ˆf T T I,α ). (4) We can interpret ˆf T T I,α o as an oracle estimator. It knows the best α to choose so that the MSE is minimized. In the simulations in section 3, we compare TTI with the data driven estimate of α from (4) to the oracle estimate using α from (3). To illustrate this, consider Figures 2 and 3. Figure 2 shows the MSEs calculated for 50 equally spaced α values in [0, α T I ]. The function being estimated is blocks as shown in Figure 1 sampled at 256 equispaced points with an snr of 5. The upper plot is the MSE calculated between y and ˆf T T I,α while the lower plot replaces the observed data y with the unobserved function f. The oracle selects the best possible value of α, represented by the vertical dotted line in the lower plot. Using this α the oracle estimate ˆf T T I,α o has an MSE of TTI uses y to determine the best value of α. It is shown in the upper plot as a vertical dashed line. Using this value of α TTI has an MSE of The shape of the MSE curve based on y is quite similar to the curve with f, although the magnitudes are not close. The best α values are not exactly the same for each plot, but they are close. Additionally, the MSEs between each estimate and f are quite similar, and both are an improvement over the MSE between f and the TI estimate (0.475). The oracle estimate and TTI have MSEs that are about 12% lower than the MSE of TI. Figure 3 shows the variability of the original TI shift-unshift estimates for this example data. The sample standard deviation is used to measure this variability. The values of ˆα and α o from Figure 2 are shown as horizontal dashed and dotted lines, respectively. The TTI estimator will reduce the standard deviation at each point x i where the standard deviation of the shift-unshift estimated for that point is above the line. These are the points where the TI shift-unshift estimates disagree the most and coincide with those portions of blocks near the jumps as seen in Figure 1. Many points x i will experience no modification. These are the x i that have standard deviation are below the line and correspond to the smooth portions of blocks. 3 Simulation Next, we compare the performance of TTI with TI, the oracle and VisuShrink. Using sample sizes ranging from n = 128 to n = 2048 and signal to noise ratios of 5 and 7, these methods were tested on the Blocks and Heavisine functions of Donoho and Johnstone (1994), and on a flat piecewise function defined on [0,1] with a jump at 1/3. Plots of these three functions 7

8 MSE (y) α MSE (f) α Figure 2: MSE curves for 50 α [0, α T I ]. The upper plot is contains the MSEs between TTI and y, the lower plot is between TTI and f. The optimal α in the upper plot is shown with a vertical dashed line. In the lower plot, it is shown as a vertical dotted line. along with their noise contaminated counterparts can be seen in Figure 4. All functions were analyzed using the Haar wavelet with periodic boundary handling, and the lowest resolution level, j 0, set to 4. It is worth noting that even though the TI estimator considers n shifts of the data, the DWT is only unique for 2 j 0 of these shifts when using periodic boundary handling. It is therefore only necessary to shift, denoise and unshift the data 16 times prior to applying the TTI method, which in turn only needed to consider a maximum of 16 points for trimming at each step. This results in a considerably faster process than having to consider n vectors. Table 1 shows simulation results of 100 repetitions with 50 α values for different combinations of function, SNR and sample size. The numbers presented in the table are the ratios of the average MSE of the TTI method compared to TI, VisuShrink and the oracle. Bolded numbers indicate cases where the average MSE of TTI is significantly different than the compared method at the 0.05 level of significance. For the two irregular functions tested, Blocks and Jump, TTI significantly outperformed TI on average in all cases. In these cases, 8

9 sd of TI Figure 3: The standard deviations of the TI shift-unshift estimates at each point x i. The values of α from Figure 2 are shown as horizontal lines. The dotted line is α o, the dashed line is ˆα. the MSE of the TTI estimator was between 9% and 25% smaller than the MSE of the TI. For the smooth function tested, Heavisine, the TTI estimator is only clearly superior to TI for the small sample size of 128. When tested with a sample size of 256 and a SNR of 5, the MSE of the TTI is significantly worse than the TI by about 7%. However in all other cases, the difference between the MSE of the TI and TTI estimators is not significant. It is also worth noting that, with the exception of the Heavisine function sampled at 256 points, the MSE of the TTI estimator was never significantly different than that of the oracle. Table 2 shows the proportion of the maximum standard deviation of the TI that the TTI estimator and the oracle choose to trim until reaching. For example, when given a sample of 128 points from the Blocks function with an SNR of 5, the TTI removed extreme values until the standard deviation was 30.58% that of the TI, whereas the oracle removed extreme values until the standard deviation was 31.46% that of the TI. This table indicates that in general, the TTI estimator reduces the variability too much. For the Blocks and Jump functions, where the MSE of the TTI was never significantly worse than the MSE of 9

10 Blocks Heavisine Jumps Noisy Blocks Noisy Heavisine Jumps Figure 4: Samples of the three test functions used in this paper. The top plots are of the functions sampled at 512 points. The bottom plots are the same functions with noise added to represent a SNR of 5. the oracle, this difference was relatively small, with the maximum difference being around 4%. However for the smoother Heavisine function, with the exception of a sample size of 128, this difference tends to become more dramatic. In the worst case, with a sample size of 256 and SNR of 5, the TTI method reduced the variability by as much as 22% more than what it should have been reduced by according to the oracle. Figures 5 and 6 show some example reconstructions. In Figure 5 single estimates using TI and TTI are compared. Note how the TTI estimate more accurately models the jumps and clearly results in a lower MSE. Figure 6 shows similar results when comparing the average of 10 TI and 10 TTI estimates. 10

11 SNR = 5 SNR = 7 Function n TI VisuShrink Oracle TI VisuShrink Oracle Blocks Heavisine Jump Table 1: Ratios of the average MSE of TTI versus the average MSEs of TI, VisuShrink and the Oracle after 100 reps for three different test functions and various sample sizes with signal to noise ratios of 5 and 7. Bold cells indicate a significant difference between the average MSE of TTI and the method indicated. SNR = 5 SNR = 7 Function n ˆα/α T I α o /α T I ˆα/α T I α o /α T I Blocks Heavisine Jump Table 2: Maximum allowed standard deviation as a proportion of the maximum standard deviation of TI (α T I ) as chosen by TTI and the oracle. 11

12 Figure 5: Comparison of a single estimate of TI (black) with a single TTI estimate (red). The test function is blocks (dotted) sampled at n = 128 points. The snr is 5 and the range of α is covered by 50 steps. 12

13 Figure 6: Comparison of the average of 10 TI estimates (black) with the average of 10 TTI estimates (red). The test function is blocks (dotted) sampled at n = 128 points. The snr is 5 and the range of α is covered by 50 steps. 13

14 4 Discussion The TTI method presented here was developed with the goal of reducing the MSE of the TI estimator. For irregular, jumpy functions, this new estimator does just that, with the decrease in MSE lying between 9% and 25%. On the other hand, when dealing with smooth functions, the TTI method should not be expected to offer any improvement over TI. When attempting to reconstruct a function by averaging together many representations, having more representations will generally lead to a smoother reconstruction. Since the TTI tends to reduce the number of representations available for averaging, its reconstructions will be less smooth. This is not ideal when dealing with smooth functions. That said, the results from the previous section indicate that when the sample sizes are large, the MSE of the TTI estimator is not significantly different from that of the TI estimator. References Cai, T. (1999). Adaptive wavelet estimation: A block thresholding and oracle inequality approach. Ann. Statist Cai, T. and Silverman, B. (2001). Incorporating information on neighboring coefficients into wavelet estimation. Sankhya Ser. B Chicken, E. (2003). Block thresholding and wavelet estimation for nonequispaced samples. J. Statist. Plann. Inference Chicken, E. (2005). Block-dependent thresholding in wavelet regression. Journal of Nonparametric Statistics Coifman, R. and Donoho, D. (1995). Translation-invariant wavelet denoising. In A. Antoniadis and G. Oppenheim, eds., Wavelets and Statistics. Springer-Verlag, New York, Daubechies, I. (1992). Ten Lectures on Wavelets. SIAM, Philadelphia. Donoho, D. and Johnstone, I. (1994). Ideal spatial adaptation via wavelet shrinkage. Biometrika Donoho, D. and Johnstone, I. (1998). Minimax estimation via wavelet shrinkage. Annals of Statistics Jin, J. and Shi, J. (2001). Automatic feature extraction of waveform signals for in-process diagnostic performance improvement. Journal of Intelligent Manufacturing Mallat, S. (1999). A Wavelet Tour of Signal Processing. Academic Press, San Diego, 2nd ed. Nason, G. and Silverman, B. (1995). The stationary wavelet transform and some statistical applications. In A. Antoniadis and G. Oppenheim, eds., Wavelets and Statistics. Springer-Verlag, New York,

15 Ogden, R. (1997). Essential Wavelets for Statistical Applications and Data Analysis. Birkhauser, Boston. Vidakovic, B. (1999). Statistical Modeling by Wavelets. John Wiley and Sons, New York. Walker, E. and Wright, S. P. (2002). Comparing curves using additive models. Journal of Quality Technology