Instrument Specific Signal

Transcription

1 Wavelet transform used to preprocessing of Apple near infrared spectra Wang Xiaoli,, Zhang Xiaoli *, Zhou Guomin. Beijing Forestry University, BeiJing, Institue of agricultural information, Chinese academy of agricultural sciences, BeiJing, 0008 Abstract: Due to original near infrared (NIR) spectra data containing a lot of noise and large amount of data, so it is very necessary to do preprocessing for spectral data before spectral data analysis. Near infrared spectrum data preprocessing mainly includes two tasks: one is the noise reduction to improve the robustness of the model and the accuracy of the prediction results; second, data compression, in order to facilitate the storage of data, and improve the modeling speed. Traditional NIR spectral data preprocessing s have limitations, and are difficult to get a satisfactory effect in both aspects. This paper will use wavelet analysis on preprocessing of Apple NIR spectral data, select peak signal-to-noise ratio (PSNR) and normalized correlation coefficient (NC) as evaluation indexes, and compared with the commonlyused Savitzky-Golay smoothing filter and multiple scattering correction, wavelet can not only effectively realize the data compression, and also superior to the other two preprocessing algorithm in terms of noise reduction and spectral details maintain. Keywords: Near infrared spectrum; Preprocessing; Wavelet analysis; Smooth; Multiple scattering correctio. Introduction NIR (Near Infrared Reflectance Spectroscopy, NIRS) analysis technique can make nondestructive testing on physical properties, chemical properties and mechanical properties of organic samples quickly, efficiently and accurately, is a new nondestructive testing technology, is widely drawn attention and promoting around the world [~3].The original near infrared spectrum data collected by instrument, in addition to contain the information reflecting the sample composition, also inevitably contain noise signals generated from high frequency random noise, baseline drift, signal background, non-uniform sample, light scattering[4].these noise signals will produce interfere to the spectrum information, effect the correction modeling, thereby reduce the accuracy of components and properties prediction of unknown samples. On the other hand, in the near infrared spectrum analysis, spectral data is generally larger, and the required storage space is also bigger, and the required modeling time is longer[5]. So if you want to analyze the composition of the sample by near infrared spectroscopy, you should do spectra preprocessing first, there are mainly two tasks in preprocessing: one is the noise reduction, to reduce the influence of various kinds of interference, and improve the robustness of the model and the accuracy of the prediction results; second, data compression, in order to facilitate the storage of data, and improve the modeling speed. The preprocessing s of near infrared spectral data mainly include Savitzky-Golay convolution smoothing of spectrum data, multiple scattering Correction (MSC), etc. Savitzky-Golay convolution smoothing is most common used in near infrared spectrum signal denoising, this assumes that the noise contained the spectrum is white noise (mean value 0), through smooth filtering can improve the SNR, but also lead to distortion of signal, such as the mutation signal becomes relatively flat [6~7]. MSC put forward by Martens and others, this is mainly used to eliminate the influence of scattering generated by uneven distribution and different sizes of the sample particles on near infrared spectra, and that is mainly eliminate the ideal linear scattering effect. Under the condition of the absorbance and concentration of sample maintained a good linear relationship, MSC correction effect is better; but in many practical cases, the effect of sample particle scattering on the near infrared spectrum is not a linear, so MSC needs to be improved [8~9]. Wavelet analysis has the characteristics of multi-resolution analysis, which has the ability of denoting local signal characteristics in both time domain and frequency domain, known as the "mathematical microscope", is a kind of new in the field of signal processing following Fourier analysis[0~]. This paper uses wavelet analysis in preprocessing of near infrared spectral data of apples, compared with the commonly-used Savitzky-Golay smoothing filter and the multiple scattering correction, wavelet has its unique feature in noise reduction and spectrum detail signals maintaining and data compression.. Wavelet analysis used for near infrared spectral data preprocessing Wavelet transform coefficient Spectra Remaining Coefficients Instrument Specific Signal Figure. The Wavelet Transform of NIR Spectra The main roles of wavelet transform used for near infrared spectrum data preprocessing are denoising and data compression. Near infrared spectrum data can be decomposed into the background information (low frequenc, component information (intermediate frequenc and noise information (high frequenc by wavelet transform, through appropriate treatment of low frequency and high frequency coefficients, can effectively remove the background information and noise information, thus to achieve the purpose of de-noising and data compression, and improve the precision of model robustness and forecasting for subsequent analysis. The preprocessing steps of near infrared spectral data by wavelet analysis are as follows: ) Wavelet decomposition: to determine a wavelet basis function through repeatedly screening by experiments, and determine the decomposition layer number N, and then carry on N-layer wavelet decomposition to the original spectrum signals; Journal of Residuals Science & Technology, Vol. 3, No. 7, DEStech Publications, Inc. doi:783/issn /3/7/53

2 ) Coefficient of processing: choose soft threshold, the threshold determined by equation (). Then carry on soft threshold quantization processing from the first layer to the N layer of high frequency coefficient; according to the prior knowledge, delete the coefficient thought to be sufficiently small but not useful information from low frequency coefficients. sgn( Y )( y t ) T Y, t) = 0 Y t ( () Y < t In equation () the sgn is a symbolic function, t is the threshold, Y is the high frequency wavelet coefficient, T is wavelet coefficient after soft threshold processing. 3) Wavelet reconstruction: according to the N-layer low frequency coefficient after processing and quantitative treatment, spectral signal reconstruction is carried on the high frequency coefficients of the first layer to the N layer. 3. Preprocessing evaluation index ) PSNR Peak signal to noise ratio PSNR This paper introduces peak signal-to-noise ratio PSNR to evaluate the denoising ability of preprocessing, the greater of PSNR, the denoising ability is stronger, and the preprocessing effect is better, which are defined as follows: [ I max I min ] [55 0] PSNR = 0 log0 ( ) = 0 log0 ( ) () In equation () and (3), ) both size are 557 MSE MSE M M MSE = [ I ( x, I( x, ] M M x= y= I ( x, y is the original spectrum data matrix, ( x, I and 30 ; MSE is called as mean square error; max I is the spectral data matrix after preprocessing, (3) I are the maximum and minimum of spectral data, min respectively. ) NC Normalized correlation coefficient This paper adopts the normalized correlation coefficient NC to evaluate the similar degree of spectral image before and after preprocessing, the greater of NC, the more spectral details are maintained after preprocessing, spectrum after preprocessing is more similar to the original spectrum. NC is defined as follows: N N In equation (4), ( x, 4. Experimental analysis NC x= y= = N N W is original spectral data matrix, ) [ ] x= y= y is spectral data matrix after preprocessing. To choose uniform size, scatheless Huaniu apples (producing area of Gansu), common Fuji apples (producing area of Shaanxi province), special grade Fuji apples (producing area of Shandong province), 30 of each kind to constitute experimental samples. Apple samples are stored at the laboratory with temperatures of 0 ~ 5 for hours. Spectrum collector used for experiment is Antaris II near infrared spectrum analyzer of Thermo Fisher company. Antaris II near infrared spectrum analyzer shall be warmed up for an hour, to collect near infrared spectrum of the apple by reflection integrating sphere model, the near infrared spectrum analyzer scans 3 times on each sample to obtain the average diffuse reflection spectrum of the sample. Spectral scanning wave number is 4000~0000cm-, scanning interval is 3.856cm-, it collects a 557-dimensional spectral data from each sample. In order to reduce error, each apple sample is sampling along the equator trajectory for 3 times, take the average value as the final test data. The original near infrared spectrum diagram of three kinds of apples are as shown in figure (), 3(), 4(). This paper adopts SavitZky-Golay convolution smoothing, multiple scattering correction and wavelet analysis for preprocessing the near infrared spectrum data of three kinds of apples, respectively, the results are shown in figure, figure 3. SavitZky- Golay convolution smoothing selects filter bandwidth of 9, the wavelet analysis selects db3 wavelet for 3-layer decomposition of spectral data of apples, according to the noise intensity using the soft threshold to separate signal and noise. It can be seen from figure to figure that, the spectral image after preprocessing by SavitZky-Golay convolution smoothing and wavelet analysis has maintained high similarity with original spectral image, while the spectral image after preprocessing by MSC has great difference with original spectral image. Table lists peak signal-to-noise ratio PSNR of three kinds of apple spectrum after preprocessing by three preprocessing s, table lists normalized correlation coefficient NC of three kinds of apple spectrum after preprocessing by three preprocessing s. It can be seen from that, after the preprocessing of three kinds of apples spectra, PSNR of the wavelet analysis is the maximum, SavitZky-Golay smoothing is the second, MSC is the minimum, and wavelet analysis shows the strongest denoising ability. After the preprocessing of three kinds of apples spectra, SavitZky-Golay smoothing and the wavelet analysis can maintain NC value to, which shows that two kinds of preprocessing s can better maintain the similarity of the spectral data, that is can well reserve the detail of the spectral information. MSC is somewhat worse in the aspect of similarity maintaining. (4) Journal of Residuals Science & Technology, Vol. 3, No. 7, DEStech Publications, Inc. doi:783/issn /3/7/53

3 .6.4 Hua niu 花牛苹果 Apple.6.4 SavitZky-Golay Convolution 卷积平滑法 smoothing HN.. 单位 unit Wave Number/cm wave number 波数 /c m () NIR of common Huaniu apples Hua Niu Apples () SavitZky-Golay The spectra preprocessed by SavitZky-Golay convolution algorithm.6.4 花牛苹果 Wave Number/cm Wave Number/cm- Figure. preprocessing of NIR spectra of Huaniu apples The NIR Spectra of Wave Number/cm-.3.. () NIR of common Fuji apples The General Fuji Apples Wave Number/cm- () The spectra preprocessed by SavitZky-Golay convolution algorithm.6.4. The Spectra Preprocessed by SavitZky-Golay Convolution Smoothing Wave Number/cm Wave Number/cm- Figure 3. preprocessing of NIR spectra of common Fuji apples Journal of Residuals Science & Technology, Vol. 3, No. 7, DEStech Publications, Inc. doi:783/issn /3/7/53

4 .6 The NIR Spectra of Wave Number/cm-.3.. () NIR of special grade Fuji apples The Super Fuji Apples Wave Number/cm- () The spectra preprocessed by SavitZky-Golay convolution algorithm.6.4. The Spectra Preprocessed by SavitZky-Golay Convolution Smoothing Wave Number/cm Wave Number/cm- Figure 4. Preprocessing of near infrared spectral data of special grade Fuji apple Table. PSNR comparison of three preprocessing s SavitZky-Golay smoothing preprocessing algorithm wavelet analysis Huaniu apple common Fuji apple Special grad Fuji apple preprocessing algorithm Table. NC comparison of three preprocessing s SavitZky-Golay smoothing MSC wavelet analysis Huaniu apple common Fuji apple Special grad Fuji apple Based on above analysis, the wavelet analysis can not only effectively realize the data compression, but also show superiority in both noise reduction and spectral details maintain. In general, in preprocessing of near infrared spectral data of apple, the wavelet analysis is superior to SavitZky-Golay smoothing and MSC. 5. Conclusion Wavelet transform used to preprocessing of Apple near infrared spectra, on the one hand, can remove noise effectively, thus improve the prediction accuracy and robustness of the subsequent modeling; on the other hand, it effectively realizes the compression of the data, reduce the data storage space and improve the speed of the subsequent modeling. NIR analysis technology may affected by various factors, such as: the influence of sample scattering property, physical property, collinearity, etc., it can select different preprocessing according to different analysis systems, and different analysis purposes in specific application, can even be combined with several different preprocessing s. But what kind of other preprocessing can be exactly combined with and the operation sequence of these s still need to be continuously explored and optimized in practice. Acknowledgements National High-tech R&D Program (863 Program): Orchard and canola crop production Digital Management Platform based on the model, No.03AA0405; SFA Public Welfare Forestry Scientific Research Project: Study on Key Technologies for Monitoring and Warning of Major Forest Plague of Insects, No References [] Wang P, Zhang H. Yang H, et al., Rapid determination of major bioactive isoflavonoid compounds during the extraction process of Kudzu (Pueraria lobata) by Near-infrared transmission spectroscopy, Spectrochim. Acta A. 05, 37, pp, [] Rohman A, Nugroho A, Lukitaningsih E., Application of vibrational spectroscopy in combination with chemometrics techniques for authentication of herbal medicine, Appl. Spectrosc. Rev, 05, 49, pp, Journal of Residuals Science & Technology, Vol. 3, No. 7, DEStech Publications, Inc. doi:783/issn /3/7/53

5 [3] Zhang C, Su J., Application of near infrared spectroscopy to the analysis and fast quality assessment of traditional Chinese medicinal products. Acta Pharmaceut. Sinica, 04, 4, pp, 8-9. [4] Walker D D, Beaucamp A T H, Bingham R G., The precessions process for efficient production of aspheric optics for large telescopes and their instrumentation. Proceedings of SPIE-The International Society for Optical Engineering, 003, 48(4), pp, [5] Shatskiy A F, Litasov K D, Palyanov Y N., Phase relations in carbonate systems at pressures and temperatures of lithospheric mantle: review of experimental data. Russian Geology & Geophysics, 05, 56(-), pp, 3-4. [6] Walker D D, Beaucamp A T H, Bingham R G., Novel CNC polishing process for control of form and texture on aspheric surfaces. Proceedings of SPIE-The International Society for Optical Engineering, 00, 4767, pp, [7] Gao B, Xie D G, Yao Y X, et al., New technology of ballonet tool for polishing. Optical Technique, 004, 30(3), pp, [8] Xie D G, Gao B, Yao Y X, et al., Study of Local Material Removal Model of Bonnet Tool Polishing. Key Engineering Materials, 006, , pp, [9] Song J F, Yao Y X, Xie D G., Orthogonal experiment and analysis on polishing parameters of bonnet polishing. Optical Technique. 009, 35(), pp, [0] Peyre P, Fabbro R. Laser shock processing: a review of the physics and applications. Optical & Quantum Electronics, 995, 7(), pp, 3-9. [] Peyre P, Scherpereelm X, L Berthem, et al., Current trends in laser shock processing. Surface Engineering, 997, 4, pp, [] WAN, Jun, and Hang ZHANG, Discussion on City Slow-Traveling Traffic System Based on Low-Carbon Concept. Western China Communications Science & Technology, 00, (7), pp. -5. Journal of Residuals Science & Technology, Vol. 3, No. 7, DEStech Publications, Inc. doi:783/issn /3/7/53