Gap-Filling of Solar Wind Data by Singular Spectrum Analysis

Size: px

Start display at page:

Download "Gap-Filling of Solar Wind Data by Singular Spectrum Analysis"

Giles Moore
5 years ago
Views:

1 Gap-Filling of Solar Wind Data by Singular Spectrum Analysis Dmitri Kondrashov University of California, Los Angeles Yuri Shprits University of California, Los Angeles Motivation 1. Observational data sets in the geosciences are often short, contain noise (errors) and are gappy: this is both an obstacle and an incentive. Continuous data is needed for modeling (boundary conditions), standard spectral estimation algorithms, etc. 2. Phenomena in the geosciences often have both regular ( cycles ) and irregular ( noise ) aspects. Regularities include (quasi-)periodicity - singular spectrum analysis (SSA) Powerful method for spectral estimation, noise filtering and gap-filling (producing estimates of the missing points) 2. Space Physics Application: Apply SSA to fill-in large gaps in historical solar wind and interplanetary magnetic field (IMF) data required for the state-of-the art empirical global magnetospheric magnetic field models. Important for radiation belt modeling. dkondras@atmos.ucla.edu

2 Outline 1. Short brief on Singular Spectrum Analysis. 2. Illustrate SSA gap-filling on synthetic examples. 3. Solar driver/inner magnetosphere response SSA gap-filling of combined continuous innner-magnetospheric indices and gappy solar wind and IMF data. 4. Verification of the algorithm on synthetic gaps in continuos data. 5. First results of filled-in gappy data; comparison with alternatives. 6. Conclusions.

3 SSA of Nino-3 index (El-Nino) SSA decomposes (geophysical & other) time series into Temporal EOFs (T-EOFs) and Temporal Principal Components (T-PCs), based on the seriesʼ lag-covariance matrix Selected parts of the series can be reconstructed, via Reconstructed Components (RCs) Time series T-EOFs RCs λ Pairs oscillations (nonlinear) sine + cosine pair Vautard & Ghil (1989: VG) Physica, 35D, Statistical dimension k SSA isolates oscillatory behavior via paired eigenelements.

4 SSA Power Spectra & Reconstruction A. Transform pair: M X(t + s) = a k (t)e k (s), e k (s) EOF For given window M, ekʼʼs are adaptive filters (empirical orthogonal functions) M a k (t) = X(t + s)e k (s), a k (t) P C s=1 k=1 the akʼʼs are filtered time series, principal components in time domain. B. Power spectra S X (f) = M S k (f); S k (f) = ˆR k (s); R k (s) 1 T k=1 T 0 a k (t)a k (t + s)dt C. Reconstruction X K (t) = 1 M M a k (t s)e k (s); k K s=1 in particular: K = {1, 2,..., S} or K = {k} or K = {l, l + 1; λ l λ l+1 }

5 SSA Gap-filling algorithm 1. Choose window M and set K=1. Flag fraction of dataset X(t)(t=1:N) as missing for cross-validation. 2. Update mean and lag-covariance matrix, find leading K EOFs D = X(1) X(2).. X(M) X(2) X(3).. X(M + 1)..... a X(N 1)... X(N 1) X(N ) X(N + 1).. X(N) C X = 1 N Dt D; C X E k = λ k E k 3. Reconstruct missing points using K EOFs A k (t) = M j=1 X(t + j 1)E k(j) R K (t) = 1 M t k K Ut j=l t A k (t j + 1)E k (j); 4. When convergence for missing points: K = K +1. Check cross-validation å error, and Go to Step 2 if necessary. Extension of spatial EOFs for gapfilling: Beckers, J. and Rixen, M.: EOF calculations and data filling from incomplete oceanographic data sets, J. Atmos. Ocean. Technol., 20, , Utilize both spatial and temporal correlations to iteratively compute maximum likelihood estimates of mean and lag-covariance matrix => can be applied to very gappy data. A few K leading EOFs correspond to the smooth modes, while the rest is noise and can be discarded. Cross-validation provides error estimates and optimum SSA parameters. D. Kondrashov and M. Ghil, 2006: Spatiotemporal filling of missing points in geophysical data sets, Nonl. Proc. Geophys., 13,

6 Synthetic Gaps in Noisy Oscillatory Signal % &'()*++*,-(-&.(/ % $ 0 3'()*++*,-(-&.(/ 2 % %2 0 <'(>?9==@&+*5&0*9,(ABC(<??9? %72 % $ 6 % 897(9:(;95<= % 1'()*++*,-(-&.(// % $ 0 5'()*++*,-(-&.(// :'(CCD(=.<30?E;F(BG % 7 7 )?<HE<,3I x(t) = sin( 2π 2π 300t) cos( 40 t + π 2π 2 sin 120 t)

7 Synthetic Gaps in Multivariate data x10 Dataset with 50% of missing data e+00 Estimated smooth component Space Space -2.00e e x10 Original noisy dataset 2.00e x10 Smooth component of full dataset 2.00e e+00 Space Space Time Time e x e Time Time

8 Gappy Solar Driver/Continuous Magnetospheric Response Ni B. et al., JGR, 2009 Gaps in solar wind satellite data before the launch of the WIND spacecraft in Continuous inner-magnetosphere indices (Kp, Dst) are ground measured time-lagged magnetic disturbances caused by interaction of Earthʼs magnetosphere with solar wind. A measure of geomagnetic activity used to assess the severity of magnetic storms.

9 Filling-in Synthetic Gaps: Bz ) '( ) $ % & & & &$ &% $ % $ 9% Verification skill: Bz CORR Norm. RMS $ % : : : :$ :% $ $ $ $$ $% ; ; ; ;$ ;% % % % %$ %% < < < <$ <% & *+,,-.+/ =+>-)?@8A53B *+617+8/ Gaps of are applied to hourly data MSSA mode :,2*;*<8=*0> '( - $ % &? )*+,-./01234 $ % & B B A $ % & )*+,-./ SSA Window M=25hr.

10 Filling-in Synthetic Gaps: Dynamic Pressure & '()**+() -.//) *, 5)( Verification skill: Dynamic Pressure CORR Norm. RMS, $ % & & & &$ &% & $ % $ 9% & $ % : : : :$ :% $ & $ $ $$ $% ; ; ; ;$ ;% % & % % %$ %% < < < <$ <% & =.>),?@8+(*A Gaps of are applied to hourly data; SSA Window M=15hr. B,> A: ( ' MSSA mode :, ;+*/</=9>/4? )*+,,-*+ $ % &./ *,5 ' $ % & ' ( $ % &./ *,5 1

11 Filling-in Gaps: Bz ) '( $ % & & & &$ &% $ % $ 9% $ % : : : :$ :% $ $ $ $$ $% ; ; ; ;$ ;% % % % %$ %% < < < <$ <% & *+,,-.+/ 0123 =+>-)?@8AB3C 4+/56-/78/) ( )* 778 ( 9:;5 ' / <,= > $ % & +,-./ ( $ % & ( ' $ % & +,-./ / Comparison with Interpolated solar wind and IMF data by Qin et al. 2007: SPACE WEATHER, VOL. 5, S11003, doi: /2006sw000296, 2007

12 Filling-in Gaps: Dynamic Pressure & '()**+() -.//) *, 5.1, & & & & $ % & & & &$ &% $ % $ 6% $ % $ 7% $ $ $ $$ $% $ 8% % % % %$ %% $ 9% & >; ) ( ' *+,--.+, 778 9:;- <0= 2 ' $ % & /01, ( $ % & (? $ % & /01, Comparison with Interpolated solar wind and IMF data by Qin et al. 2007: SPACE WEATHER, VOL. 5, S11003, doi: /2006sw000296, 2007

SSA-MTM Toolkit Freeware ported to Sun, Dec, SGI, PC Linux, and Mac OS X Graphics support for IDL and Grace (free) Includes Blackman-Tukey FFT, Maximum Entropy Method, Multi- Taper Method (MTM), SSA

13 SSA-MTM Toolkit Freeware ported to Sun, Dec, SGI, PC Linux, and Mac OS X Graphics support for IDL and Grace (free) Includes Blackman-Tukey FFT, Maximum Entropy Method, Multi- Taper Method (MTM), SSA and M-SSA. Spectral estimation, decomposition, reconstruction & prediction. Significance tests of oscillatory modes vs. noise. SSA Gap-filling. For details and publications, please visit: TCD 30/32

14 Conclusions Promising first results of applying SSA to estimate the missing data in gappy solar wind data by using inner-magnetospheric indices. Future work: Consider longer datasets, varying temporal scales. Closer look at the SSA modes responsible for successful reconstruction. Include more solar wind parameters: solar wind speed (Vsw) and solar wind density (Nsw) and other components of magnetic field () Do systematic search for optimum combination of inner-magnetospheric indices Ghil M., R. M. Allen, M. D. Dettinger, K. Ide, D. Kondrashov, M. E. Mann, A. Robertson, A. Saunders, Y. Tian, F. Varadi, and P. Yiou, 2002: Advanced spectral methods for climatic time series, Rev. Geophys.,40(1), pp , /2000RG D. Kondrashov and M. Ghil, 2006: Spatio-temporal filling of missing points in geophysical data sets, Nonl. Proc. Geophys., 13, SSA-MTM Toolkit,

Project RECOLOUR (REconstruction of COLOUR scenes) - SR/00/111 RESEARCH PROGRAMME FOR EARTH OBSERVATION STEREO II - BELGIAN SCIENCE POLICY

Project RECOLOUR (REconstruction of COLOUR scenes) - SR/00/111 RESEARCH PROGRAMME FOR EARTH OBSERVATION STEREO II - BELGIAN SCIENCE POLICY MUMM - RBINS Cloud filling of TSM, CHL and SST remote sensing products by the Data Interpolation with Empirical Orthogonal Functions methodology (DINEOF), application to the BELCOLOUR-1 database. Damien