The Curse of Dimensionality. Panagiotis Parchas Advanced Data Management Spring 2012 CSE HKUST

Transcription

1 The Curse of Dimensionality Panagiotis Parchas Advanced Data Management Spring 2012 CSE HKUST

2 Multiple Dimensions As we discussed in the lectures, many times it is convenient to transform a signal(time series, picture) to a point in multidimensional space. This transformation is handy as we can apply conventional database indexing techniques for queries such as NN, or search This transform may lead as to very high dimensionality (hundreds of dimensions) In high dimensionality, there is a number of problems (geometrical and index performance) that are usually referred to as the Curse of Dimensionality In this presentation: Some intuition about the Curse. Explore techniques that try to overcome it.

3 The Curse Volume and area depend exponentially on the number of dimensions. No intuitive effects: Geometric effects concerning the volume of hyper cubes and spheres Indexing effects Effects in the Database environment (query selectivity)

4 a)geometric Effects Lemma: A sphere touching or intersecting all the d-1 borders of a cube, will contain the center. True for 2D and 3D (by visualization) It should be true for higher dimensions (hyper cubes, hyper spheres) It is NOT!

5 b)indexing Effects

6 b)indexing effects[cont] The higher the dimensionality the more coarse the indexing (which renders it useless ) This affects all the indexing techniques. CHRISTIAN BOHM, 2001

7 c)query selectivity

8 When is NN meaningful? Kevin Beyer et all, 1999

9 What is the spell for the curse? Various attempts of multidimensional indexing where proved that don t make sense for a big category of data distributions [CHRISTIAN BOHM, 2001] There has been a lot of research on Dimensionality Reduction techniques. They basically apply ideas of compression, to data, in order to reduce the dimensionality. In the next we will focus mainly in Time Series.

10 Introduction 11.5 Euro-HK$ exchange rate D space Data points 9/1/ /1/ /1/ /1/2011 1/1/2012 2/1/2012

11 DFT DWT SVD APCA PAA PLA Tutorial in IEEE ICDM 2004 by Dr. Keogh

12 Discrete Fourier Transform (DFT) Every signal, no matter how complex, can be represented as a summation of sinusoids Idea: Find the hiddensinusoids that form the time series Store twonumbers for each: (A, φ) magnitude phase Larger frequency sins generally correspond to details of the time series We can discard them and keep just the first ones (low frequency) Then we use Inverse DFT to get the approximation of the time series. DFT: Inverse DFT:

13 DFT example TIME SERIES A φ DFT We store 8+8=16 values!

14 DFT example(cont) A φ Approximate TS DFT approximation IDFT

15 DFT

16 DFT (pros & cons) O(nlogn) complexity Hardware Implementations Good ability to compress most signals Many applications Not good approximation for bursty signals Not good approximation if the signal contains both flat and busy segments Cannot support other distance metrics Contains info only for the frequency distribution The time domain?

17 Why DFT is not enough? 2 It gives us information about the frequency component of a time series, without telling where this frequency lies in the time domain x(t)=sin(5*t)+sin(10*t) 3500 Fourier Decomposition (Spectrum) 1 z(t)=sin(5*t), sin(10*t)

18 Discrete Wavelet Transform(DWT) This comes as a solution to the previous problem. The wavelet transform contains information both for the frequency domain AND the time domain. The basic Idea is to express the time series as a linear combination of a wavelet basis function. Haar Wavelet is mostly used:

19 DWT: Graphical Intuition The wavelet is stretchedand shifted in time and this is done for all the possible stretches and shifts. Afterwards, each is multiplied with the TS. We keep only the ones with high product.

20 DWT: Numerical Intuition Resolution Averages Details 4 [ ] 2 [8 4] [1-1] 1 [6] [2]

21 Example taken by Stollnitz, E. et all 1995

22 DWT 11.4 Wavelet Approximation In our example: We had 128pts The approximation (red line) uses only 16 haar coefficients

23 DWT(Pros & Cons) Good ability to compress stationary signals. Fast linear time algorithms for DWT exist. Able to support some interesting non-euclidean similarity measures. Signals must have a length n= 2 some_integer Works best if Nis = 2 some_integer. Otherwise wavelets approximate the left side of signal at the expense of the right side. Cannot support weighted distance measures.

24 Singular Value Decomposition(SVD) All the previous methods, try to transform each time series independently of the others. What if we take into account all the Time Series contained in the Database? We can then achieve the desired dimensionality reduction for the specific Dataset

25 q SVD: Basic Idea [1]

26 q SVD: Basic Idea (2)

27 q SVD: Basic Idea (3)

28 SVD [more] The goal is to find the axes with the biggest variance. Highvariance A lot of Important axes Information Axes Low variance axes Little Information/ Noise Axes can be truncated

29 SVD[more] In the previous intuition, we can keep the coefficients of the projections to the new axis. This can be efficiently done by SVD. So we perform the dimensionality reduction in an aggregate way taking into account the whole dataset. This idea was traditionally used in linear algebra for matrix compression. A = UΣV The idea was to find the (nearly) linearly dependent columns of a matrix A and eliminatethem. It can be proved that this compression is optimal. T

30 SVD: compression Projection to the axis denoted by the biggest singular value s1 q MINIMUM information loss Good for compression

31 SVD: Clustering Projection to the axis denoted by the smallest singular value s2 q MAXIMUM information loss Good for clustering

32 SVD(Pros & Cons) Optimal linear dimensionality reduction technique. The eigenvalues tell us something about the underlying structure of the data. Computationally very expensive. Time: O(Mn 2 ) Space: O(Mn) An insertion into the database requires recomputing the SVD. Cannot support weighted distance measures or non Euclidean measures.

33 Piecewise Aggregate Approximation Very simple, intuitive (PAA) Represent the time series as a summation of boxes of equal length. PAA approximation We keep 13 boxes

34 PAA(Pros & Cons) Fast, easy to implement, intuitive The authors claim it is as efficient as other approaches (empirically) Supports queries of arbitrary lengths Supports non Euclidean measures It seems as a simplification of DWT, that cannot be generalized to other types of signals

35 Adaptive Piecewise Constant What about signals with flat areas and peaks? Approximation (APCA) Raw Data (Electrocardiogram) IDEA: generalize PAA so it can automatically adapt itself to the correct box size. (we should now keep both the length and height of the box) Adaptive Representation (APCA) Reconstruction Error 2.61 HaarWavelet or PAA Reconstruction Error 3.27 DFT Reconstruction Error example by E.KeoghIEEE ICDM 2004

36 APCA [more] In order to implement it, the authors propose first a DWT transformation that is followed by merging of the similar, adjacent wavelets. It is very efficient in some specific datasets However the indexing is more complicated than PAA since we need two numbers for each box. That is the reason why is not used very often.

37 Piecewise Linear Approximation (PLA) Linear segments for representation (not necessarily connected) Although efficient in some cases, The implementation is slow and it is not indexable example for visualization only

38 Non Linear Techniques Dimensionality Reduction: A Comparative Review, L.J.P. van der Maaten 2008

39 Non Linear techniques [2] A lot of techniques hveemerged the last years. However,[Maatenet al 2008] compared them with the PCA (equivalent to SVD) and in most of the datasets all these complicated techniques turn out to be worse. The reasons the authors claim, are data over fitting and curse of dimensionality

40 Conclusion All the before mentioned techniques have their strong and weak points. DrKeogh tested them over 65 different datasets with different characteristics: On average, they are all about the same. In particular, on 80% of the datasets they are all within 10% of each other. So the choice for the best method depends on the characteristics of the Dataset