Dimensionality Reduction PCA

Transcription

1 Dimesioality Reductio PCA Machie Learig CSE446 David Wadde (slides provided by Carlos Guestri) Uiversity of Washigto Feb 22, 2017 Carlos Guestri

2 Dimesioality reductio Iput data may have thousads or millios of dimesios! e.g., text data has Dimesioality reductio: represet data with fewer dimesios easier learig fewer parameters visualizatio hard to visualize more tha 3D or 4D discover itrisic dimesioality of data high dimesioal data that is truly lower dimesioal Carlos Guestri

3 Lower dimesioal projectios Rather tha pickig a subset of the features, we ca create ew features that are combiatios of existig features Let s see this i the usupervised settig just X, but o Y Carlos Guestri

4 Liear projectio ad recostructio x[2] project ito 1-dimesio z x[1] recostructio: oly kow z, what was (x[1],x[2]) Carlos Guestri

5 Pricipal compoet aalysis basic idea Project d-dimesioal data ito k-dimesioal space while preservig iformatio: e.g., project space of words ito 3-dimesios e.g., project 3-d ito 2-d Choose projectio with miimum recostructio error Carlos Guestri

6 Liear projectios, a review Project a poit ito a (lower dimesioal) space: poit: x i = (x i [1],,x i [D]) select a basis set of basis vectors (u 1,,u K ) we cosider orthoormal basis: u i u i =1, ad u i u j =0 for i¹j select a ceter x, defies offset of space best coordiates i lower dimesioal space defied by dot-products: (z i [1],,z i [K]), z i [j] =(x i x) u j Carlos Guestri

7 PCA fids projectio that miimizes recostructio error Give N data poits: x i = (x i [1],, x i [D]), N Will represet each poit as a projectio: KX ˆx i = x + z i [j]u j where: x = 1 NX x i ad z i [j] =(x i x) u j N j=1 PCA: Give K<D, fid (u 1,,u K ) miimizig recostructio error: NX error K = (x i ˆx i ) 2 x 2 x 1 Carlos Guestri

8 Uderstadig the recostructio error Note that x i ca be represeted exactly by d-dimesioal projectio: DX x i = x + z i [j]u j j=1 Rewritig error: 2 NX NX error K = (x i ˆx i ) 2 = 4 x + 2 NX = 4 DX 3 2 z i [j]u j 5 = j=1 ˆx i = x + z i [j] =(x i 0 13 DX KX z i [j]u x + z i [j]u j A5 2 NX 4 KX z i [j]u j j=1 x) u j Give K<D, fid (u 1,,u K ) miimizig recostructio error: NX error K = (x i ˆx i ) 2 j=k+1 j=k+1 j=k+1 `>j NX DX = (z i [j]) 2 j=k+1 j=1 3 DX DX DX z i [j]u j u j z i [j]+2 z i [j]u j u`z i [`] 5 2 Carlos Guestri

9 Recostructio error ad covariace matrix NX DX error K = [u j (x i x)] 2 j=k+1 NX DX = DX = j=k+1 j=k+1 DX = N u T j j=k+1 = 1 N m` = 1 N u T j (x i x)(x i x) T u j " X # (x i x)(x i x) T u j u T j u j NX (x i x)(x i x) T NX (x i [m] x[m])(x i [`] x[`]) Carlos Guestri

10 Miimizig recostructio error ad eige vectors Miimizig recostructio error equivalet to pickig (ordered) orthoormal basis (u 1,,u D ) miimizig: error K = N Eige vector: DX j=k+1 u T j u j u = u Miimizig recostructio error equivalet to pickig (u K+1,,u D ) to be eige vectors with smallest eige values Carlos Guestri

11 Basic PCA algoritm Start from m by data matrix X Receter: subtract mea from each row of X X c X X Compute covariace matrix: S 1/N X ct X c Fid eige vectors ad values of S Pricipal compoets: k eige vectors with highest eige values Carlos Guestri

12 PCA example ˆx i = x + KX z i [j]u j j=1 Carlos Guestri

13 PCA example recostructio ˆx i = x + KX z i [j]u j j=1 oly used first pricipal compoet Carlos Guestri

14 Eigefaces [Turk, Petlad 91] Iput images: Pricipal compoets: Carlos Guestri

15 Eigefaces recostructio Each image correspods to addig 8 pricipal compoets: Carlos Guestri

16 Scalig up Covariace matrix ca be really big! S is D by D Say, oly features fidig eigevectors is very slow Use sigular value decompositio (SVD) fids to K eigevectors great implemetatios available, e.g., scipy.lialg.svd Carlos Guestri

17 SVD Write X = W S V T X data matrix, oe row per datapoit W weight matrix, oe row per datapoit coordiate of x i i eigespace S sigular value matrix, diagoal matrix i our settig each etry is eigevalue l j V T sigular vector matrix i our settig each row is eigevector v j Carlos Guestri

18 PCA usig SVD algoritm Start from m by data matrix X Receter: subtract mea from each row of X X c X X Call SVD algorithm o X c ask for k sigular vectors Pricipal compoets: k sigular vectors with highest sigular values (rows of V T ) Coefficiets become: Carlos Guestri

19 What you eed to kow Dimesioality reductio why ad whe it s importat Simple feature selectio Pricipal compoet aalysis miimizig recostructio error relatioship to covariace matrix ad eigevectors usig SVD Carlos Guestri