Fundamentals of Linear Algebra, Part II

Transcription

1 -7/8-797 Machine Learning for Signal Processing Fundamentals of Linear Algebra, Part II Class 3 August 9 Instructor: Bhiksha Raj Administrivia Registration: Anone on waitlist still? We have a second TA Sohail Bahmani sbahmani@andrewcmuedu Homework: Slightl delaed Linear algebra Adding some fun new problems Use the discussion lists on blackboardandrewcmuedu Blackboard if ou are not registered on blackboard please register Overview Vectors and matrices Basic vector/matrix operations Vector products Matrix products Various matrix tpes Matrix inversion Matrix interpretation Eigenanalsis Singular value decomposition The Identit Matrix Y An identit matrix is a square matrix where All diagonal elements are All off-diagonal elements are Multiplication b an identit matrix does not change vectors 3 4 Diagonal Matrix Diagonal matrix to transform images Y All off-diagonal elements are zero Diagonal elements are non-zero Scales the axes Ma flip axes How? 6

2 Stretching Location-based representation 6-7/8-797 representation Scaling matrix onl scales the X axis The Y axis and pixel value are scaled b identit ot a good wa of scaling 3 Aug 7 Stretching D = -7/8-797 Better wa ) x ( ewpic DA A 3 Aug 8 Modifing color B G R P -7/8-797 Scale onl Green P ewpic 3 Aug 9 Permutation Matrix x z z x 3 4 (3,4,) X Y Z 4 3 X (old Y) Y (old Z) Z (old X) -7/8-797 A permutation matrix simpl rearranges the axes The row entries are axis vectors in a different order The result is a combination of rotations and reflections The permutation matrix effectivel permutes the arrangement of the elements in a vector ( ) 3 Aug Permutation Matrix P P 6-7/8-797 Reflections and 9 degree rotations of images and objects 6 3 Aug Permutation Matrix P P -7/8-797 Reflections and 9 degree rotations of images and objects Object represented as a matrix of 3-Dimensional position vectors Positions identif each point on the surface z z z x x x 3 Aug

3 Rotation Matrix x' x cos sin ' xsin cos (x,) Y R cos sin sin cos x X x' X new ' Y R X X new (x, ) (x,) Rotating a picture cos 4 R sin 4 sin cos X X x x A rotation matrix rotates the vector b some angle Alternatel viewed, it rotates the axes The new axes are at an angle to the old one 3 ote the representation: 3-row matrix Rotation onl applies on the coordinate rows The value does not change Wh is pacman grain? 4 3-D Rotation Xnew Ynew Y Znew Z X degrees of freedom separate angles What will the rotation matrix be? Projections What would we see if the cone to the left were transparent if we looked at it along the normal to the plane The plane goes through the origin Answer: the figure to the right How do we get this? Projection 6 Projections Projections 9degrees W W projection Each pixel in the cone to the left is mapped onto to its shadow on the plane in the figure to the right The location of the pixel s shadow is obtained b multipling the vector V representing the pixel s location in the first figure b a matrix A Shadow (V )= A V The matrix A is a projection matrix 7 Consider an plane specified b a set of vectors W, W Or matrix [W W ] An vector can be projected onto this plane b multipling it with the projection matrix for the plane The projection is the shadow 8 3

4 Projection Matrix 9degrees Projections W W projection Given a set of vectors W, W, which form a matrix [W W ] The projection matrix that transforms an vector X to its projection on the plane is P = W (W T W) - W T We will visit matrix inversion shortl Magic an set of vectors from the same plane that are expressed as a matrix will give ou the same projection matrix P = V (V T V) - V T 9 HOW? Projections Projections Draw an two vectors W and W that lie on the plane AY two so long as the have different angles Compose a matrix [W W] Compose the projection matrix P = W (W T W) - W T Multipl ever point on the cone b P to get its projection View it I m missing a step here what is it? The projection actuall projects it onto the plane, but ou re still seeing the plane in 3D The result of the projection is a 3-D vector P = W (W T W) - W T = 3x3, P*Vector = 3x The image must be rotated till the plane is in the plane of the paper The Z axis in this case will alwas be zero and can be ignored How will ou rotate it? (remember ou know W and W) Projection matrix properties Projections: A more phsical meaning Let W, W W k be bases We want to explain our data in terms of these bases We often cannot do so But we can explain a significant portion of it The projection of an vector that is alread on the plane is the vector itself Px = x if x is on the plane If the object is alread on the plane, there is no further projection to be performed The projection of a projection is the projection P (Px) = Px That is because Px is alread on the plane Projection matrices are idempotent P = P 3 Aug Follows from the above -7/ The portion of the data that can be expressed in terms of our vectors W, W, W k, is the projection of the data on the W W k (hper) plane In our previous example, the data were all the points on a cone The interpretation for volumetric data is obvious 4 4

5 Projection : an example with sounds Projection: one note The spectrogram (matrix) of a piece of music The spectrogram (matrix) of a piece of music How much of the above music was composed of the above notes Ie how much can it be explained b the notes spectrogram; note P = W (W T W) - W T Projected Spectrogram = P * M 6 Projection: one note cleaned up Projection: multiple notes The spectrogram (matrix) of a piece of music The spectrogram (matrix) of a piece of music Floored all matrix values below a threshold to zero P = W (W T W) - W T Projected Spectrogram = P * M 7 8 Projection: multiple notes, cleaned up The spectrogram (matrix) of a piece of music P = W (W T W) - W T Projected Spectrogram = P * M Projection and Least Squares Projection actuall computes a least squared error estimate For each vector V in the music spectrogram matrix Approximation: V approx = a*note + b*note + c*note3 a V approx b c Error vector E = V V approx Squared error energ for V e(v) = norm(e) Total error = sum_over_all_v { e(v) } = V e(v) Projection computes V approx for all vectors such that Total error is minimized It does not give ou a, b, c Though note note note3 That needs a different operation the inverse / pseudo inverse 9 3

6 Orthogonal and Orthonormal matrices Orthogonal Matrix : AA T = diagonal Each row vector lies exactl along the normal to the plane specified b the rest of the vectors in the matrix Orthonormal Matrix: AA T = A T A = I In additional to be orthogonal, each vector has length exactl = Interesting observation: In a square matrix if the length of the row vectors is, the length of the column vectors is also 3 Orthogonal and Orthonormal Matrices Orthonormal matrices will retain the relative angles between transformed vectors Essentiall, the are combinations of rotations, reflections and permutations Rotation matrices and permutation matrices are all orthonormal matrices The vectors in an orthonormal matrix are at 9degrees to one another Orthogonal matrices are like Orthonormal matrices with stretching The product of a diagonal matrix and an orthonormal matrix 3 Matrix Rank and Rank-Deficient Matrices Matrix Rank and Rank-Deficient Matrices P * Cone = Rank = Rank = Some matrices will eliminate one or more dimensions during transformation These are rank deficient matrices The rank of the matrix is the dimensionalit of the trasnsformed version of a full-dimensional object 33 Some matrices will eliminate one or more dimensions during transformation These are rank deficient matrices The rank of the matrix is the dimensionalit of the transformed version of a full-dimensional object 34 Projections are often examples of rank-deficient transforms on-square Matrices P = W (W T W) - W T ; Projected Spectrogram = P * M The original spectrogram can never be recovered P is rank deficient P explains all vectors in the new spectrogram as a mixture of onl the 4 vectors in W There are onl 4 independent bases Rank of P is x x x x x 9 6 z z on-square matrices add or subtract axes More rows than columns add axes But does not increase the dimensionalit of the data Fewer rows than columns reduce axes Ma reduce dimensionalit of the data x z X = D data P = transform PX = 3D, rank 36 6

7 on-square Matrices The Rank of a Matrix x x x z z z X = 3D data, rank 3 3 x x on-square matrices add or subtract axes More rows than columns add axes But does not increase the dimensionalit of the data Fewer rows than columns reduce axes Ma reduce dimensionalit of the data x P = transform PX = D, rank The matrix rank is the dimensionalit of the transformation of a fulldimensioned object in the original space The matrix can never increase dimensions Cannot convert a circle to a sphere or a line to a circle The rank of a matrix can never be greater than the lower of its two dimensions 38 The Rank of Matrix Matrix rank is unchanged b transposition Projected Spectrogram = P * M Ever vector in it is a combination of onl 4 bases The rank of the matrix is the smallest no of bases required to describe the output Eg if note no 4 in P could be expressed as a combination of notes, and 3, it provides no additional information Eliminating note no 4 would give us the same projection The rank of P would be 3! If an -D object is compressed to a K-D object b a matrix, it will also be compressed to a K-D object b the transpose of the matrix 4 Matrix Determinant (r) (r+r) Matrix Determinant: Another Perspective Volume = V Volume = V (r) (r) (r) The determinant is the volume of a matrix Actuall the volume of a parallelepiped formed from its row vectors Also the volume of the parallelepiped formed from its column vectors Standard formula for determinant: in text book The determinant is the ratio of -volumes If V is the volume of an -dimensional object O in - dimensional space O is the complete set of points or vertices that specif the object If V is the volume of the -dimensional object specified b A*O, where A is a matrix that transforms the space A = V / V 4 4 7

8 Matrix Determinants Matrix determinants are onl defined for square matrices The characterize volumes in linearl transformed space of the same dimensionalit as the vectors Rank deficient matrices have determinant Since the compress full-volumed -D objects into zero-volume -D objects Eg a 3-D sphere into a -D ellipse: The ellipse has volume (although it does have area) Conversel, all matrices of determinant are rank deficient Since the compress full-volumed -D objects into zero-volume objects Multiplication properties Properties of vector/matrix products Associative A (BC) (A B) C Distributive A (B C) A B A CC OT commutative!!! A B BA left multiplications right multiplications Transposition T T T A B B A Determinant properties Associative for square matrices Scaling volume sequentiall b several matrices is equal to scaling once b the product of the matrices Volume of sum!= sum of Volumes A B C A B C ( B C) B C The volume of the parallelepiped formed b row vectors of the sum of two matrices is not the sum of the volumes of the parallelepipeds formed b the original matrices Commutative for square matrices!!! A B B A A B The order in which ou scale the volume of an object is irrelevant Matrix Inversion A matrix transforms an - D object to a different - D object What transforms the new object back to the original? The inverse transformation The inverse transformation is called the matrix inverse 8 T ??? Q??? T??? 4 46 Matrix Inversion Inverting rank-deficient matrices T T - T - T = I The product of a matrix and its inverse is the identit matrix Transforming an object, and then inverse transforming it gives us back the original object Rank deficient matrices flatten objects In the process, multiple points in the original object get mapped to the same point in the transformed object It is not possible to go back from the flattened object to the original object Because of the man-to-one forward mapping Rank deficient matrices have no inverse

9 Revisiting Projections and Least Squares Projection computes a least squared error estimate For each vector V in the music spectrogram matrix Approximation: V approx = a*note + b*note + c*note3 W note note note3 Error vector E = V V approx V approx a W b c Squared error energ for V e(v) = norm(e) Total error = Total error + e(v) Projection computes V approx for all vectors such that Total error is minimized But WHAT ARE a b and c? The Pseudo Inverse (PIV) V approx a W b c a V W b c a b PIV( W)* V c We are approximating spectral vectors V as the transformation of the vector [a b c] T ote we re viewing the collection of bases in W as a transformation The solution is obtained using the pseudo inverse This give us a LEAST SQUARES solution If W were square and invertible Pinv(W) = W -, and V=V approx 49 Explaining music with one note Explanation with multiple notes X =PIV(W)*M X=PIV(W)*M Recap: P = W (W T W) - W T, Projected Spectrogram = P*M Approximation: M W*X The amount of W in each vector = X = PIV(W)*M W*Pinv(W)* Projected Spectrogram = P*M W*Pinv(W) = Projection matrix = W (W T W) - W PIV(W) = (W T W) - W T X = Pinv(W) * M; Projected matrix = W*X = W*Pinv(W)*M How about the other wa? Pseudo-inverse (PIV) V = Pinv() applies to non-square matrices Pinv ( Pinv (A))) = A A*Pinv(A)= projection matrix! Projection onto the columns of A? U =? If A = K x matrix and K >, A projects -D vectors into a higher-dimensional K-D space Pinv(A)*A = I in this case WV \approx M M * Pinv(V) U = WV 3 4 9

10 Matrix inversion (division) The inverse of matrix multiplication ot element-wise division!! Provides a wa to undo a linear transformation Inverse of the unit matrix is itself Inverse of a diagonal is diagonal Inverse of a rotation is a (counter)rotation (its transpose!) Inverse of a rank deficient i matrix does not exist! But pseudoinverse exists Pa attention to multiplication side! A B C, A C B, B A C Matrix inverses defined for square matrices onl If matrix not square use a matrix pseudoinverse: A B C, A C B, B A C MATLAB sntax: inv(a), pinv(a) What is the Matrix? Dualit in terms of the matrix identit Can be a container of data An image, a set of vectors, a table, etc Can be a linear transformation A process b which to transform data in another matrix We ll usuall start with the first definition and then appl the second one on it Ver frequent operation Room reverberations, mirror reflections, etc Most of signal processing and machine learning are matrix operations! 6 Eigenanalsis If something can go through a process mostl unscathed in character it is an eigen-something Sound example: A vector that can undergo a matrix multiplication and keep pointing the same wa is an eigenvector Its length can change though How much its length changes is expressed b its corresponding eigenvalue Each eigenvector of a matrix has its eigenvalue Finding these eigenthings is called eigenanalsis 7 EigenVectors and EigenValues Black vectors are eigen vectors 7 M 7 Vectors that do not change angle upon transformation The ma change length MV V V = eigen vector = eigen value Matlab: [V, L] = eig(m) L is a diagonal matrix whose entries are the eigen values V is a maxtrix whose columns are the eigen vectors 8 Eigen vector example Matrix multiplication revisited 7 M Matrix transformation transforms the space Warps the paper so that the normals to the two vectors now lie along the axes 9 6

11 A stretching operation 4 8 A stretching operation Draw two lines Stretch / shrink the paper along these lines b factors and The factors could be negative implies flipping the paper The result is a transformation of the space Draw two lines Stretch / shrink the paper along these lines b factors and The factors could be negative implies flipping the paper The result is a transformation of the space 6 6 Phsical interpretation of eigen vector The result of the stretching is exactl the same as transformation b a matrix The axes of stretching/shrinking are the eigenvectors The degree of stretching/shrinking are the corresponding eigenvalues The EigenVectors and EigenValues conve all the information about the matrix 63 Phsical interpretation of eigen vector V V V L M VLV The result of the stretching is exactl the same as transformation b a matrix The axes of stretching/shrinking are the eigenvectors The degree of stretching/shrinking are the corresponding eigenvalues The EigenVectors and EigenValues conve all the information about the matrix 64 Eigen Analsis Smmetric Matrices ot all square matrices have nice eigen values and vectors Eg consider a rotation matrix 7 7 cos sin R sin cos x X x' X new ' This rotates ever vector in the plane o vector that remains unchanged In these cases the Eigen vectors and values are complex Some matrices are special however 6 Matrices that do not change on transposition Row and column vectors are identical Smmetric matrix: Eigen vectors and Eigen values are alwas real Eigen vectors are alwas orthogonal At 9 degrees to one another 66

12 Smmetric Matrices 7 7 Eigen vectors point in the direction of the major and minor axes of the ellipsoid resulting from the transformation of a spheroid The eigen values are the lengths of the axes Smmetric matrices Eigen vectors V i are orthonormal V it V i = V it V j =, i!= j Listing all eigen vectors in matrix form V V T = V - V T V = I V V T = I M V i = V i In matrix form : M V = V L L is a diagonal matrix with all eigen values V L V T 67 68