Linear Algebra Part I - Linear Spaces Simon Julier Department of Computer Science, UCL S.Julier@cs.ucl.ac.uk http://moodle.ucl.ac.uk/course/view.php?id=11547 GV01 - Mathematical Methods, Algorithms and Implementations: Linear Algebra Part I - Linear Spaces
Structure Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 2
Introduction Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 3
What is Linear Algebra? 4
What is Linear Algebra? Scaling and adding things together! The tools and techniques are central for many of the modules you are going to encounter 5
Linear Algebra in Image Processing An image that we look at is given in its "standard basis Find new basis where different features (shapes, edges, noise) are represented as basis vectors Do signal processing Turn back to the original basis 6
Linear Algebra in Computer Graphics Linear algebra is frequently used to construct scene graphs: The geometry of a scene is composed of many parts connected together in rigid or non-rigid ways The algebra cascades the transformations through the scene graph 7
Example Application of Vector Spaces One problem with processing eye witness reports is getting an accurate representation of a perpetrator Verbal descriptions are ambiguous and inaccurate Identikits are often used to mock up faces However, can be slow and difficult to use 8
EigenFIT System 9
Interactive Genetic Algorithm One solution is to generate novel sequences of face images and let people home in on the correct face 10
GUI and User Selection of Fit However, where do these faces come from? 11
Generating the Faces Faces are represented using a vector-space representation and are sampled A large set of training data of faces is collected 150 standard landmarks identified All the images warped so that these landmarks line up in each image A principal component analysis (PCA) is carried out This is something we ll touch on with eigendecompositions, but it basically means find the most important vectors 12
Generating the Faces Faces are generated using the expressions Geometry (shape) Texture (appearance) Mean values Weighted vectors 13
Results 14
Linear Algebra in Computer Vision Most computer vision (and graphics) assumes pinhole camera models Situation is described by projective geometry, which is generalised from affine geometry 15
Coordinate-Free Approach Where possible, these lectures will emphasise a coordinate free approach to linear algebra The reason for using this abstraction is that it emphasises that we are dealing with geometric objects Just looking at the mechanics (coordinates only) can be misleading: Points are not vectors, but they are often represented the same way (e.g., array of 3 doubles) Doesn t easily generalise to more than 2 or 3 dimensions 16
Coordinate-Free Concepts in Java3D 17
Roadmap of the Material General linear algebra concepts Tuples Vector Space Relative displacement, subspaces, basis Specific geometric applications Affine Space Points and geometric objects Euclidean Space Angles, distances, areas, volumes Cartesian Frame The standard (x,y,z) coordinate system 18
Structure Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 19
Tuples Much of linear algebra is about manipulating tuples A tuple is an ordered list of elements Ordered list -> the position matters Element -> what sits in each position (can be another tuple) Vectors and points are special examples of tuples which we ll work with extensively 20
Tuples Describe Directed Graphs A directed graph consists of a set of vertices and edges 21
Tuples Describe Colour Additive (RGB) Subtractive (CMYK) There are lots of ways to parameterise colour depending on the application, physics, etc. Most use 3 or 4 real numbers HSV YUV 22
Arithmetic Operations on Tuples Addition (when defined): Subtraction (when defined): Scalar multiplication and division (when defined): 23
Multiplication of Tuples The only general product between tuples is the Cartesian product The number of elements in each tuple does not have to be the same The type of elements in each tuple does not have to be the same 24
Tuple Mini-Quiz Consider the following set of tuples: The suits: The faces: The numbers: 25
Tuple Mini-Quiz Compute the following: 1. i+j 2. 17i-3j 3. s c 4. i s c 26
Real-Valued Tuples We shall mostly be concerned with real-valued tuples of the same length Therefore, we can also define the inner product 27
Properties of Tuple Products Commutative: Associative: Distributive: 28
Summary Tuples are the basic building blocks for linear algebra They are ordered lists of elements Defined operations include: Addition (of the same type when it s sensible) Scaling (when it s sensible) Multiplication (can mix tuples of different types) For most of this module, we are concerned with realvalued tuples and so we can use the inner product 29
Vectors and Vector Spaces Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 30
Vectors The most fundamental element in linear algebra is a vector Vectors are special types of tuples which satisfy various types of scaling and addition operations We shall meet several kinds in this course: Coordinate vector spaces Functional vector spaces (perhaps) Probabilistic vector spaces (perhaps) Vectors actually live in a vector space, but we can t talk about this until we ve look a bit more at the properties of vectors 31
Geometric Interpretation of Vectors In geometry, a vector encodes relative displacement Length Direction It does not encode absolute position 32
Equivalency of Vectors Equivalent vectors are parallel and of the same length Two vectors are the same iff (if and only if): They have the same direction They have the same length 33
Scaling Vectors by a Positive Constant Length changed Direction unchanged 34
Scaling Vectors by a Negative Constant Length changed Direction reversed 35
Addition and Subtraction of Vectors Summation or differencing of vectors closes the triangle 36
Other Properties of Vectors Commutativity Associativity Distributivity of addition over multiplication Distributivity of multiplication over addition 37
Vector Spaces A vector space has the following properties: Addition and subtraction is defined, and the result is another vector The set is closed under linear combinations: Given and the combination There is a zero vector such that 38
Vector and Vector Space Mini-Quiz Which of the following are vector spaces? 1. A tuple of n real numbers: 2. The zero-vector: 3. The coefficients of quadratic polynomials: 4. The angle displayed on a compass: 39
Summary Vectors encode relative displacement, not absolute position Scaling a vector changes its length, but not its direction All vectors live in a vector space The vector space is closed under linear combinations and possesses a zero vector 40
Subspaces, Basis, Dimensions and Coordinates The previous part showed that we can add vectors However, we can look at the structure of vector spaces in greater detail The means for doing this is, surprisingly enough, to consider a way of representing a vector as a set of coordinates with respect to a spanning set 41
Coordinate Representation of a Vector Consider the following problem I am given a spanning set of vectors, I am given a vector I form the linear combination of vectors in S 42
Coordinate Representation of a Vector What properties must be obeyed by S if I can always find a unique set of coefficients so that 43
Coordinate Representation of a Vector From the closure property, we know that any linear combination must automatically obey the property Therefore, we need to prove two things: The coefficients are unique Any vector in can be written in this form 44
Arbitrary S Can Be Ambiguous 45
2D Vector Example Here, Therefore, 46
Polynomial Example Consider the vector space for the coefficients of quadratics, If our spanning set is we compute the polynomial 47
Linear Independence The number of vectors we can put into the spanning set S is unlimited However, if some of the vectors can be written in terms of other vectors, this leads to redundancy The same vector can be written in an infinite number of ways using the spanning set The way to eliminate this redundancy is to use vectors which are linearly independent 48
Linear Independence A set is linearly independent if only for the special case In other words, we can t write any member of the set as a linear combination of the other members of the set 49
Linearly Independent Vectors 50
Arbitrary S Might Not Fill the Space The spanning set of vectors might be independent, but might not point in all the directions in 51
Polynomial Example For our polynomial, Suppose our spanning set is This is linearly independent, but we can only construct quadratics of the form 52
Subspaces We formalise this by saying that given the spanning set the linear combination comprises the subspace 53
Dimension of a Subspace The dimension n of is the maximum number of linearly independent vectors in If the dimension of is less than that of then we are dealing with a subspace The coursework will ask you to prove a method which can be used to compute the number of dimensions from a spanning set 54
Basis of a Subspace Since the spanning set lies in the subspace, we can also say If the vectors in the spanning set are linearly independent, then S is a basis for In other words, any vector which lies in can be written using S and a unique set of coefficients 55
Subspaces and Basis The first two vectors form a basis for the subspace S All three vectors form a basis for the (sub)space V 56
Vector Subspaces Mini-Quiz Consider the set of linearly independent vectors Are these spanning sets dependent or independent? What s the dimension and basis of the subspace? 57
The Question Recall the question for any vector and where can we find a unique set of coefficients such that 58
The Answer For S a basis of, we can always represent the vector uniquely and exactly For S a basis of just a subspace, we can only represent some of the vector exactly Expressing the closest approximation of the vector in the subspace is a kind of projection operation, which we ll get back to later 59
The Answer 60
Summary Vector spaces can be decomposed into a set of subspaces Each subspace is a vector space in its own right (closure; zero vector) The dimension of a subspace is the maximum number of linearly independent vectors which can be constructed within that subspace 61
Summary A spanning set is an arbitrary set of vectors which comprise a subspace If the spanning set is linearly independent, it s also known as a basis for that subspace The coordinate representation of a vector in a subspace is unique with respect to a basis for that subspace 62
Changing Basis In the last few slides we said we could write the coordinates of a vector uniquely given a basis set However, for a given subspace the choice of a basis is not unique For some classes of problems, we can make the problem significantly easier by changing the basis to reparameterise the problem 63
Example of Changing Basis Earlier we considered the vector space of quadratics, One basis could be However, another basis is 64
Example of Changing Basis One way to represent position on the Earth is to use Earth Centered Earth Fixed (ECEF) Coordinates Locally at each point on the globe, however, it s more convenient to use East- North-Up (ENU) coordinates If we move to a new location, the ENU basis has to change in the ECEF frame Local Tangent Plane 65
Changing Basis Suppose we would like to change our basis set from to where both basis span the same vector space 66
Changing Basis 67
Changing Basis Since the subspaces are the same, each vector in the original basis can be written as a linear combination of the vectors from the new basis, 68
Changing Basis Clanking through the algebra, we can repeat this for all the other vectors in the original basis, 69
Changing Basis Now consider the representation of our vector in the original basis, Substituting for just the first coefficient, we get 70
Changing Basis Substituting for all the other coefficients gives 71
Changing Basis Mini-Quiz Earlier we considered the vector space of quadratics, Given a set of coordinates in the basis Compute the coordinates in the new basis 72
Summary of Changing Basis Sometimes changing a basis can make a problem easier We can carry this out if our original basis is a subspace of our new vector space The mapping is unique, and corresponds to writing the old basis in terms of the new basis (It s much neater to do it with matrix multiplication) 73
So What s the Problem with Vector Spaces? We have talked about vector spaces They encode displacement There are rules for adding vectors together and scaling them We can define subspaces, dimensions and sets of basis vectors We can even change our basis However vector spaces leave a lot out! 74
The Bits Which Are Missing There are no points There is no way to represent actual geometric objects There is no formal definition of what things like angles, or lengths mean Therefore, we can t consider issues like orthogonality We haven t discussed the idea of an origin Everything floats in free space Affine spaces start to redress this by throwing points into the mix 75
Affine Spaces Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 76
Affine Spaces An affine space extends a vector space by adding a set of points to it The vectors are referenced by The points are referenced by The dimension of the vector space is the dimension of 77
Geometric Interpretation of Points In geometry a point encodes a zero-dimensional place It contains no information about length, area, volume, etc. (This is still not absolute because we have no origin) 78
Important Axioms The relationship between points and vectors are laid out using two important axioms: The Coordinate Axiom (defines multiplicative operations on points) The Head-to-Tail Axiom (codifies point subtraction and how points and vectors fit together) 79
The Coordinate Axiom Multiplying a point by 1 returns the original point, Multiplying a point by 0 returns the zero vector Multiplying a point by any other scalar is not allowed 80
Head-to-Tail Axiom: Part I 81
Head-to-Tail Axiom: Part II 82
Head-to-Tail Axiom: Part III 83
Head-to-Tail Axiom Mini-Quiz Prove the following identities: 84
Representing Lines Using Affine Combinations Now that we have points, we can represent objects as an (infinite) set of those points For example, a line of finite length consists of all the points which lie between the start point Q and the end point P 85
Representing Lines Using Affine Combinations The vector oriented along the line is A point R lying on the line can be written as Useful abuse of notation! 86
Representing Planes Using Affine Combinations The spanning set for the plane is computed from three points O, P,Q which lie in the plane, Any point R on the plane can be written as 87
General Affine Combinations In general, the affine combination is written as The combination must obey the constraint The reason for this restriction will become apparent later 88
Affine Combination Mini-Quiz Which of these are valid affine combinations? 89
General Affine Combination Mini-Quiz Consider three points O, P, Q and the affine combination What 2D shape is described by this combination when 90
Affine Frames So far we have not had to resort to any notion of an origin However, in many cases origins are useful An affine frame for an affine space is formally defined to consist of a basis and an origin, where and are a basis for 91
Illustration of a 3D Affine Frame The basis is linearly independent 92
Vectors and Points in Affine Frames Given an affine frame, we can define points and vectors accordingly, The arithmetic operations from the head-to-tail axiom can be carried out directly in the affine frame 93
Frame-Based Addition of Vectors Vectors add in a completely straightforward manner Given two vectors, Then 94
Frame-Based Scalar Multiplication of Vectors Scaling a vector by a scalar scales the coordinates of the vector This changes the length but not the direction of the vector 95
Frame-Based Point and Vector Arithmetic Adding a point to a vector gives a new point The position is the sum of the point position vector and the additive vector 96
Frame-Based Point Arithmetic Subtracting points leads, as we expect, to a vector 97
Frame-Based Point and Vectors Mini-Quiz So What s wrong with calculating P+Q? 98
Frame-Based Affine Combinations The coordinate axiom also explains the seemingly arbitrary constraint on affine combinations that Recall that an affine combination is In turn, each point can be written as 99
Frame-Based Affine Combinations Substituting into the expression for the affine combination, This is only a valid point if the coefficient on is 1, which can only happen if the coefficients sum to 1 100
Barycentric Coordinates So far we ve treated the coordinate representation of points and vectors a bit differently The relationship between affine combinations and is no coincidence It is possible to express both using barycentric coordinates 101
Barycentric Coordinates We consider each point used in our affine combination as being used to form a spanning set, 102
Barycentric Coordinates A point is simply an affine combination of the basis points or the n-dimensional simplex, where if P is a valid point then 103
Barycentric Coordinates in Action 104
Vectors in Barycentric Coordinates Now, a vector can be written as but we d like to similarly express it in barycentric coordinates in the form 105
Vectors in Barycentric Coordinates Substituting the expressions for the points in terms of the vectors, What we want Additional point bits 106
Vectors in Barycentric Coordinates We eliminate the point bits by setting the constraint In other words, vectors are represented as barycentric coordinates whose values sum to zero 107
Barycentric Coordinates Mini-Quiz How do you think affine subspaces can be represented using barycentric coordinates? How do you think affine independence can be represented using barycentric coordinates? 108
Summary of Barycentric Coordinates We can use sets of points to define a basis For an n-dimensional space we use n+1 points known as an n-dimensional simplex In this basis, points are represented by coordinates that sum to 1 Vectors are represented by coordinates that sum to 0 Affine subspaces correspond to lower dimensional simplexes Affine independence happens when no point is an affine combination of the other points 109
So What s the Problem with Affine Spaces? We ve talked about affine spaces We now have points as well as vectors We have discussed the algebra which links points and vectors together We can describe geometric objects using affine combinations and barycentric coordinates However, a few useful things are still missing! 110
More Missing Bits We still don t have a notion of distances Just ratios on lines between points We still don t have a notion of angles We still don t have an absolute origin These are all introduced in Euclidean geometry 111
Euclidean Spaces Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 112
Euclidean Spaces Euclidean spaces extend affine spaces by adding notions of length and angle The Euclidean structure is determined by the forms of the equations used to calculate these quantities We are going to just give these without proof However, we can motivate the problem by considering the problem of orthogonally projecting one vector onto another vector First, though, we need some additional vector notation 113
Some More Vector Notation Since we are going to define lengths and directions, we can now decompose a vector into the product of its length and an orthonormal vector which defines its direction Length (+ve) Orthonormal vector (length=1) 114
Projecting Vectors onto Vectors Consider two vectors and that occupy the same affine space What s the orthogonal projection of onto? 115
The Answer Projected vector 116
The Answer Projected vector Projected vector 117
Computing the Answer We need to compute both the direction and length of the projection vector The direction of the vector must be parallel to Therefore, we are going to define the orthogonal projection as Scale factor Orthonormal vector parallel to the right direction 118
Length of the Scaled Projection From basic trigonometry, the length of the parallel projection is From our assumed solution the length must be The dodgy bit! 119
Special Case of the Scalar Product The analysis from the past few slides shows that Furthermore, cosθ gives us the right sign behaviour Therefore, 120
General Case of the Scalar Product If we now let both of our vectors be non-normalised, then The scalar product is a special case of an inner product (we might meet these again later) 121
Lengths, Angles and Distances Lengths and angles are (circularly) defined as The distance function (or metric) between two points is the length of the vector between them, 122
Properties of Scalar Products Bilinearity: Positive definiteness: 123
Properties of Scalar Products Commutativity: Distributivity of the dot product over vector addition, Distributivity of vector addition over the dot product, 124
Scalar Product and Projection Mini-Quiz What s the value of 125
Summary The Euclidean structure is defined by the scalar product The scalar product is used to compute the orthogonal projection of one vector onto another vector The scalar product works in any number of dimensions It has many useful properties including bilinearity However, it only defines a one dimensional quantity (length) Vector products generalise this 126
Vector Products The vector product is the familiar cross product and is defined to be where is orthogonal to and The vector product is a special case of an exterior product and is a pseudovector (we won t meet these again) The vector product is only defined in 3D 127
Magnitude of the Vector Product The magnitude of the vector product is the area of the parallelogram having the vectors as sides 128
Direction of the Vector Product In general, in 3D we have 3 axes of rotation However, if we rotate and such that they still lie in the same plane, still points in the same direction Therefore, encodes information about two axes of rotation 129
Sign of the Vector Product Given the plane defined by the vectors, there are two possible choices of the normal vector out of or into the plane The choice is determined by the sign of the angle between the vectors Anticlockwise is positive 130
Right Hand Rule 131
Properties of Vector Products Anticommutativity: Distributivity: Distributivity: Parallelism: 132
Scalar Triple Product and Volumes We have shown that: Scalar products define lengths and angles Vector products define areas and a sense of perpendicularity Therefore, is it possible to extend this, at least in 3D, to the notion of a volume? The scalar triple product computes the volume of the parallelepiped defined by three linearly independent vectors 133
Scalar Triple Product and Volumes 134
Volume of the Parallelepiped The volume is computed as follows This is the absolute value of the scalar triple product 135
Scalar Triple Product The term inside the absolute value is the pseudoscalar which is known as the scalar triple product Pseudo-scalars reverse their sign under inversion, Fortunately we don t need to look at them any further! 136
Cartesian Frames So far we ve said an affine frame is simply an origin point and a set of linearly independent vectors Now that we can talk about angles and lengths, we can define a Cartesian coordinate frame A Cartesian Frame has the property that its basis vectors are orthonormal 137
Orthonormal Basis Set An orthonormal basis set has the property that its vectors: Are orthogonal to one another Are of unit length More compactly, 138
Illustration of the Cartesian Basis Set The basis is orthonormal 139
Cartesian Frames in 3D (Finally!) The basis set in 3D is written using the familiar vectors, Any vector can be written as 140
Scalar and Vector Product in the Cartesian Frame The scalar product is given by The vector product is given by 141
Euclidean Space Summary Euclidean spaces add notions of length and angles through the scalar product These can be extended by the vector and vector triple products to give areas and volumes in 3D The Cartesian Coordinate frame is a special case with a dedicated origin and an orthonormal basis set 142