Linear Algebra Part I - Linear Spaces

Transcription

1 Linear Algebra Part I - Linear Spaces Simon Julier Department of Computer Science, UCL S.Julier@cs.ucl.ac.uk GV01 - Mathematical Methods, Algorithms and Implementations: Linear Algebra Part I - Linear Spaces

2 Structure Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 2

3 Introduction Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 3

4 What is Linear Algebra? 4

5 What is Linear Algebra? Scaling and adding things together! The tools and techniques are central for many of the modules you are going to encounter 5

6 Linear Algebra in Image Processing An image that we look at is given in its "standard basis Find new basis where different features (shapes, edges, noise) are represented as basis vectors Do signal processing Turn back to the original basis 6

7 Linear Algebra in Computer Graphics Linear algebra is frequently used to construct scene graphs: The geometry of a scene is composed of many parts connected together in rigid or non-rigid ways The algebra cascades the transformations through the scene graph 7

8 Example Application of Vector Spaces One problem with processing eye witness reports is getting an accurate representation of a perpetrator Verbal descriptions are ambiguous and inaccurate Identikits are often used to mock up faces However, can be slow and difficult to use 8

9 EigenFIT System 9

10 Interactive Genetic Algorithm One solution is to generate novel sequences of face images and let people home in on the correct face 10

11 GUI and User Selection of Fit However, where do these faces come from? 11

12 Generating the Faces Faces are represented using a vector-space representation and are sampled A large set of training data of faces is collected 150 standard landmarks identified All the images warped so that these landmarks line up in each image A principal component analysis (PCA) is carried out This is something we ll touch on with eigendecompositions, but it basically means find the most important vectors 12

13 Generating the Faces Faces are generated using the expressions Geometry (shape) Texture (appearance) Mean values Weighted vectors 13

14 Results 14

15 Linear Algebra in Computer Vision Most computer vision (and graphics) assumes pinhole camera models Situation is described by projective geometry, which is generalised from affine geometry 15

16 Coordinate-Free Approach Where possible, these lectures will emphasise a coordinate free approach to linear algebra The reason for using this abstraction is that it emphasises that we are dealing with geometric objects Just looking at the mechanics (coordinates only) can be misleading: Points are not vectors, but they are often represented the same way (e.g., array of 3 doubles) Doesn t easily generalise to more than 2 or 3 dimensions 16

17 Coordinate-Free Concepts in Java3D 17

18 Roadmap of the Material General linear algebra concepts Tuples Vector Space Relative displacement, subspaces, basis Specific geometric applications Affine Space Points and geometric objects Euclidean Space Angles, distances, areas, volumes Cartesian Frame The standard (x,y,z) coordinate system 18

19 Structure Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 19

20 Tuples Much of linear algebra is about manipulating tuples A tuple is an ordered list of elements Ordered list -> the position matters Element -> what sits in each position (can be another tuple) Vectors and points are special examples of tuples which we ll work with extensively 20

21 Tuples Describe Directed Graphs A directed graph consists of a set of vertices and edges 21

22 Tuples Describe Colour Additive (RGB) Subtractive (CMYK) There are lots of ways to parameterise colour depending on the application, physics, etc. Most use 3 or 4 real numbers HSV YUV 22

23 Arithmetic Operations on Tuples Addition (when defined): Subtraction (when defined): Scalar multiplication and division (when defined): 23

24 Multiplication of Tuples The only general product between tuples is the Cartesian product The number of elements in each tuple does not have to be the same The type of elements in each tuple does not have to be the same 24

25 Tuple Mini-Quiz Consider the following set of tuples: The suits: The faces: The numbers: 25

26 Tuple Mini-Quiz Compute the following: 1. i+j 2. 17i-3j 3. s c 4. i s c 26

27 Real-Valued Tuples We shall mostly be concerned with real-valued tuples of the same length Therefore, we can also define the inner product 27

28 Properties of Tuple Products Commutative: Associative: Distributive: 28

29 Summary Tuples are the basic building blocks for linear algebra They are ordered lists of elements Defined operations include: Addition (of the same type when it s sensible) Scaling (when it s sensible) Multiplication (can mix tuples of different types) For most of this module, we are concerned with realvalued tuples and so we can use the inner product 29

30 Vectors and Vector Spaces Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 30

31 Vectors The most fundamental element in linear algebra is a vector Vectors are special types of tuples which satisfy various types of scaling and addition operations We shall meet several kinds in this course: Coordinate vector spaces Functional vector spaces (perhaps) Probabilistic vector spaces (perhaps) Vectors actually live in a vector space, but we can t talk about this until we ve look a bit more at the properties of vectors 31

32 Geometric Interpretation of Vectors In geometry, a vector encodes relative displacement Length Direction It does not encode absolute position 32

33 Equivalency of Vectors Equivalent vectors are parallel and of the same length Two vectors are the same iff (if and only if): They have the same direction They have the same length 33

34 Scaling Vectors by a Positive Constant Length changed Direction unchanged 34

35 Scaling Vectors by a Negative Constant Length changed Direction reversed 35

36 Addition and Subtraction of Vectors Summation or differencing of vectors closes the triangle 36

37 Other Properties of Vectors Commutativity Associativity Distributivity of addition over multiplication Distributivity of multiplication over addition 37

38 Vector Spaces A vector space has the following properties: Addition and subtraction is defined, and the result is another vector The set is closed under linear combinations: Given and the combination There is a zero vector such that 38

39 Vector and Vector Space Mini-Quiz Which of the following are vector spaces? 1. A tuple of n real numbers: 2. The zero-vector: 3. The coefficients of quadratic polynomials: 4. The angle displayed on a compass: 39

40 Summary Vectors encode relative displacement, not absolute position Scaling a vector changes its length, but not its direction All vectors live in a vector space The vector space is closed under linear combinations and possesses a zero vector 40

41 Subspaces, Basis, Dimensions and Coordinates The previous part showed that we can add vectors However, we can look at the structure of vector spaces in greater detail The means for doing this is, surprisingly enough, to consider a way of representing a vector as a set of coordinates with respect to a spanning set 41

42 Coordinate Representation of a Vector Consider the following problem I am given a spanning set of vectors, I am given a vector I form the linear combination of vectors in S 42

43 Coordinate Representation of a Vector What properties must be obeyed by S if I can always find a unique set of coefficients so that 43

44 Coordinate Representation of a Vector From the closure property, we know that any linear combination must automatically obey the property Therefore, we need to prove two things: The coefficients are unique Any vector in can be written in this form 44

45 Arbitrary S Can Be Ambiguous 45

46 2D Vector Example Here, Therefore, 46

47 Polynomial Example Consider the vector space for the coefficients of quadratics, If our spanning set is we compute the polynomial 47

48 Linear Independence The number of vectors we can put into the spanning set S is unlimited However, if some of the vectors can be written in terms of other vectors, this leads to redundancy The same vector can be written in an infinite number of ways using the spanning set The way to eliminate this redundancy is to use vectors which are linearly independent 48

49 Linear Independence A set is linearly independent if only for the special case In other words, we can t write any member of the set as a linear combination of the other members of the set 49

50 Linearly Independent Vectors 50

51 Arbitrary S Might Not Fill the Space The spanning set of vectors might be independent, but might not point in all the directions in 51

52 Polynomial Example For our polynomial, Suppose our spanning set is This is linearly independent, but we can only construct quadratics of the form 52

53 Subspaces We formalise this by saying that given the spanning set the linear combination comprises the subspace 53

54 Dimension of a Subspace The dimension n of is the maximum number of linearly independent vectors in If the dimension of is less than that of then we are dealing with a subspace The coursework will ask you to prove a method which can be used to compute the number of dimensions from a spanning set 54

55 Basis of a Subspace Since the spanning set lies in the subspace, we can also say If the vectors in the spanning set are linearly independent, then S is a basis for In other words, any vector which lies in can be written using S and a unique set of coefficients 55

56 Subspaces and Basis The first two vectors form a basis for the subspace S All three vectors form a basis for the (sub)space V 56

57 Vector Subspaces Mini-Quiz Consider the set of linearly independent vectors Are these spanning sets dependent or independent? What s the dimension and basis of the subspace? 57

58 The Question Recall the question for any vector and where can we find a unique set of coefficients such that 58

59 The Answer For S a basis of, we can always represent the vector uniquely and exactly For S a basis of just a subspace, we can only represent some of the vector exactly Expressing the closest approximation of the vector in the subspace is a kind of projection operation, which we ll get back to later 59

60 The Answer 60

61 Summary Vector spaces can be decomposed into a set of subspaces Each subspace is a vector space in its own right (closure; zero vector) The dimension of a subspace is the maximum number of linearly independent vectors which can be constructed within that subspace 61

62 Summary A spanning set is an arbitrary set of vectors which comprise a subspace If the spanning set is linearly independent, it s also known as a basis for that subspace The coordinate representation of a vector in a subspace is unique with respect to a basis for that subspace 62

63 Changing Basis In the last few slides we said we could write the coordinates of a vector uniquely given a basis set However, for a given subspace the choice of a basis is not unique For some classes of problems, we can make the problem significantly easier by changing the basis to reparameterise the problem 63

64 Example of Changing Basis Earlier we considered the vector space of quadratics, One basis could be However, another basis is 64

65 Example of Changing Basis One way to represent position on the Earth is to use Earth Centered Earth Fixed (ECEF) Coordinates Locally at each point on the globe, however, it s more convenient to use East- North-Up (ENU) coordinates If we move to a new location, the ENU basis has to change in the ECEF frame Local Tangent Plane 65

66 Changing Basis Suppose we would like to change our basis set from to where both basis span the same vector space 66

67 Changing Basis 67

68 Changing Basis Since the subspaces are the same, each vector in the original basis can be written as a linear combination of the vectors from the new basis, 68

69 Changing Basis Clanking through the algebra, we can repeat this for all the other vectors in the original basis, 69

70 Changing Basis Now consider the representation of our vector in the original basis, Substituting for just the first coefficient, we get 70

71 Changing Basis Substituting for all the other coefficients gives 71

72 Changing Basis Mini-Quiz Earlier we considered the vector space of quadratics, Given a set of coordinates in the basis Compute the coordinates in the new basis 72

73 Summary of Changing Basis Sometimes changing a basis can make a problem easier We can carry this out if our original basis is a subspace of our new vector space The mapping is unique, and corresponds to writing the old basis in terms of the new basis (It s much neater to do it with matrix multiplication) 73

74 So What s the Problem with Vector Spaces? We have talked about vector spaces They encode displacement There are rules for adding vectors together and scaling them We can define subspaces, dimensions and sets of basis vectors We can even change our basis However vector spaces leave a lot out! 74

75 The Bits Which Are Missing There are no points There is no way to represent actual geometric objects There is no formal definition of what things like angles, or lengths mean Therefore, we can t consider issues like orthogonality We haven t discussed the idea of an origin Everything floats in free space Affine spaces start to redress this by throwing points into the mix 75

76 Affine Spaces Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 76

77 Affine Spaces An affine space extends a vector space by adding a set of points to it The vectors are referenced by The points are referenced by The dimension of the vector space is the dimension of 77

78 Geometric Interpretation of Points In geometry a point encodes a zero-dimensional place It contains no information about length, area, volume, etc. (This is still not absolute because we have no origin) 78

79 Important Axioms The relationship between points and vectors are laid out using two important axioms: The Coordinate Axiom (defines multiplicative operations on points) The Head-to-Tail Axiom (codifies point subtraction and how points and vectors fit together) 79

80 The Coordinate Axiom Multiplying a point by 1 returns the original point, Multiplying a point by 0 returns the zero vector Multiplying a point by any other scalar is not allowed 80

81 Head-to-Tail Axiom: Part I 81

82 Head-to-Tail Axiom: Part II 82

83 Head-to-Tail Axiom: Part III 83

84 Head-to-Tail Axiom Mini-Quiz Prove the following identities: 84

85 Representing Lines Using Affine Combinations Now that we have points, we can represent objects as an (infinite) set of those points For example, a line of finite length consists of all the points which lie between the start point Q and the end point P 85

86 Representing Lines Using Affine Combinations The vector oriented along the line is A point R lying on the line can be written as Useful abuse of notation! 86

87 Representing Planes Using Affine Combinations The spanning set for the plane is computed from three points O, P,Q which lie in the plane, Any point R on the plane can be written as 87

88 General Affine Combinations In general, the affine combination is written as The combination must obey the constraint The reason for this restriction will become apparent later 88

89 Affine Combination Mini-Quiz Which of these are valid affine combinations? 89

90 General Affine Combination Mini-Quiz Consider three points O, P, Q and the affine combination What 2D shape is described by this combination when 90

91 Affine Frames So far we have not had to resort to any notion of an origin However, in many cases origins are useful An affine frame for an affine space is formally defined to consist of a basis and an origin, where and are a basis for 91

92 Illustration of a 3D Affine Frame The basis is linearly independent 92

93 Vectors and Points in Affine Frames Given an affine frame, we can define points and vectors accordingly, The arithmetic operations from the head-to-tail axiom can be carried out directly in the affine frame 93

94 Frame-Based Addition of Vectors Vectors add in a completely straightforward manner Given two vectors, Then 94

95 Frame-Based Scalar Multiplication of Vectors Scaling a vector by a scalar scales the coordinates of the vector This changes the length but not the direction of the vector 95

96 Frame-Based Point and Vector Arithmetic Adding a point to a vector gives a new point The position is the sum of the point position vector and the additive vector 96

97 Frame-Based Point Arithmetic Subtracting points leads, as we expect, to a vector 97

98 Frame-Based Point and Vectors Mini-Quiz So What s wrong with calculating P+Q? 98

99 Frame-Based Affine Combinations The coordinate axiom also explains the seemingly arbitrary constraint on affine combinations that Recall that an affine combination is In turn, each point can be written as 99

100 Frame-Based Affine Combinations Substituting into the expression for the affine combination, This is only a valid point if the coefficient on is 1, which can only happen if the coefficients sum to 1 100

101 Barycentric Coordinates So far we ve treated the coordinate representation of points and vectors a bit differently The relationship between affine combinations and is no coincidence It is possible to express both using barycentric coordinates 101

102 Barycentric Coordinates We consider each point used in our affine combination as being used to form a spanning set, 102

103 Barycentric Coordinates A point is simply an affine combination of the basis points or the n-dimensional simplex, where if P is a valid point then 103

104 Barycentric Coordinates in Action 104

105 Vectors in Barycentric Coordinates Now, a vector can be written as but we d like to similarly express it in barycentric coordinates in the form 105

106 Vectors in Barycentric Coordinates Substituting the expressions for the points in terms of the vectors, What we want Additional point bits 106

107 Vectors in Barycentric Coordinates We eliminate the point bits by setting the constraint In other words, vectors are represented as barycentric coordinates whose values sum to zero 107

108 Barycentric Coordinates Mini-Quiz How do you think affine subspaces can be represented using barycentric coordinates? How do you think affine independence can be represented using barycentric coordinates? 108

109 Summary of Barycentric Coordinates We can use sets of points to define a basis For an n-dimensional space we use n+1 points known as an n-dimensional simplex In this basis, points are represented by coordinates that sum to 1 Vectors are represented by coordinates that sum to 0 Affine subspaces correspond to lower dimensional simplexes Affine independence happens when no point is an affine combination of the other points 109

110 So What s the Problem with Affine Spaces? We ve talked about affine spaces We now have points as well as vectors We have discussed the algebra which links points and vectors together We can describe geometric objects using affine combinations and barycentric coordinates However, a few useful things are still missing! 110

111 More Missing Bits We still don t have a notion of distances Just ratios on lines between points We still don t have a notion of angles We still don t have an absolute origin These are all introduced in Euclidean geometry 111

112 Euclidean Spaces Introduction Tuples Vectors and Vector Spaces Affine Spaces Euclidean Spaces 112

113 Euclidean Spaces Euclidean spaces extend affine spaces by adding notions of length and angle The Euclidean structure is determined by the forms of the equations used to calculate these quantities We are going to just give these without proof However, we can motivate the problem by considering the problem of orthogonally projecting one vector onto another vector First, though, we need some additional vector notation 113

114 Some More Vector Notation Since we are going to define lengths and directions, we can now decompose a vector into the product of its length and an orthonormal vector which defines its direction Length (+ve) Orthonormal vector (length=1) 114

115 Projecting Vectors onto Vectors Consider two vectors and that occupy the same affine space What s the orthogonal projection of onto? 115

116 The Answer Projected vector 116

117 The Answer Projected vector Projected vector 117

118 Computing the Answer We need to compute both the direction and length of the projection vector The direction of the vector must be parallel to Therefore, we are going to define the orthogonal projection as Scale factor Orthonormal vector parallel to the right direction 118

119 Length of the Scaled Projection From basic trigonometry, the length of the parallel projection is From our assumed solution the length must be The dodgy bit! 119

120 Special Case of the Scalar Product The analysis from the past few slides shows that Furthermore, cosθ gives us the right sign behaviour Therefore, 120

121 General Case of the Scalar Product If we now let both of our vectors be non-normalised, then The scalar product is a special case of an inner product (we might meet these again later) 121

122 Lengths, Angles and Distances Lengths and angles are (circularly) defined as The distance function (or metric) between two points is the length of the vector between them, 122

123 Properties of Scalar Products Bilinearity: Positive definiteness: 123

124 Properties of Scalar Products Commutativity: Distributivity of the dot product over vector addition, Distributivity of vector addition over the dot product, 124

125 Scalar Product and Projection Mini-Quiz What s the value of 125

126 Summary The Euclidean structure is defined by the scalar product The scalar product is used to compute the orthogonal projection of one vector onto another vector The scalar product works in any number of dimensions It has many useful properties including bilinearity However, it only defines a one dimensional quantity (length) Vector products generalise this 126

127 Vector Products The vector product is the familiar cross product and is defined to be where is orthogonal to and The vector product is a special case of an exterior product and is a pseudovector (we won t meet these again) The vector product is only defined in 3D 127

128 Magnitude of the Vector Product The magnitude of the vector product is the area of the parallelogram having the vectors as sides 128

129 Direction of the Vector Product In general, in 3D we have 3 axes of rotation However, if we rotate and such that they still lie in the same plane, still points in the same direction Therefore, encodes information about two axes of rotation 129

130 Sign of the Vector Product Given the plane defined by the vectors, there are two possible choices of the normal vector out of or into the plane The choice is determined by the sign of the angle between the vectors Anticlockwise is positive 130

131 Right Hand Rule 131

132 Properties of Vector Products Anticommutativity: Distributivity: Distributivity: Parallelism: 132

133 Scalar Triple Product and Volumes We have shown that: Scalar products define lengths and angles Vector products define areas and a sense of perpendicularity Therefore, is it possible to extend this, at least in 3D, to the notion of a volume? The scalar triple product computes the volume of the parallelepiped defined by three linearly independent vectors 133

134 Scalar Triple Product and Volumes 134

135 Volume of the Parallelepiped The volume is computed as follows This is the absolute value of the scalar triple product 135

136 Scalar Triple Product The term inside the absolute value is the pseudoscalar which is known as the scalar triple product Pseudo-scalars reverse their sign under inversion, Fortunately we don t need to look at them any further! 136

137 Cartesian Frames So far we ve said an affine frame is simply an origin point and a set of linearly independent vectors Now that we can talk about angles and lengths, we can define a Cartesian coordinate frame A Cartesian Frame has the property that its basis vectors are orthonormal 137

138 Orthonormal Basis Set An orthonormal basis set has the property that its vectors: Are orthogonal to one another Are of unit length More compactly, 138

139 Illustration of the Cartesian Basis Set The basis is orthonormal 139

140 Cartesian Frames in 3D (Finally!) The basis set in 3D is written using the familiar vectors, Any vector can be written as 140

141 Scalar and Vector Product in the Cartesian Frame The scalar product is given by The vector product is given by 141

142 Euclidean Space Summary Euclidean spaces add notions of length and angles through the scalar product These can be extended by the vector and vector triple products to give areas and volumes in 3D The Cartesian Coordinate frame is a special case with a dedicated origin and an orthonormal basis set 142