Introduction to Vector Space Models

Transcription

1 Vector Span, Subspaces, and Basis Vectors

2 Linear Combinations (Review) A linear combination is constructed from a set of terms v, v 2,..., v p by multiplying each term by a constant and adding the result: c = α v + α 2 v α n v n = n α i v i i= α i s are scalar constants v i terms can be scalars, vectors (usually), or matrices.

3 Linear Combinations Geometrically 2a+b a b -.5a-2b

4 Linear Dependence Geometrically Two vectors are linearly dependent if they are multiples of each other. In other words, they point in the same direction (or exact opposite direction)

5 Linear Dependence Geometrically Two vectors are linearly dependent if they are multiples of each other. In other words, they point in the same direction (or exact opposite direction) More than two are linearly dependent if at least one is a linear combination of the others.

6 Vector Span

7 Vector Span The span of a single vector v is the set of all scalar multiples of v: span(v) = {αv for any constant α}

8 Vector Span The span of a single vector v is the set of all scalar multiples of v: span(v) = {αv for any constant α} The span of a collection of vectors, V = {v, v 2,..., v n } is the set of all linear combinations of these vectors: span(v) = {α v +α 2 v 2 + +α n v n for any constants α,..., α n }

9 Vector Span - Example Question: What is the span of e, the first elementary vector?

10 Vector Span - Example Question: What is the span of e, the first elementary vector? span(e2) span(e) Answer: The x-axis! (or the first axis, however you want to label them!)

11 Vector Span - Example Question: What is the span of two linearly independent vectors, a and b, in R 2? a 2a+b b -.5a-2b

12 Vector Span - Example Question: What is the span of two linearly independent vectors, a and b, in R 2? a 2a+b b -.5a-2b Answer: All of R 2 (The whole space)!

13 Vector Span - Example Question: What about two linearly independent vectors in R 3?

14 Vector Span - Example Question: What about two linearly independent vectors in R 3? Answer: The plane (hyperplane) that contains both vectors. (A two-dimensional subspace of R 3 )

15 Subspace A subspace, S of R n is thought of as a flat (having no curvature) surface within R n. It is a collection of vectors which satisfies the following conditions:

16 Subspace A subspace, S of R n is thought of as a flat (having no curvature) surface within R n. It is a collection of vectors which satisfies the following conditions: The origin ( vector) is contained in S

17 Subspace A subspace, S of R n is thought of as a flat (having no curvature) surface within R n. It is a collection of vectors which satisfies the following conditions: The origin ( vector) is contained in S 2 If x and y are in S then the sum x + y is also in S

18 Subspace A subspace, S of R n is thought of as a flat (having no curvature) surface within R n. It is a collection of vectors which satisfies the following conditions: The origin ( vector) is contained in S 2 If x and y are in S then the sum x + y is also in S 3 If x is in S and α is a constant then αx is also in S.

19 Subspace In other words, it is an infinite subset of vectors (points) from a larger space (R n ) which satisfies the same properties as R n.

20 Subspace In other words, it is an infinite subset of vectors (points) from a larger space (R n ) which satisfies the same properties as R n. The dimension of a subspace is the minimum number of vectors it takes to span the space. (Think: # of axis.)

21 Hyperplane - Terminology A hyperplane is a subspace that has one dimension less than its ambient space.

22 Hyperplane - Terminology A hyperplane is a subspace that has one dimension less than its ambient space. Thus, in 3-dimensional space a hyperplane would be a 2-dimensional plane.

23 Hyperplane - Terminology A hyperplane is a subspace that has one dimension less than its ambient space. Thus, in 3-dimensional space a hyperplane would be a 2-dimensional plane. In 4-dimensional space, a hyperplane would be a 3-dimensional space (since we can t imagine this situation, it helps to keep the same picture. It is a flat subspace in 4-dimensions!)

24 Hyperplane - Terminology A Hyperplane cuts the ambient space in half - One half is above the hyperplane, one half is below the hyperplane.

25 Let s Practice Is the vector x = ( ) {( )} 4 in the span? 3

26 Let s Practice Is the vector x = ( ) {( )} 4 in the span? 3 2 Describe the span of one vector in R 3.

27 Let s Practice Is the vector x = ( ) {( )} 4 in the span? 3 2 Describe the span of one vector in R 3. 3 Describe the span of two linearly dependent vectors in R 3.

28 Let s Practice Is the vector x = ( ) {( )} 4 in the span? 3 2 Describe the span of one vector in R 3. 3 Describe the span of two linearly dependent vectors in R 3. {( )} {( ) ( )} 4 2 Compare the span to the span,. 2

29 Let s Practice Is the vector x = ( ) {( )} 4 in the span? 3 2 Describe the span of one vector in R 3. 3 Describe the span of two linearly dependent vectors in R 3. {( )} {( ) ( )} 4 2 Compare the span to the span,. 2 {( ) ( )} 5 2 What is the dimension of the span,? 2

30 The Coordinate Plane e = ( ) and e 2 = span(e2) ( ). span(e) The axes of this plane are examples of basis vectors. A group of vectors make a basis for a space (or subspace) if they are linearly independent and span the space.

31 Coordinate Pairs Coordinate pairs are represented in a basis. Each coordinate tells you how far to move along each basis direction. (2,3) e 2 e

32 Coordinate Pairs Coordinate pairs are represented in a basis. Each coordinate tells you how far to move along each basis direction. (2,3) e 2 e ( ) 2 When we say that a = we think go 2 units along the x 3 direction (e ) and then 3 units along the y direction (e 2 ).

33 Basis Vectors give Directions ( ) 2 When we say that a = we think go 2 units along the x 3 direction (e ) and then 3 units along the y direction (e 2 ). This implies a linear combination of the basis vectors e and e 2. a = 2 ( ) ( ) + 3 = 2e + 3e 2 The coordinates in this basis are the coefficients 2 and 3.

34 Basis Vectors give Directions a = 2 ( ) ( ) + 3 = 2e + 3e 2 The coordinates in this basis are the coefficients 2 and 3. (2,3)

35 A Different Basis? { ( ) What about a different basis, say v =, v 2 = Can we talk about coordinates in this basis? ( )}? (2,3) v v 2

36 A Different Basis? { ( ) What about a different basis, say v =, v 2 = Can we talk about coordinates in this basis? (2,3) ( )}? v v 2 a = ( ) 2 = α 3 v + α 2 v 2

37 A Different Basis? { ( ) What about a different basis, say v =, v 2 = Can we talk about coordinates in this basis? ( ) ( ) ( ) 2 a = = α 3 + α 2 ( )}? Sure. The coordinates in this basis would be the coefficients α and α 2. How do I find them? Solve a system of equations: ( ) ( ) α = α 2 α = 2.5, α 2 =.5 ( ) 2 3

38 A Different Basis? a = ( ) ( ) ( ) 2 = α 3 + α 2 α = 2.5, α 2 =.5 (2,3)

39 A Different Basis? a = ( ) ( ) ( ) 2 = α 3 + α 2 α = 2.5, α 2 =.5 (2,3)

40 A Different Basis? a = ( ) ( ) ( ) 2 = α 3 + α 2 α = 2.5, α 2 =.5 (2,3)

41 New Basis Old Basis { ( ) ( )} Our new basis vectors, v =, v 2 = are just linear combinations of our old (elementary) basis vectors. In fact, v = e + e 2 v 2 = e + e 2 With that in mind, we can always get back and forth between new and old basis: a = 2.5v +.5v 2 = 2.5(e + e 2 ) +.5( e + e 2 ) = 2e + 3e 2

42 A new set of variables... { ( ) ( )} This new basis v =, v 2 = represents a new set of variables. If our original variables were height and weight then v = height + weight v 2 = height + weight (2,3) v v 2

43 In General... When we say that x = x x 2. x n We generally consider the x i s as coordinates in the elementary basis, B = {e, e 2,..., e n }, Since x = x e + x 2 e x n e n Example: =

44 In General... x x 2 When we say that x =. We generally consider the x i s as x n coordinates in the elementary basis, B = {e, e 2,..., e n }, Since x = x e + x 2 e x n e n Methods like PCA change this basis to a new basis like in the previous example, giving us a new set of variables that are combinations of the original variables.

45 Let s Practice ( ) 4 What are the coordinates of the vector x = in the basis 3 {( ) ( )},? Draw a picture to make sure your answer lines up with intuition.

46 Let s Practice ( ) 4 What are the coordinates of the vector x = in the basis 3 {( ) ( )},? Draw a picture to make sure your answer lines up with intuition. 2 In the following picture what would be the signs (+/-) of the coordinates of the green point in the basis {v, v 2 }? v v2

47 Why consider a new basis Most often we want to create a new basis and then use a subset of the new basis vectors (i.e. axes, or new features)

48 Why consider a new basis Most often we want to create a new basis and then use a subset of the new basis vectors (i.e. axes, or new features) In many cases we hope that, since they are combinations of original features, we might be able to interpret these features. Hard to see with simple height + weight" and -height + weight" example from previous slides.

49 Suppose your original features were: height, weight, head_circumference, verbal_ability, quant_ability, income, capital_gains, mortgage_payment. The hope would be that your new features combined those in some meaningful way. Feature : height.5 weight.5 head_circumference.3 verbal_ability quant_ability income capital_gains mortgage_payment

50 Suppose your original features were: height, weight, head_circumference, verbal_ability, quant_ability, income, capital_gains, mortgage_payment. The hope would be that your new features combined those in some meaningful way. Feature 2: height weight head_circumference verbal_ability.7 quant_ability.8 income capital_gains mortgage_payment

51 Suppose your original features were: height, weight, head_circumference, verbal_ability, quant_ability, income, capital_gains, mortgage_payment. The hope would be that your new features combined those in some meaningful way. Feature 3: height weight head_circumference verbal_ability quant_ability income.6 capital_gains.6 mortgage_payment.44

52 Suppose your original features were: height, weight, head_circumference, verbal_ability, quant_ability, income, capital_gains, mortgage_payment. The hope would be that your new features combined those in some meaningful way. Feature 3: height weight head_circumference verbal_ability quant_ability income.6 capital_gains.6 mortgage_payment.44 There are 5 more features/basis vectors that were omitted from the analysis. More on this later.

53 A more complete example New basis vectors and new coordinates for our data...

54 Example - Factors in Text Document Document 2 Document 3 Document 4 My cat likes to eat dog food. It s insane. He won t eat tuna, but dog food? He s all over it. Check out this video of my dog chasing my cat around the house! He never gets tired! Simon! The cat is not a dog toy! Dumb dog. I injured my ankle playing football yesterday. It is bruised and swollen. Maybe sprained? So tired of being injured. My ankle just won t get better! I sprained it 2 months ago!

55 Example - Factors in Text B = doc doc2 doc3 doc4 cat 2 dog 2 3 tired injured ankle sprained

56 Example - Factors in Text B = doc doc2 doc3 doc4 cat 2 dog 2 3 tired injured ankle sprained In this model, the elementary basis vectors represent words from the dictionary: cat = dog =

57 Example - Factors in Text B = doc doc2 doc3 doc4 cat 2 dog 2 3 tired injured ankle sprained Each document is a linear combination of these term basis vectors: doc = + 2

58 Example - Factors in Text B = doc doc2 doc3 doc4 cat 2 dog 2 3 tired injured ankle sprained We can approximate this matrix using a matrix factorization.

59 Example - Factors in Text B = doc doc2 doc3 doc4 cat 2 dog 2 3 tired injured ankle sprained B Factor Factor2 cat. dog.6 tired.4.4 injured.8 ankle.8 sprained.8 ( doc doc2 doc3 doc4 )

60 Example - Factors in Text B = doc doc2 doc3 doc4 cat 2 dog 2 3 tired injured ankle sprained B doc doc2 doc3 doc4 cat.7 dog tired injured ankle sprained.72.88

61 Example - Factors in Text B Factor Factor2 cat. dog.6 tired.4.4 injured.8 ankle.8 sprained.8 ( doc doc2 doc3 doc4 ) Each column of B (each document) can be written as a linear combination of Factors:

62 Example - Factors in Text B Factor Factor2 cat. dog.6 tired.4.4 injured.8 ankle.8 sprained.8 ( doc doc2 doc3 doc4 ) Each column of B (each document) can be written as a linear combination of Factors: B 2 = doc2.7factor +.Factor 2

63 Example - Factors in Text B Factor Factor2 cat. dog.6 tired.4.4 injured.8 ankle.8 sprained.8 ( doc doc2 doc3 doc4 ) Each column of B (each document) can be written as a linear combination of Factors: B 2 = doc2.7factor +.Factor 2 B 3 = doc3 Factor +.9Factor 2

64 Example - Factors in Text B Factor Factor2 cat. dog.6 tired.4.4 injured.8 ankle.8 sprained.8 ( doc doc2 doc3 doc4 ) B 2 = doc2.7factor +.Factor 2 B 3 = doc3 Factor +.9Factor 2 This is a change of basis. (Technically there are 4 additional basis vectors, but there was very little information there so they ve been chopped off. The data is essentially 2-dimensional. More on this later)

65 Example - Factors in Text Factor Factor2 cat. dog.6 doc doc2 doc3 doc4 ( ) tired B injured ankle.8 sprained.8 Each Factor is a linear combination of original terms: Factor = }{{} cat }{{} dog }{{} tired

66 Example - Factors in Text Each Factor is a linear combination of original terms: Factor = }{{} cat +.6 }{{} dog +.4 }{{} tired Notice that the words injured, ankle, and sprained do not contribute to this factor at all. How might you interpret this factor (i.e. this new variable)?

67 Example - Factors in Text The entries in the Factor vectors are called loadings. They give some measure of how significant each variable is to that Factor. Factor cat. dog.6 tired.4 injured ankle sprained Variable dog is the most significant on Factor injured, ankle and sprained are the least significant.

68 Example - Factors in Text The entries in the coordinate matrix are called either scores or coordinates. They give us an idea of how important each factor is to each observation (document). doc doc2 doc3 doc4 ( ) Factor..7. Factor2..9. Documents and 2 are more aligned with Factor (Pets). Documents 3 and 4 are more aligned with Factor 2 (Injuries).

69 The Major Ideas from Today linear combination geometrically linear (in)dependence geometrically vector span subspace dimension of subspace hyperplane basis vectors coordinates in different bases (generic) factor analysis loadings scores/coordinates