Subspace Video Representations. CS 510 Lecture #22 April 14 th, 2014

Transcription

1 Subspace Video Representations CS 510 Lecture #22 April 14 th, 2014

2 Standard BoW : Video Edition Research in video analysis is still new BoW is currently the most common method for comparing videos STIPs for interest points For still images, fixed grids are now common HoF or Hof+HoG feature descriptors K-Means clustering to create codebook TF-IDF or histogram correlation to determine similarity 2

3 Video Matching : CSU Edition We (CSU) do research on video matching Lui et al F&G 08, CVPR 10, F&G 11 O Hara et al F&G 11, IVC 12, AVSS 12, CVPR 12, WACV 13 Marrinan et al CVPR 14 We also do other video-related work Motion segmentation : Mo et al ECCV 12 Video stabilization : Dutta et al WACV 14 Face video challenge : Beveridge et al BTAS 13 3

4 CSU video BoW We never use BoW to match videos Assumes temporal segmentation Short, pre-attended videos Too slow Many feature descriptors Many label look-ups Too selective (ignores data) Most pixels not part of any STIP feature Only edge/flow information contributes Complex training process Our approach is non-standard Not in OpenCV or Szeliski s text but promising 4

5 CSU s Approach to Video Analysis Goal : Describe videos in natural language (english) 1. Video è Tracklets A mini-video that tracks one moving object A bounding box for each frame Size & location of box varies across frames 2. Tracklet è Label Streams One label per frame per stream Streams: Appearance, Action, Trajectory Generate nouns, verbs, relations 3. Label Stream è Event Description In sentences (with nouns, verbs & relations) Less well developed than #1 and #2 above 5

6 Example Video Play video 6

7 Action Labeling There are many parts to this process And most need further development Today, we focus on action labeling How do we label a tracklet segment as Walking? Running? Digging? Picking up? 7

8 Action Labeling Framework Start with short training videos of actions 1.5 seconds of a person digging 1.5 seconds of a person walking 1.5 seconds of Possibly many examples per verb Now, given a new video : Divide it into tracklets For every 1.5 second window of each tracklet Measure it s similarity to training actions Return label of best fit (most similar training action) 8

9 Action Labeling as Video Matching Every window is a short, small video So is every training sample Labeling reduces to Measuring similarity between videos Selecting the most similar video If similarity score too low, no match Training samples extracted automatically The more training samples, the better How do we label them? Wigness, et al WACV 14 9

10 Measuring Video Similarity Goal: measure the similarity between two videos: Videos are small and short Videos focus on one actor Use all available information Step 1 : resample Resample test video to match height & width of training video Sliding window removes need for temporal resampling Result: two video cubes of the same size 10

11 Video as Subspace A video is a sequence of image frames Each frame is a point in image space Dimensionality : h w A video can be modeled by the image subspace spanned by its frames Note: h w >> t (# of frames) The subspace basis is computed using PCA Dimensionality = # of non-zero eigenvalues Eigenvectors define a basis for the subspace 11

12 Video as Subspace (II) Represent video as matrix X Every column is a vectorized frame Compute PCA XX T = RλR T Eigenvectors R are the subspace basis Remove eigenvectors with zero eigenvalues 12

13 Video as Subspace : Why? Why represent a video as a subspace? It contains every frame of the video It contains every linear combination of frames In effect, it interpolates between frames Motion interpolation Illumination interpolation But only linear interpolation Compact # of basis # of frames 13

14 Comparing Subspaces To compare videos, we compare their subspaces If two subspaces intersect The videos contain the same frame, or A combination of frames from video 1 equals a combination of frames from video 2 But this hardly ever happens (never) How do we compare non-intersecting subspaces? 14

15 Visualizing Subspace Comparison Imagine 2 1D subspaces in 3D i.e. two vectors in three space Call one A, the other B The vectors are normalized (basis vectors) If they are same, the subspaces intersect Otherwise, the smaller the angle between them, the more similar the subspaces So AB measures similarity 15

16 Visualizing (II) Now imagine 2 2D subspaces in 4D Each subspace is a plane In 4D, they may not intersect In fact, assume they don t Call one subspace A and the other B There exists an angle between any vector in A and any vector in B There exists a pair of vectors for which this angle is smallest Call these vectors the principal vector Call this angle the 1 st principal angle Among all vectors perpendicular to the principal vectors, there is a pair with the second smallest angle Call this the 2 nd principal angle 16

17 Visualizing (III) In other words, the 1 st principal angle is: argmin a A,b B, a =1, b =1 And if a 1, b 1 are the 1 st principal vectors, the 2 nd principal angle is: argmin a b ( ) a 2 A,b 2 B,a 1 a 2 =0,b 1 b 2 =0, a 2 =1, b 2 =1 a 2 b 2 ( ) The pattern repeats for higher dimensional subspaces 17

18 Computing Principal Angles There are in infinite number of vectors in every subspace, but We can compute the principal angles as: SVD( A T B) = RλR T Where A and B are sets of orthogonal basis vectors The principal angles are the eigenvalues The principal vectors are the eigenvectors 18

19 CSU Version #1 Compare tracklets by Using PCA to compute the subspace basis set of tracklet #1 Use PCA again to compute the subspace of tracklet #2 Use PCA to compute principal angle between the subspaces The smallest principal angle wins 19

20 Could you use this for PA3? Perhaps Substitute STIP cube for tracklet Use basis vectors as descriptor Principal angle becomes similarity measure Inverse becomes distance Cluster using pairwise distances Hint: use agglomerative clustering K-Means doesn t work well ( mean is difficult) Use in bag of words 20