Cut-And-Stitch: Efficient Parallel Learning of Linear Dynamical Systems on SMPs

Transcription

1 School of Computer Science Cut-And-Stitch: Efficient Parallel Learning of Linear Dynamical Systems on SMPs Lei Li, WenjieFu, Fan Guo, Todd C. Mowry, Christos Faloutsos Computer Science Department Carnegie Mellon University 1

2 Outline Motivation Background Proposed Method: Cut-And-Stitch Experiments Conclusion 2

3 Motivation Motion Capture, Human Motion Modeling financial data modeling, prediction Common tool: Linear Dynamical System (LDS) Challenge? Learning LDS is slow sensor data, pattern mining Need For Speed 3

4 Related Work Parallel mining of closed sequential patterns, [KDD 05, S. Cong, J. Han, D. Padua] Mining terabytes of data for frequent itemsets, [PPoPP07, G. Buehrer, S. Parthasarathy, S. Tatikonda, T. Kurc, J. Saltz] Parallelize the discovery of frequent patterns in large graphs, [IPDPS 07, S. Reinhardt, G. Karypis] Parallel algorithms for SVM Cascade SVM, [NIPS 04, H. Graf, E. Cosatto, L. Bottou, I. Durdanovic, V. Vapnik] Parallel Mixture of SVMs, [NIPS 02, R. Collobert, S. Bengio, Y. Bengio] PSVM (open src), [NIPS 07, E. Chang, K. Zhu, H. Wang, H. Bai, J. Li, Z. Qiu, H. Cui] MulticoreMap-Reduce approach, [NIPS 06, C. Chu, S. Kim, Y. Lin, Y. Yu, G. Bradski, A. Ng, K. Olukotun] 4

5 Problem Definition Problem: Givena sequence of data, findthe best model parameters for Linear Dynamical System Traditional Method: Maximum Likelihood Estimation via Expectation- Maximization(EM) algorithm Objective: Parallelize the learning algorithm Assumption: shared memory architecture 5

6 Linear Dynamical System aka. Kalman Filter Parameters: θ=(u 0, V 0, A, Γ, C, Σ) Observation: y 1 y n Hidden variables: z n (u 0, V 0 ) Z 1 (A, Γ) Z 2 (A z 2, Γ) Z 3 (A z 3, Γ) Z 4 (A z 4, Γ) Z 5 (C, Σ) (C z 2, Σ) (C z 3, Σ) (C z 4, Σ) (C z 5, Σ) Y 1 Y 2 Y 3 Y 4 Y 5 6

7 Example z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 given positions, estimate dynamics (i.e. params) Position of left elbow Time 7

8 Traditional: How to learn LDS? 8

9 Sequential Learning (EM) z 2 z 2 z 3 z 3 z 4 z 4 z 5 z 5 z 6 z 6 y 2 y 4 y 4 y 5 y 5 y 6 y 6 Compute P( y 1 ) Position of left elbow Measured Estimated Time

10 Sequential Learning (EM) z 2 z 3 z 3 z 4 z 4 z 5 z 5 z 6 z 6 y 4 y 4 y 5 y 5 y 6 y 6 From P( y 1 ) Compute P(z 2 y 1, y 2 ) Position of left elbow Measured Intuition: z 2 may be close to Estimated Time 10

11 Sequential Learning (EM) z 2 z 3 z 4 z 4 z 5 z 5 z 6 z 6 y 4 y 4 y 5 y 5 y 6 y 6 Position of left elbow From P(z 2 y 1, y 2 ) Compute P(z 3 y 1, y 2, ) Measured Estimated Time 11

12 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 5 z 6 z 6 y 4 y 5 y 5 y 6 y 6 From P(z 3 y 1, y 2, ) Compute P(z 4 y 1, y 2,, y 4 ) Position of left elbow Measured Estimated Time 12

13 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 z 6 y 4 y 5 y 6 y 6 From P(z 4 y 1, y 2,, y 4 ) Compute P(z 5 y 1, y 2,, y 4, y 5 ) Position of left elbow Measured Estimated Time 13

14 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 5 y 1, y 2,, y 4, y 5 ) Compute P(z 6 y 1, y 2,, y 4, y 5, y 6 ) Position of left elbow Measured Estimated Time 14

15 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 6 y 1, y 2,, y 4, y 5, y 6 ) Compute P(z 5 y 1, y 2,, y 4, y 5, y 6 ) Position of left elbow Measured Intuition: take the future backward Estimated Time 15

16 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 6 y 1, y 2,, y 4, y 5, y 6 ) Compute P(z 4 y 1, y 2,, y 4, y 5, y 6 ) Position of left elbow Measured Estimated Time 16

17 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 4 y 1, y 2,, y 4, y 5, y 6 ) Compute P(z 3 y 1, y 2,, y 4, y 5, y 6 ) Position of left elbow Measured Estimated Time 17

18 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 3 y 1, y 2,, y 4, y 5, y 6 ) Compute P(z 2 y 1, y 2,, y 4, y 5, y 6 ) Position of left elbow Measured Estimated Time 18

19 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 2 y 1, y 2,, y 4, y 5, y 6 ) Compute P( y 1, y 2,, y 4, y 5, y 6 ) Position of left elbow Measured Estimated Time 19

20 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From all posterior,z 2,z 3,z 4,z 5,z 6 P( y 1, y 2,, y 4, y 5, y 6 ), P(z 2 y 1, y 2,, y 4, y 5, y 6 ) Compute sufficient statistics E[z i ] E[z i z i ] E[z i-1 z i ]

21 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 with sufficient statistics, compute argmax likelihood(θ) θ Position of left elbow Measured reconstructed signal Time 21

22 Sequential Learning (EM) z 2 z 3 z 4 z 5 z 6 y 4 y 5 y 6 From P(z 6 y 1, y 2,, y 4, y 5, y 6 ) Compute P(z 5 y 1, y 2,, y 4, y 5, y 6 ) Position Estimated Measured Time 22

23 z 3 z 4 z 2 z 6 z 5 y 4 y 5 y 6 Speed Bottleneck: sequential computation of posterior How to parallelizeit? 23

24 Leap of faith start computation without feedback from previous node (cut), and reconcile later (stitch) 24

25 Proposed Method: Cut-And-Stitch z 3 z 4 z 2 z 6 z 5 y 4 y 5 y 6 start computation without feedback from previous node (cut) reconcile later (stitch) υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z 2 z 3 z 4 z' 4 z 5 z 6 y 4 y 5 y 6 25

26 Cut-And-Stitch υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z' 2 z 2 z 2 z 3 z 4 z 4 z' 4 z' 4 z 5 z 6 z 6 y 2 Position of left elbow Measured Estimated Cut step: Estimate posteriors (E) y 4 y 4 y 5 y 6 y 6 P( y 1 ), P(z 3 ), P(z 5 y 5 ) Intuition: compute all three at once Time

27 Cut-And-Stitch υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z' 2 z 2 z 3 z 4 z' 4 z' 4 z 5 z 6 Position of left elbow Measured Estimated Cut step: Estimate posteriors (E) y 4 y 5 y 6 Time

28 Cut-And-Stitch υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z' 2 z 2 z 3 z 4 z' 4 z' 4 z 5 z 6 Position of left elbow Measured Cut step: Estimate posteriors (E) y 4 y 5 y 6 Intuition: backward adjust all at once Time

29 Cut-And-Stitch Stitch step: Collect sufficient statistics for each block (C) E[z i ] E[z i z i ] E[z i-1 z i ] υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z' 2 z 2 z 3 z 4 z' 4 z' 4 z 5 z 6 y 4 y 5 y 6

30 Cut-And-Stitch υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z' 2 z 2 z 3 z 4 z' 4 z' 4 z 5 z 6 Position of left elbow Measured y 4 y 5 y 6 Stitch step: Collect sufficient Statistics (C) Maximize parameters (M) Time

31 Cut-And-Stitch υ 1,Φ 1,η 1,Ψ 1 υ 2,Φ 2,η 2,Ψ 2 υ 3,Φ 3,η 3,Ψ 3 z' 2 z 2 z 3 z 4 z' 4 z 5 z 6 Position of left elbow Measured Stitch together: Re-estimate block parameters (R) Intuition: exchange messages cross block y 4 y 5 y 6 Iterate reconstructed signal Time

32 Experiments Q1: How much speed up can we get? Q2: How good is the reconstuction accuracy? 32

33 Experiments Dataset: 58 human motion sequences, frames Each frame with 93 bone positions in body local coordinates Setup: Supercomputer: SGI Altixsystem, distributed shared memory architecture Multi-core desktop: 4 Intel Xeon cores, shared memory Task: Learn the dynamics, hidden variables and reconstruct motion 33

34 Q1: How much speed up? Supercomputer Result speedup ideal average of 58 # of processors 34

35 Q1: How much speed up? Multi-core Result ideal speedup average of 58 # of cores 35

36 Q2: How good? 2.500% 2.000% Normalized Reconstruction Error 1.500% 1.000% Sequential alg Cut-And-Stitch 0.500% 0.000% walking (#22) jumping (#1) running (#45) Result: ~ IDENTICAL accuracy 36

37 Conclusion & Contributions General approximate parallel learning algorithm for LDS Near linear speed up Accuracy (NRE): ~ identical to sequential learning Easily extended to HMM and other chain Markovian models Software (C++ w. openmp) and datasets: 37

38 Thank you Questions? WELCOME TO POSTER TONIGHT Contact info: Wenjie Fu Lei Li Christos Faloutsos Fan Guo Todd Mowry 38

39 Promising Extensions Extension HMM other Markov models (similar graphical model) Open Problem: Can prove the error bound? 39

40 cpu1 cpu2 cpu3 cpu4 E E E E E C C C C M R R R R C C C C M R R R R 40 Timeline Stitch Cut Stitch Cut Initial iteration Iteration 1 I

41 Approximate Parallel Learning Iteration 0 Warm Up Iteration 1 Iteration 2 Iteration 3 Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 41