2. Related Work Hand-crafted Features Based Trajectory Prediction Deep Neural Networks Based Trajectory Prediction

Transcription

1 Encodng Crowd Interacton wth Deep Neural Network for Pedestran Trajectory Predcton Yanyu Xu ShanghaTech Unversty Zhxn Pao ShanghaTech Unversty Shenghua Gao ShanghaTech Unversty Abstract P P 3 Pedestran trajectory predcton s a challengng task because of the complex nature of humans. In ths paper, we tackle the problem wthn a deep learnng framework by consderng moton nformaton of each pedestran and ts nteracton wth the crowd. Specfcally, motvated by the resdual learnng n deep learnng, we propose to predct dsplacement between neghborng frames for each pedestran sequentally. To predct such dsplacement, we desgn a crowd nteracton deep neural network (CIDNN) whch consders the dfferent mportance of dfferent pedestrans for the dsplacement predcton of a target pedestran. Specfcally, we use an LSTM to model moton nformaton for all pedestrans and use a mult-layer perceptron to map the locaton of each pedestran to a hgh dmensonal feature space where the nner product between features s used as a measurement for the spatal affnty between two pedestrans. Then we weght the moton features of all pedestrans based on ther spatal affnty to the target pedestran for locaton dsplacement predcton. Extensve experments on publcly avalable datasets valdate the effectveness of our method for trajectory predcton.. Introducton Pedestran trajectory predcton ams to predct a contnuous set of locaton coordnates of a pedestran n future based on ts hstory path, and t s an mportant task n computer vson because of ts potental applcatons n behavor predcton [24] [4], traffc flow segmentaton [22], crowd moton analyss [3], crowd countng and segmentaton [27], abnormal detecton [6], etc.. Tremendous efforts have been made to solve ths problem [3] [] [29] [3]. However, due to the complex nature of pedestrans, t remans a challengng problem. In practce, to make the problem tractable, some work has attempted to model the Equal contrbutons Correspondng author P T P t-2 P T t- P T t P P P 3 P T t P 6 t- t -2 + t P 3 t- t-2 P 4 P 4 t t- t-p t 2 Fgure. CIDNN motvaton llustraton. The moton for the target pedestran (P T ) from tme t to t+ depends on ts moton, and ts spatal affnty to other pedestrans(p and P 6) at tme t, and other pedestrans moton. Though ts dstance to pedestran P s far, but P moves fast, so t also nfluences the movement of P T. So trajectory predcton of P T should consder more pedestrans other than ts neghbors wth a fxed dstance, and dfferent pedestrans also have dfferent level of nfluence on the target pedestran. task by only consderng a few factors related to pedestran trajectory, ncludng decson makng process of ndvduals [], nteractons between the dfferent pedestrans [2], and hstorcal moton statstcs of each pedestran [26]. In lght of the success of deep learnng n computer vson, t has also been ntroduced to pedestran trajectory predcton, of whch Behavor Convolutonal Neural Network (Behavor CNN) [2] and Socal Long-Short Term Memory (Socal LSTM) [] are two representatve ones. Behavor CNN represents hstorcal trajectores of all pedestrans wth a poston dsplacement map n the mage space, and then a CNN s adapted to assocate each pedestran wth ts neghbors for future trajectory predcton. But such method cannot model the potental nteractons between pedestrans n a more dstant future. For example, as shown n Fg. a pedestran walkng very fast n a far dstance may also nfluence the walkng trajectory of the target pedestran, or f a group of pedestrans are walkng towards the target or they are standng n the target pedestran s walkng drecton, even they are far from the target pedestran, he/she may change hs/her walkng drecton n advance to keep away P 2 t-2 P 2 t P 4 t-2

2 from these people n advance. To prevent ths, Socal LSTM [] s proposed. It desgns a socal poolng layer to capture dependences between multple pedestrans and nteractons that could occur n a more dstant future, thus acheves better performance. However, such a socal poolng does not dfferentate the effect of neghborng pedestrans based on ther spatal postons and ther moton nformaton. In ths paper, we propose a Crowd Interacton Deep Neural Network framework (CIDNN) to sequentally predct the coordnate dsplacement between two frames for each pedestran. We assume the movement of the target pedestran depends on ts moton nformaton(speed, acceleraton), other pedestrans moton nformaton, as well as the spatal affnty between the target and all the rest pedestrans, where the spatal affnty measures the level of nfluence of the rest pedestrans to the target pedestran. To model the moton of each pedestran, an LSTM model s adapted whose nput s the coordnate sequence at dfferent moments of the pedestran. To measure the spatal affnty of each pedestran to the target pedestran at a gven moment, we feed the coordnates of a pedestran nto a mult-layer perceptron, and use the nner product between the coordnate feature of the pedestran and that of the target to measure spatal affnty. Then we module the nteractons between the target and all pedestrans ncludng tself as the product between ther spatal affnty and the moton feature of the correspondng pedestran, and feed the nteracton features nto another mult-layer perceptron for coordnate dsplacement predcton of the target n next frame. We consder all pedestrans n the scene as well as ther spatal affnty for trajectory predcton, thus as demonstrated n Table, our method outperforms both LSTM [] and Behavor CNN[2]. Further, dfferent from Socal LSTM and Behavor CNN that drectly predct the coordnates, we propose to predct the locaton dsplacement between between next and current frame, whch further valdates the effectveness of resdual learnng n computer vson [8][9]. The contrbutons of our paper can be summarzed as follows: Frstly, we propose a CIDNN archtecture for trajectory predcton, whch consders all pedestrans n a scene for trajectory predcton. Our CIDNN has three features: )an LSTM based moton encodng strategy; ) locaton based spatal affnty measurement; and ) coordnate dsplacement based trajectory predcton. We propose to use locaton based spatal affnty measurement module, whch expermentally shown ts good performance than dstance based spatal affnty. Ours takes coordnates as nput to enrch the number of tranng samples and facltate the network tranng. Consequently, our network archtecture of CIDNN s smple and can be paralleled easly. Therefore, our trajectory predcton s more effectve and effcent than exstng methods; Fnally, extensve experments valdate the effectveness of our model for trajectory predcton. 2. Related Work 2.. Hand-crafted Features Based Trajectory Predcton Socal force models and topc models are commonly used for hand-crafted features based trajectory predcton. Socal force models learn the moton patterns based on the nteractons between pedestrans. It s frst proposed to model the attractve and repulsve forces n []. Later Mehran et al. propose to use socal force model to learn the nteracton forces between people n [6]. Antonn et al. [2] propose a dscrete choce framework to predct pedestran s next step under assumpton that the destnaton and the route are known. Dfferent from socal force models, topcs models [22] [] [6] model the moton pattern based on spatal and temporal nformaton. Further, Trajectory clusterng [3] [7] [2] are also used for crowd flow estmaton by clusterng dfferent trajectores nto dfferent classes. However, all these methods are based hand-crafted features, whch lmts the performance of trajectory predcton Deep Neural Networks Based Trajectory Predcton Deep learnng based methods have been ntroduced for pedestran trajectory predcton [] [2] [7] n lght of ts good performance for many computer vson tasks [9][2]. Specfcally, Behavour-CNN [2] employs a 2D map to encode the hstory walkng path and use a CNN to model the nteractons between dfferent pedestrans, yet t doesn t consder the effect of pedestrans n a more dstant future. Socal LSTM [] for human trajectory predcton desgn a Socal Poolng layer to capture dependences between multple correlated sequences and nteractons that could occur n a more dstant future, but t doesn t consder the d- fferent mportance of dfferent pedestrans. In [4] Lee et al. employ RNN to capture past moton hstores, the semantc scene context and nteractons among multple a- gents for trajectory predctons n dynamc scenes. In [7], Su et al. propose to deploy long short-term memory (LST- M) networks wth socal-aware recurrent Gaussan processes to model the complex transtons and uncertantes of the crowd and acheves good performance for trajectory predcton. But t also only consders the neghborng pedestrans and does not treat them dfferently. As aforementoned, some pedestrans n a far dstance but wth a fast movement speed may also nfluence the target pedestran s trajectory n next moment, and dfferent pedestrans have a dfferent nfluence level on the target pedestrans trajectory. In ths paper, we propose to take both factors nto our consderaton for trajectory predcton.

3 Locaton Encoder Module fc S fc2 fc3 t h t, a t Crowd Interacton z t Moton Encoder Module LSTM LSTM S, S,..., St 2 fc fc fc 2 3 S t h t, a t z t LSTM LSTM, S S2,..., S t N S t fc 2 Inner product fc fc3 N h t Scalar multplcaton, N a t Sum N z t LSTM c t LSTM fc N, N N S S2,..., S t S t Dsplacement Predcton Module Fgure 2. The archtecture of crowd nteracton deep neural network (CIDNN). 3. Method 3.. The Formulaton for Pedestran Trajectory Predcton Assume that there are N pedestrans p,..., p N, and t s current tme stamp (frame). The spatal locaton (coordnate) of the th pedestran p ( [, N]) at tme t s denoted as S t = [x t, y t], where x t [, X], y t [, Y ], and [X, Y ] s the spatal resoluton of vdeo frames. Gven the spatal coordnates S :t of each pedestran from begnnng to tme t, trajectory predcton ams at predctng the coordnates n the future tme perod from t + to t + T,.e., S t+:t+t. Dfferent from prevous work [2] whch predcts all the coordnates n all these frames smultaneously, we sequentally predct the coordnates n each future frame. Further, much prevous work shows that the resdual learnng or dsplacement predcton s easer for mage classfcaton [8], face algnment [28], as well as pose estmaton []. Snce our work sequentally estmates the coordnates at each tme stamp, therefore we propose to predct the locaton dsplacement correspondng to a current frame for each pedestran. Mathematcally, our work ams at learnng a nonlnear functon F by mnmzng the followng objectve functon: obs+t F = arg mn S t+ (S t + F (S :t )) 2 () F t=obs Here obs s the number of observed frames and F functon as an estmaton of locaton dsplacement for each pedestran. Such locaton dsplacement or the movement from current frame to next frame s related to the pedestran s hstory moton, other pedestrans spatal affnty to the target pedestran as well as ther hstory moton. To model these factors for trajectory predcton, we ntroduce a Crowd Interacton Deep Neural Network (CIDNN) for dsplacement predcton. The archtecture of CIDNN s depcted n Fg. 2. Specfcally, CIDNN conssts of four modules ncludng moton encoder module, locaton encoder module, crowd nteracton module, and dsplacement predcton module. Next, we wll detal these four modules sequentally Moton Encoder Module Moton encoder module s desgned to model moton pattern of pedestrans, ncludng dfferent hstory path and drecton, dfferent velocty and acceleraton. Long Short- Term Memory (LSTM) networks have been proved successful n moton modelng [][7]. By followng these work, we also employ LSTM networks to encode the moton nformaton for each pedestran. In our mplementaton, we stack two LSTM together for moton encodng. For each pedestran, we sequentally feed the hstory coordnates nto the stacked LSTM. For pedestran p, we denote the output of staked LSTM as z t t at tme t, then mathematcally z t = f(s,..., S t) (2) where the functon f( ) represents the nput-output functon of stacked LSTM. In our mplementaton, the number of nodes n hdden layer of both LSTM s fxed to be, and all pedestrans share the same stacked LSTM for moton encodng Locaton Encoder Module As aforementoned, the movement of a target pedestran from the current frame to next frame s related to all pedestrans moton nformaton, ncludng the pedestran hmself/herself as well ther spatal affnty to the target pedestran. So a straghtforward way s to lnearly combne the moton features of all pedestrans for dsplacement predcton, and the weght s based on the spatal affnty of each pedestran to the target pedestran, and the spatal affnty

4 measures the level of nfluence of each pedestran to the target pedestran. We denote the spatal affnty between p and p j at tme t as a,j t, then we can use some kernel functon κ(st, S j t ) for a,j t measurement, for example, Gaussan k- ernel κ(st, S j t ) = exp( λ St S j t 2 ). However, such Gaussan kernel only consders the spatal dstance between two pedestrans for spatal affnty measurement. It s worth notng that gven two pedestrans, even ther dstance to the target person are the same, ther spatal affnty to the target pedestran may be dfferent. There are two possble reasons for ths: ) As shown n Fg., there stands some pedestrans between p and p t, though the Eucldean dstance of p 3 to the target s smlar as that of p to the target, but p probably nfluences the trajectory of target more than p 3. ) Because of the vew angle of the camera, even though the dstances of two pedestran pars calculated based on coordnates n the mage are the same, t s possble the actual ground dstance are dfferent, consequently the spatal affnty of these two pedestran par should be dfferent too. For example, the dstance between two pedestrans n upper left corner may be the same wth that of two pedestrans n lower rght corner n Fg. 3, though ther coordnates based dstance are the same. Therefore, coordnates based spatal affnty s more meanngful than dstance based affnty measurement for trajectory predcton. So s there any way to automatcally learn an optcal spatal affnty measurement? The kernel trck says that κ(st, S j t ) = φ(st), φ(s j t ), here φ( ) s some nonlnear functon that maps the nput to a hgh dmensonal feature space, and, s the nner product operaton. However, such φ( ) s usually unknown. Motvated by the kernel trck, we propose to map the nput (coordnates) nto a hgh dmensonal feature space wth some neural network and use the nner product between the hdden nodes for spatal affnty measurement. Specfcally, we use a mult-layer perceptron as locaton encoder, whch contans 3 layers, and ReLU actvaton functon s used. The number of hdden nodes n these layers s 32, 64, 28, respectvely. We denote the output of locaton encoder for pedestran p at tme t as h t, then h t = g(s t) (3) Here g( ) represents the nput-output functon of the multlayer perceptron of local encoder Crowd Interacton Module Based on the output of locaton encoder, we can measure the spatal affnty between two pedestrans. For the a pedestran p j, we denote ts spatal affnty to the target pedestran p at tme t as a,j t, then a,j t = exp( h t, h j t ) j exp( h t, h j t It s worth notng that snce h t, h j t does not necessarly between [,], we use a softmax way to normalze t to [,], and use t as the affnty measurement. We can see that a,j t and a j, t are dfferent, and ths s reasonable because the movement of each pedestran s based on hmself/hereself as well as ts neghbors. Even though p s the nearest neghbour for p j, but p j may not be the nearest neghbour for p. Therefore, the level of nfluence of p to p j and the level of nfluence of of p j to p are dfferent. Based on the defnton of spatal affnty, we can model the level of nfluence of all pedestrans to person p, whch s denoted as c t, as follows: c t = j ) (4) a,j t z j t () Then we can use c t to predct the locaton dsplacement between tme t and t + for person p. Here we consder both the spatal affnty and the moton nformaton of dfferent pedestran for the trajectory predcton of the target pedestran. If the spatal affnty s larger or the pedestran moves fast, then t s lkely that the pedestran may nfluence the target more. 3.. Dsplacement Predcton Module We use one fully connected layer wth lnearty to map the total effect of all pedestrans to the target p to estmate the locaton dsplacement (δs t+) between tme t and t+: δs t+ = W c t + b (6) Here W, b s the parameters n ths fully connected layer. Once we get the locaton dsplacement, we can compute the coordnate of person p at tme t + : S ˆ t+ = S t + δst+. It s also worth notng that we predct the trajectory for each pedestran separately. Therefore our framework can be easly paralleled n mplementaton. Further, compared wth Behavor CNN [2] and Socal LSTM [], the number of hdden nodes n our framework s very small. Therefore our method s very effcent n mplementaton, especally when the number of pedestrans s small n a scene. 4. Experments 4.. Expermental Setup We mplement our solutons wth the PyTorch framework, and mn-batch based stochastc gradent descent s

5 Successful Cases Falure Cases Fgure 3. Qualtatve results: hstory trajectory (red), ground truth (blue), and predcted trajectores from our model (green). The frst three columns show some successful cases and last column shows some falure cases. We can see that our predcton always overlaps wth ground truth, whch shows the effectveness of our method. Please enlarge the fgure for better vsualzaton. used to optmze the objectve functon. We tran our network wth the followng hyper-parameters settng: mnbatch sze (26), learnng rate (.3), momentum (.9), weght decay (.), and number of epochs (,). The parameters are ntalzed wth Xaver. Datasets. We evaluate our method wth the followng publcly avalable human trajectory datasets: New York Grand Central (GC) [24], ETH [8], UCY [], the CUHK Crowd Dataset [2] and the subway staton dataset[3]. As shown n [8], these datasets also cover very challengng group behavors such as couples walkng together, groups crossng each other and groups formng and dspersng n some scenes. The GC dataset conssts of around 2,6 pedestrans and t s about one hour long. By followng the same expermental setup wth [2], 499 short clps are unformly segmented from GC dataset, and one sample can be obtaned from each clp. The frst 9% samples are used for tranng whle the remanng for test. The ETH dataset contans two scenes each wth 7 dfferent pedestrans splt nto two sets (ETH and Hotel). The UCY dataset ncludes two scenes wth 786 people. Ths dataset has 3-components: ZARA-, ZARA- 2 and UCY. These datasets represent crowded real-world settngs wth thousands of non-lnear trajectores. The CUHK Crowd Dataset contans many crowd vdeos wth dfferent denstes and perspectve scales n many envronments. The subway staton dataset s a 3-mnute sequence col- lected n the New York Grand Central Staton, wth each contanng more than 4, keypont trajectores n total. Followng the same expermental setup and evaluaton crtera as [], we use a leave-one-out approach on the sets of ETH and UCY. We tran and valdate our model on 4 sets and test on the remanng set. We repeat ths for all the sets. We also use the same tranng and testng procedure for other baselne methods used for performance comparson. Measurement. By followng the work [2], we use Average Dsplacement Error (ADE) as metrc to measure the performance of dfferent methods. ADE s the mean square error (MSE) overall estmated ponts of a trajectory and the true ponts. It can be mathematcally defned as follows: P Pobs+T ADE = t=obs kst+ S t+ k2 nt (7) In our experments, we have observed the trajectory for frames and use them to predct the trajectory for the next frames, therefore obs = and T =. Actually the data on the GC dataset s sampled from real vdeos wth tme nterval 2 frames, so the tme nterval between two neghbourng frames s.8 sec, and the predcton of the th frame s the coordnates n the comng 4 sec. Baselnes. Followng the expermental setup n [2], we desgn the followng baselnes: ) The constant acceleraton regressors were used to predct future walkng path of each pedestran, and ths baselne s termed as const acc; We also compare our method wth the

6 dataset const acc SF [23] S-LSTM [] B-CNN [2] SRGP [7] Ours ETH NA.9 HOTEL NA. ZARA NA. ZARA NA. UCY NA.2 GC NA.2 CUHK Crowd.46 NA.34 NA.29.8 subway staton.64 NA.33 NA.3.6 Besdes spatal affnty, the movement of someone also depends on hs movement velocty, drecton, etc. We show two examples n Fg. 4. We can see that the one walks slow would let the one walks fast go frst for both ground-truth and predcton n collson cases. Table. The performance comparson of dfferent methods on the GC, ETH, UCY, CUHK Crowd and subway staton datasets. obs gt pred obs gt pred followng state-of-the-art baselnes : ) Socal force (SF) [23] whch employs an agent for each pedestran to smulate the trajectory makng process. ) Behavour-CNN (B-CNN) [2] where a deep neural network (Behavor-CNN) s proposed to model pedestran behavors n crowd scenes; v) Socal LSTM (S-LSTM) [] where generated multple LSTMs for each pedestran are used to estmate ther postons consderng the neghborng pedestrans. v) SRGP [7] where long short-term memory (LSTM) networks wth socal-aware recurrent Gaussan processes s used to model the complex transtons and uncertantes of the crowd Performance Comparson We compare our method and other baselne methods on GC, ETH, UCY, the CUHK Crowd Dataset and the subway staton dataset n Table. We can see that our method sgnfcantly outperforms all exstng methods and other baselnes on all the datasets, whch valdates the effectveness of our soluton. The comparson based metrcs used n [] Fnal dsplacement error (FDE) and Average non-lnear dsplacement error (ANDE) s lsted n Table 2. Methods ADE FDE ANDE GC Subway GC Subway GC Subway B-CNN.24 NA.49 NA.284 NA S-LSTM Ours Table 2. Comparsons on more evaluaton metrc. We further show the predcted trajectory and ts ground truth on the GC dataset n Fg. 3. As aforementoned, every pedestrans trajectory wll be nfluenced by near other people, but our crowd nteracton module can learn dfferent patterns of ths nfluences. The frst three columns show that the model can well predct the trajectores even f they ntersect wth others. Meanwhle, n the last column of Fg. 3, we also show some falure cases, whch s probably due to the sudden changes of destnaton n ground truth. Even though such sudden change s hard to model, our method stll predcts very smlar trajectory compared wth ground truth. Because the same expermental setup of [], [2] and same tranng/testng sets are used, we drectly adapted the results of baselnes from [], [2]. The performance of Socal LSTM [], Behavor CNN [2], S- RGP [7], and SF [23] on these datasets are drectly adapted from the correspondng papers. Fgure 4. Examples of collson case Evaluaton of Dfferent Components n CIDNN Coordnate regresson vs. dsplacement regresson In our dsplacement predcton module, we use a fully connected layer to map the weghted features to estmate the dsplacement δst+ between tme t+ and tme t for pedestran p. Besdes the δs regresson, we also try to use the weghted features to drectly estmate St+. We show results of these two dfferent strateges n Table 3. We can see that dsplacement regresson always acheves hgher accuracy than drectly predctng the ground truth. Ths s because the LSTM encodes the velocty, acceleraton between contnuous frames well, and t s easer to predct the dsplacement only than predct (dsplacement + current coordnates). The good performance of dsplacement regresson strategy valdates the effectveness of resdual regresson n trajectory predcton, whch agrees wth exstng work for mage classfcaton and facal/body key ponts detecton. Wth Crowd Interacton module vs. Wthout Crowd Interacton module To valdate the effectveness of crowd nteracton module, we also tran a network wthout the crowd nteracton module,.e., we drectly estmate the dsplacement based on moton features extracted from stacked LSTM. We compare the performance wth/wthout crowd nteracton module n Table 3. We can see that that network wth crowd nteracton module performs better than the one wthout crowd nteracton module. Ths s because the network wth crowd nteracton module takes the dfferent mportance of dfferent neghborng pedestrans nto consderaton. The evaluaton of moton encoder To evaluate the mportance of moton encoder, we propose to replace t wth the dsplacements between all prevous neghborng frames n CIDNN, and we term such baselne as CIDNN w/o LST- M. The comparson between our method and CIDNN w/o LSTM s lsted n Table 3. We can see that our CIDNN acheves better performance, whch valdates the effectveness of LSTM for moton characterzaton.

7 Datasets Regresson Strategy Crowd Interacton Coordnate Dsplacement Wth Wthout ETH ZARA GC Datasets Spatal Affnty Moton Encoder Ours Gaussan Wth LSTM Wthout LSTM ETH ZARA GC Table 3. Performance evaluaton of dfferent components n CIDNN. We also nvestgate the performance of stacked LSTM wth dfferent layers for moton encoder on GC. When the number layer of stacked LSTM s, 2, and 3, the MSE results are.4,.2, and.3.consderng accuracy and effcency, we fx the layers to be 2. Dfferent spatal affnty measurement To evaluate the mportance of our locaton encoder, we also compare our method wth Gaussan kernel κ(st, S j t ) = exp( λ St S j t 2 ), whch s a dstance based spatal affnty measurement. The performance comparson s shown n Table 3. The good performance of our method valdates the effectveness of our locaton encoder module for spatal affnty measurement. We further show the relatonshp between our spatal affnty and the Eucldean dstance between two pedestran on GC n Fg. (a). We can see that the spatal affnty measured by our method s usually larger for ponts wth s- maller dstance, and smaller for ponts wth larger dstance. Sa shown n Fg. (b), snce the Eucldean dstance between pedestran par (P 2 P T 2 ), and (P 3, P T 2 ) s smlar, and the spatal affnty between them s also smlar, whle the Eucldean dstance between pars (P 2 P T 2 ) s less than that of (P 4 P T 2 ), the spatal affnty of (P 4 P T 2 ) s smaller. Further, as we dscussed earler, even two pedestran pars are wth dfferent dstances computed based coordnates, ther spatal affnty may be smlar because coordnates based dstance s not the actual ground plane dstance due to effect of vew angle. We also gve an example n Fg. (b), and we can see that even though the coordnate based dstance between pedestran par (P, P T ) s smaller than that of pedestran par (P 2, P T 2 ), but because of vew angle, ther ground dstance s smlar, so ther spatal affnty s also smlar. Fg. 6 shows spatal affnty for dfferent pedestrans at dfferent tme stamps n a scene on the GC dataset. We can see that neghborng pedestrans are usually wth larger spata affnty, whch valdates the effectveness of our spatal affnty defnton. The performance on challengng data To show the performance of dfferent methods for more challengng data, we splt the GC dataset nto two subsets (nonstraght/straght trajectores). Results are shown n 4. Re- Fgure. (a) The relatonshp between the our spatal affnty and Eucldean dstance on the GC dataset. We normalze the coordnates to [,] when we feed the coordnates to locaton encoder. So the dstance s also normalzed. (b) An nstance for llustratng the relatonshp between Eucldean dstance and spatal affnty. sults show our method acheves the best performance on both subsets. Method straght subset (97.8%) non-straght subset (2.2%) Ours B-CNN S-LSTM.4.24 Table 4. Predcton errors for straght non-straght trajectores. We randomly contamnate a fracton of tranng data wth gaussan nose N (, v 3 ), where v s ther mean velocty. When gaussan nose s %, %, %, and 2%, the MSE results on GC dataset s.2,.37,.43, and.4, whch shows the robustness of our method. Trajectory predcton for a longer tme The ADE of trajectory predcton for.8 sec, 4 sec, and 8 sec s.,.2, and.34, respectvely, on GC, whch s better than that of socal LSTM, whch acheves.9,.2, and.4, respectvely. We further see that the performance of trajectory predcton degenerates for a longer tme. But the mprovement of our method over socal LSTM ncreases as tme goes longer because our formulaton consders all possble pedestrans whch may contrbute the targets trajectory predcton n future. The nput of moton encoder In our mplementaton, we feed the coordnates of each pedestran at dfferent tme nto the moton encoder. We also try to replace the nput of moton encoder wth the dsplacement between neghborng frames on GC datasets. Such model acheves a score of.2 n terms of ADE on GC, whle our coordnates based model acheves.2 n terms of ADE. One possble reason s that the coordnates of pedestrans would provde extra locaton nformaton apart from moton, whch further boosts the performance of trajectory predcton Transferablty of locaton encoder and moton encoder To evaluate the transferablty of locaton encoder and moton encoder, we conduct cross doman experments. We

8 (a) Spatal layout of all pedestrans (b) Spatal affnty at t + (c) Spatal affnty at t (d) Spatal affnty at t + 3 (e) Spatal affnty at t + 4 (f) Spatal affnty at t + Fgure 6. Crowd Interacton scores for dfferent moton encoders at dfferent tme. (a) represents qualtatve results: hstory trajectory (red), ground truth (blue), and predcted trajectores wth our model (green). (b) - (f) represent spatal affnty scores at tme from t + to t + respectvely. x axs and y axs represent the ID of each pedestran. Settng Transferred components Source Target M and L M L Ours subway GC GC subway GC ETH GC HOTEL GC ZARA GC ZARA GC UCY Table. Predcton errors under dfferent transfer learnng settngs. evaluate the performance of locaton encoder (L) and moton encoder (M) traned on target doman wth the target doman. In our experments, we use GC and subway staton because of they both corresponds to subway scenes, and choose one dataset as source doman and use the other as target doman, as shown n Table.. Snce GC s much lager than subway staton, and the model traned on GC also acheves satsfactory results on subway. Further, we found moton s easer to transfer because all possble moton trend of all pedestrans can be well covered by a larger dataset. By contrast, the transferablty of locaton encoder s not so good because the scene layout as well as the camera perspectve are dfferent. Further, the performance of the model traned on GC and then fnetuned on subway s.7, and.3 f source/target s changed reversely. 4.. Tme cost We test the runnng tme of our method on the GC dataset. Our model s mplemented on an NVIDIA GeForce TITAN GPU platform and an Intel(R) Xeon(R) CPU E v3 3.4GHz CPU platform, respectvely. We run our program 2 tmes and calculate the average runnng tme for each mage. More precsely, the average runnng tme of CIDNN s.43 ms on GPU. The tme cost of CIDNN s.9 ms on CPU.. Concluson In ths paper, we desgn a crowd nteracton deep neural network (CIDNN) for dsplacement predcton. Our model consders the dfference level of nfluence of dfferent pedestrans n the crowd on the target pedestran. Specfcally, we propose to use LSTM to model the moton of each pedestran, then we weght the moton feature of all pedestrans based on ther spatal affnty to the target pedestran for locaton dsplacement predcton. Compared wth exstng work Socal LSTM[], Behavour-CNN[2], our method consders the dfferent mportance of all pedestrans based on ther spatal affnty to the target pedestran. Extensve experments on publcly avalable datasets valdate the effectveness of our method for trajectory predcton. The proposed soluton here s used for trajectory predcton. But t also can be appled to other applcatons, for example, facal keypont detecton n vdeos and human pose estmaton (body keypont detecton) n vdeos. Appearance based key pont detecton s usually tme-consumng. By combnng our method wth appearance based keypont detecton n key frames, we can avod keypont detecton for each frame, whch may mprove the effcency wthout reducng accuracy. 6. Acknowledgement Ths project s supported by the NSFC (No. 6234) and Program of Shangha Subject Chef Scentst (A type) (No.XD29).

9 References [] A. Alah, K. Goel, V. Ramanathan, A. Robcquet, L. Fe-Fe, and S. Savarese. Socal lstm: Human trajectory predcton n crowded spaces. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages 96 97, 26. [2] G. Antonn, M. Berlare, and M. Weber. Dscrete choce models of pedestran walkng behavor. Transportaton Research Part B, 4(8): , 26. [3] E. Bonabeau. Agent-based modelng: Methods and technques for smulatng human systems. Proceedngs of the Natonal Academy of Scences, 99(suppl 3): , 22. [4] B. Cancela, A. Iglesas, M. Ortega, and M. G. Penedo. Unsupervsed trajectory modellng usng temporal nformaton va mnmal paths. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages 23 26, 24. [] J. Carrera, P. Agrawal, K. Fragkadak, and J. Malk. Human pose estmaton wth teratve error feedback. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , 26. [6] R. Emonet, J. Varadarajan, and J.-M. Odobez. Extractng and locatng temporal motfs n vdeo scenes usng a herarchcal non parametrc bayesan model. In Computer Vson and Pattern Recognton (CVPR), 2 IEEE Conference on, pages IEEE, 2. [7] Y. D. B. Z. Hang Su, Jun Zhu. Forecast the plausble paths n crowd scenes. In Proceedngs of the Twenty-Sxth Internatonal Jont Conference on Artfcal Intellgence, IJCAI-7, pages , 27. [8] K. He, X. Zhang, S. Ren, and J. Sun. Deep resdual learnng for mage recognton. In Proceedngs of the IEEE conference on computer vson and pattern recognton, pages , 26. [9] K. He, X. Zhang, S. Ren, and J. Sun. Identty mappngs n deep resdual networks. In European Conference on Computer Vson, pages Sprnger, 26. [] D. Helbng and P. Molnar. Socal force model for pedestran dynamcs. Physcal revew E, ():4282, 99. [] T. M. Hospedales, J. L, S. Gong, and T. Xang. Identfyng rare and subtle behavors: A weakly supervsed jont topc model. IEEE Transactons on Pattern Analyss and Machne Intellgence, 33(2): , 2. [2] A. Krzhevsky, I. Sutskever, and G. E. Hnton. Imagenet classfcaton wth deep convolutonal neural networks. In Advances n neural nformaton processng systems, pages 97, 22. [3] J.-G. Lee, J. Han, and K.-Y. Whang. Trajectory clusterng: a partton-and-group framework. In Proceedngs of the 27 ACM SIGMOD nternatonal conference on Management of data, pages ACM, 27. [4] N. Lee, W. Cho, P. Vernaza, C. B. Choy, P. H. S. Torr, and M. Chandraker. Desre: Dstant future predcton n dynamc scenes wth nteractng agents. pages , 27. [] A. Lerner, Y. Chrysanthou, and D. Lschnsk. Crowds by example. In Computer Graphcs Forum, volume 26, pages Wley Onlne Lbrary, 27. [6] R. Mehran, A. Oyama, and M. Shah. Abnormal crowd behavor detecton usng socal force model. In Computer Vson and Pattern Recognton, 29. CVPR 29. IEEE Conference on, pages IEEE, 29. [7] B. Morrs and M. Trved. Learnng trajectory patterns by clusterng: Expermental studes and comparatve evaluaton. In Computer Vson and Pattern Recognton, 29. CVPR 29. IEEE Conference on, pages IEEE, 29. [8] S. Pellegrn, A. Ess, K. Schndler, and L. Van Gool. You ll never walk alone: Modelng socal behavor for mult-target trackng. In Computer Vson, 29 IEEE 2th Internatonal Conference on, pages IEEE, 29. [9] S. Ren, K. He, R. Grshck, and J. Sun. Faster r-cnn: Towards real-tme object detecton wth regon proposal networks. In Advances n neural nformaton processng systems, pages 9 99, 2. [2] J. Shao, C. Change Loy, and X. Wang. Scene-ndependent group proflng n crowd. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , 24. [2] X. Wang, K. T. Ma, G.-W. Ng, and W. E. L. Grmson. Trajectory analyss and semantc regon modelng usng nonparametrc herarchcal bayesan models. Internatonal journal of computer vson, 9(3):287 32, 2. [22] X. Wang, X. Ma, and W. E. L. Grmson. Unsupervsed actvty percepton n crowded and complcated scenes usng herarchcal bayesan models. IEEE Transactons on pattern analyss and machne ntellgence, 3(3):39, 29. [23] K. Yamaguch, A. C. Berg, L. E. Ortz, and T. L. Berg. Who are you wth and where are you gong? In Computer Vson and Pattern Recognton (CVPR), 2 IEEE Conference on, pages IEEE, 2. [24] S. Y, H. L, and X. Wang. Understandng pedestran behavors from statonary crowd groups. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , 2. [2] S. Y, H. L, and X. Wang. Pedestran behavor understandng and predcton wth deep neural networks. In European Conference on Computer Vson, pages Sprnger, 26. [26] S. Y, X. Wang, C. Lu, J. Ja, and H. L. l regularzed statonary-tme estmaton for crowd analyss. IEEE transactons on pattern analyss and machne ntellgence, 39():98 994, 27. [27] C. Zhang, H. L, X. Wang, and X. Yang. Cross-scene crowd countng va deep convolutonal neural networks. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , 2. [28] J. Zhang, S. Shan, M. Kan, and X. Chen. Coarse-to-fne auto-encoder networks (cfan) for real-tme face algnment. In ECCV, pages 6. Sprnger, 24. [29] B. Zhou, X. Tang, and X. Wang. Learnng collectve crowd behavors wth dynamc pedestran-agents. Internatonal Journal of Computer Vson, (): 68, 2.

10 [3] B. Zhou, X. Wang, and X. Tang. Random feld topc model for semantc regon analyss n crowded scenes from tracklets. In Computer Vson and Pattern Recognton (CVPR), 2 IEEE Conference on, pages IEEE, 2. [3] B. Zhou, X. Wang, and X. Tang. Understandng collectve crowd behavors: Learnng a mxture model of dynamc pedestran-agents. In Computer Vson and Pattern Recognton (CVPR), 22 IEEE Conference on, pages IEEE, 22.