Big Data and Big Water: Mining Ocean Vessel Trajectory Data

Transcription

1 Big Data and Big Water: Mining Ocean Vessel Trajectory Data Stan Matwin Institute for Big Data Analytics

2 Global AIS signal Satellites, ground stations Collectively represented, shows deep knowledge

3 AIS data Started 2004, designed for safety 400,000 ships At least 100M records/day Data gaps Noise in the data human error manipulation

4 Value of AIS data Monitoring Detecting anomalies Warnings Understanding activities Fisheries Marine Protected Areas Risk analysis.

5 Machine learning from AIS data clustering classification Ship trajectory representation is the key

6 Trajectory clustering Port vicinity Mid-ocean

7 Trajectory representation and reasoning Spaccapietra, et al GeoPKDD Palotta, Vespe and Bryan Soares et al 2014

8 Trajectory clustering Distance-based Instances in a cluster are closer to each other than to instances outside Density-based the neighborhood of a given radius of each instance in a cluster has to contain a minimum number of points (Ester et al. 96) TRACLUS [Lee et al.] Hybrid?

9 Density-based clustering near port DBSCANSD: Density-Based Spatial Clustering for Applications with Noise, Speed and Direction [Li et al. 2014]

10 TRACLUS and DBSANSD applying DBSCANSD on the ship trajectory dataset, geographically close trajectory points with similar direction and speed will be grouped together to form a cluster

11 Density-based clustering for near port trajectories Need to handle vessel motion and stopping There are public IMO rules for ship movement near port (no speed) We inform the clustering algorithm about IMO rules: parameter selection

12 Clusters are represented using centroid and envelope - Gravity vector GV Clusters are partitioned, each will have a number of centroids (GVs) GV consists of GVi =< COGavg, SOGavg, LATavg, LONavg>

13 Results IMO rules: purple arrows and dashed lines 46,000 records over 2 months 13 moving clusters, 1 stopping cluster: GVs are black dots

14 Results -LA 330,000 records: 100K moving, 230K stopping Again, clustering is informed by IMO rules 26 stopping clusters, 51 moving clusters (see GVs) Stopping and moving clusters correctly overlap

15 Speed anomaly detection - mid-ocean Beginning and end of a trip info missing Windows (grid) instead; size = 0.25 o x0.25 o, 600x200 cells; grid cells are graph nodes; edges if there is ship movement between nodes

16 Speed anomaly mid-ocean - idea Find shortest path between two points A* Determine divergence of the actual path from the shortest path

17 Features used Trajectory length (need Haversine correction to compute) Divergence of the trajectory from the straight line: " Φ f: Φ Θ and then area under the curve AA = NN Φ1 " ff(φ " ) ddφ " Gradient of trajectory with respect to latitude and longitude Finally, anomaly = ff = ffoooo ff rrrr LL OOOO Can be used to rank, not just to threshold

18 Speed in anomaly detection: context Region Ship type

19 Average speed distribution Heatmaps by speed in a given grid cell 3 months of data 5 ship types 9 weight categories

20 Standard deviations of speeds: Variance higher closer to coasts A rough anomaly detector = 3 σ

21 Fishing activity detection from AIS data The problem: Marine Protected Areas What is the effect of MPAs on fish stocks? Proxy question: is there more fishing going on near the MPAs than elsewhere?

22 Two approaches Hidden Markov Models: Sequence of states (Fishing, Not Fishing) and observables Y t ; Pr(S t S t-1 ) and Pr(Y t S t ) Issue: due to sparseness, we only use one attribute a Y t ; speed is a good attribute, but not for all ships.

23 Luckily, we have access to an expert and AIS data

24 Alternative: lets look at Bayesian classification Time-series approach seems reasonable model for trajectories in this setting But then we have too many attributes. We will apply attribute engineering borrowed from financial time series analysis Bollinger Band Money Flow Index Exponential Moving Averages

25 Pacific long liners - HMM Atlantic trawlers - Bayesian

26 Pacific long liners - HMM Atlantic trawlers - Bayesian

27 Future work For data representation: use more advanced techniques to represent trajectories, eg Dynamic Time Warping For labeling segments of trajectories, use Conditional Random Fields For clustering use spectral techniques with sparse data extensions (Nystrom)

28 Other current projects Human mobility Privacy Marketing Argumentation mining Trends in twitter traffic

29 Joint work with: Ron Pelot Rob Warren Erico Neves Behrouz Hajsoleimani Kristina Boerder Bo Liu Baifan Hu Casey Hillier

30 Discussion