Fishing Activity Visualization with Free Software Bigdata Analytics Institute

Size: px

Start display at page:

Download "Fishing Activity Visualization with Free Software Bigdata Analytics Institute"

Lily Washington
5 years ago
Views:

1 Fishing Activity Visualization with Free Software Bigdata Analytics Institute Erico N de Souza, PhD erico.souza@dal.ca Souza, Latouf (Bigdata Inst.) Bigdata Institute 1 / 22

2 Introduction What would you do if you received a task to manage 6.5 Billion GPS points? How to store it? How to retrieve it? How to visualize it? How to use some Machine Learning (ML) on it? This presentation does not intend to teach audience on which tool is the best You should see this as a generic guideline about what should be used to tackle the problem All tools have their usefulness, but you have to think and test when you should use them... Souza, Latouf (Bigdata Inst.) Bigdata Institute 2 / 22

3 AIS Data Automatic Identification System (AIS): protocol for vessel position report Multiple types of messages, but 90% belong to messages types: 1,2 and 3 Messsage types 1,2 and 3 contain more than 20 fields Our experience indicated that we only used 6 of them Souza, Latouf (Bigdata Inst.) Bigdata Institute 3 / 22

4 Current Workflow... Database Storage Pre-process/Cleaning DBA ExactEarth Souza, Latouf (Bigdata Inst.) Bigdata Institute 4 / 22

5 Which tools are the best to data retrieval? It really depends on the problem... Relational DBs can be very efficient to recover very large datasets The main advantages: 1 It has mathematically sound unified language (SQL); 2 Normalization guarantees (if wisely used...); 3 Data integrity The main issues with Relational DBs: 1 Not easily configurable parallel servers; 2 Schemas are not easily adaptable (cannot add easily new fields in a table); 3 Not scalable when data reaches billions of instances... If you are clever, you may design an schema with summaries of portions of the data, which will avoid touching in each data point... It all depends on your client s needs... Souza, Latouf (Bigdata Inst.) Bigdata Institute 5 / 22

6 Which tools are the best to data retrieval?(cont.) The Non-SQL solutions are an option. Which one use? Main issues: 1 Lack of normalization (Big Table); 2 Lack of an unified search language; 3 Many tools do not have good documentation; 4 Storage does not guarantee data integrity... Main reasons to use: 1 Promise of faster searches; 2 Parallelism; 3 Flexibility to create new fields for the data stored; 4 Data replication... Different tools were tested to answer the question in the beginning... Souza, Latouf (Bigdata Inst.) Bigdata Institute 6 / 22

7 Non-SQL Tools Evaluated... Cassandra Distributed hierarchical database (the order you list the fields in creation time will indicate how the table will be partitioned) Language tries to resemble SQL syntax Queries have to include all fields when user applies an restriction or projection Low adaptability MongoDB Distributed document database (purely based on JSON) Implemented in C++ and uses its own protocol for data transfer It does not require the user to define an schema, and it is more flexible to execute queries Slower than other tools to retrieve statistics from the fields, and requires some effort to configure multiple nodes. High adaptability Souza, Latouf (Bigdata Inst.) Bigdata Institute 7 / 22

8 Non-SQL Tools Evaluated... Solr Search tool (purely based on JSON) Depends on Java API to provide results to queries (Solr is a web service). Cloud configuration and data distribution simple Query is executed using a http request to a Solr server. High adaptability The decision really was between MongoDB and Solr Solr was chosen because it is faster to summarize data and easier to configure in distributed mode. This does not mean you should chose Solr as your solution... Souza, Latouf (Bigdata Inst.) Bigdata Institute 8 / 22

9 Some Details about the indexes... Each index has around 500 Million points. This number was decided to limit the size of the index. Roughly, each index has 110Gb in size Not all AIS message is available for searching. Messages 1, 2 and 3 have more than 20 different fields. We have to choose which of them will be searchable Based on previous experiences, we took the decision to store in Solr only 6 fields, and the rest can be reached by request. Souza, Latouf (Bigdata Inst.) Bigdata Institute 9 / 22

10 Solr Organization Souza, Latouf (Bigdata Inst.) Bigdata Institute 10 / 22

11 Machine Learning (ML) on Large Datasets... Problem definition: Fishing activity detection Different types of vessels: Trawlers and Longliners Data labeled by a Biologist expert (16 Longliners and 8 Trawlers - close to 1 Million points) Which tool to use? In previous works, we have heavily invested in R as tool to deal with ML tasks It can be used to build initial models and testing, but not scalable... Python offers many ML libraries and fast access tools to Solr... Souza, Latouf (Bigdata Inst.) Bigdata Institute 11 / 22

12 ML on Large Datasets...(CONT.) An fast algorithm for trajectory segmentation was implemented, and extra features based on the segments were calculated The result of this pre-processing was given to Adaboost + Random Forest for training and prediction Only a portion the labeled data from Longliners was used for training (60%), and the algorithm is able to predict fishing of Trawlers and Longliners with 90% accuracy. Souza, Latouf (Bigdata Inst.) Bigdata Institute 12 / 22

13 Segmentation Results Souza, Latouf (Bigdata Inst.) Bigdata Institute 13 / 22

14 Visualization Web applications that make searching and plotting data easy Requirements: Data secured behind user login Access based on user role Map-based tools/widgets Yii2 PHP framework: MVC Architecture User role configuration Plugins and widgets Souza, Latouf (Bigdata Inst.) Bigdata Institute 14 / 22

15 Fishing Observer Python and Pysolr Display fishing activity, both by volume and individual points CesiumJS Geospatial 3D mapping platform for creating virtual globes based on JavaScript Industry standard vector formats, such as KML, GeoJSON, and TopoJSON, including terrain clamping. Timeline and animation widgets for controlling simulation time. Datashader Python library Simply adds a layer on top of Cesium Generates heatmaps using millions+ data points quickly Souza, Latouf (Bigdata Inst.) Bigdata Institute 15 / 22

16 Monthly Fishing Data is searchable by month Heatmaps for each week and full month Souza, Latouf (Bigdata Inst.) Bigdata Institute 16 / 22

17 Monthly Fishing Souza, Latouf (Bigdata Inst.) Bigdata Institute 17 / 22

18 Monthly Fishing Souza, Latouf (Bigdata Inst.) Bigdata Institute 18 / 22

19 Exporting Data CSV files stored by user name, accessible by search information In the future, we may store saved searches Souza, Latouf (Bigdata Inst.) Bigdata Institute 19 / 22

20 Conclusions & Future Directions For Data Management... Cloud systems seem to be the only scalable method to work with very large datasets Multiple choices, but what is the correct one? (For some problems, RDBs are still a competitive method...) Learning cost in both language and technology For Machine Learning... Python is at the moment the best scalable tool to deal with large datasets... Exploit usage of parallelism within python: Numba Souza, Latouf (Bigdata Inst.) Bigdata Institute 20 / 22

21 Conclusions & Future Directions For Visualization... As presented, we cannot show every data point: all development has to balance what will be presented with its meaning Build a data summary (in our case, a heatmap...) Tools improving fast: we used python tools to build the heatmap, but it can also be achieved with Javascript... Open Questions: How to integrate different types of data? (Raster or sound) Could the indexes be smaller? What types of patterns are of interest? Algorithm efficiency for large datasets... For next version: saving queries and maps, use of cloud environment with support from Compute Canada,... Souza, Latouf (Bigdata Inst.) Bigdata Institute 21 / 22

22 Tool s URL... Souza, Latouf (Bigdata Inst.) Bigdata Institute 22 / 22

Big Data Architect.

Big Data Architect. Big Data Architect www.austech.edu.au WHAT IS BIG DATA ARCHITECT? A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional