Big Data Integrator Platform Platform Architecture and Features Dr. Hajira Jabeen Technical Team Leader-BDE University of Bonn

Transcription

1 Big Data Integrator Platform Platform Architecture and Features Dr. Hajira Jabeen Technical Team Leader-BDE University of Bonn BDE Presentation, EBDVF, 17

2 BigDataEurope 2

3 Making Big Data Accessible 3 How can we make it easy?

4 Platform Goals 4 Easy to: o o o o Install Develop Deploy Integrate

5 Platform Architecture 5

6 Platform Architecture 6

7 Platform Architecture 7

8 Platform Architecture 8 Support Layer App Layer Init Daemon Traffic Forecast GUIs Satellite Image Analysis Real-time Stream Monitoring... Platform Layer Spark Flink Monitor Kafka... Semantic Layer Ontario SANSA Semagrow Data Layer Hadoop RDF Store NOSQL Store Elasticsearch Resource Management Layer (Swarm) Hardware Layer Premises Cloud (AWS, GCE, MS Azure, ) Cassandra...

9 BDE Supported Frameworks 9 Search/indexing Data processing Apache Solr Apache Spark Data acquisition Apache Flink Apache Flume Semantic Components Message passing Strabon Apache Kafka Sextant Data storage GeoTriples Hue Silk Apache Cassandra SEMAGROW ScyllaDB LIMES Apache Hive 4Store Postgis OpenLink Virtuoso

10 Platform features 10 BDE Development Environment o Stack builder o Workflow builder o Instructions to add custom components Administrative Interface o SwarmUI o Logger Interface UI Integrator o Workflow monitor o Integrated web interface

11 BDE Integrator UI-WorkFlow 11 Git-clone Stackbuilder Select components => (Push Create-Flow) WorkFlow builder Arrange Components => (Push Monitor) New Stack WorkMonitor Deployment status of Components => (Push OK) SwarmUI BDE Logger See the scaling and scale up/down Navigate the componentui and deploy jobs BDE-IDE Integrator UI

12 Stack Builder 12

13 Stack Editor 13 Component Services/dockers

14 BDE Workflow Builder 14 Component 1 Component 2 Component 3

15 BDE Workflow Monitor 15 Component 1 Finished Component 2 Finished Component 3 Inprogress

16 Swarm UI-Pipeline 16 Increase number of instances

17 Monitor 17

18 Integrator UI 18 Component 1 Component 2

19 19 st booth-10, 1 Floor

20 Maintenance and Uptake 20 Open-Source, Community Driven o Commitment from core BDE consortium team o Independent BD components maintenance o Platform Maintenance driven by BDI users Adopters o Feuga, Eurostat, ILVO, I2cat, Vicomtech, IoF,... Follow Up Projects o HOBBIT, Special, BigDataOcean, Qrowd, BETTER,

21 BDE vs Hadoop distributions 21 Hortonworks Cloudera MapR Bigtop BDE File System HDFS HDFS NFS HDFS HDFS Installation Native Native Native Native lightweight virtualization Flexible Modular Architecture no no no no yes High Availability Single failure recovery (yarn) Single failure recovery (yarn) Self healing, mult. failure rec. Single failure recovery (yarn) Failure recovery Cost Commercial Commercial Commercial Free Free Scaling Freemium Freemium Freemium Free Free Addition of custom components Not easy No No No Yes Integration testing yes yes yes yes -- Operating systems Linux Linux Linux Linux Windows/Mac/Linux Management tool Ambari Cloudera manager MapR Control system - Docker swarm + Custom UI

22 SANSA Scalable Semantic Analytics Stack 22

23 SANSA: Vision

24 SANSA Layers

25 RDF to Tensors

26 Machine Learning Layer

27 Interactive SANSA in Browser

28 SANSA

29 Semantic Data Lake 29 Data Lake o Repository of data collected in its original formats o Structured, semi-structured, unstructured o Schema-less Semantic Data Lake o Add a Semantic Layer on top of source datasets The data is semantically lifted using ontologies Provide a uniform view over nonuniform data

30 Semantic Data Lake 30 Execution Plan Decomposing User Query SPARQL query Metadata property -> data source (type) SQL?item gho:country?country.?item gho:disease?disease.... MongoDB Finding Relevant Data Sources + Queries Translation SELECT country, disease,... FROM Observations SQL SQL Database XPath XML File XML JSON Path MongoDB

31 31 BDI on Github: Technical Questions Project Website: Thank you! SANSA Stack

32 Paris, EBDVF 22nd November 2017

33 The mobility use case in Thessaloniki Multisource datasets (speed, traffic flow, travel time) are being used in Thessaloniki for the provision of traffic status short-term prediction based on mobility/traffic patterns recognition. Integration of machine learning techniques using the travel times, traffic counts and speeds as well as the correlations of traffic speed, to train an appropriate Neural Network Model for efficient and robust traffic speed prediction.

34 The datasets Floating Car Data o o o speed measurements per minute Location, speed, orientation, status Hundreds of Gb (historical dataset)

35 Mobility services in Thessaloniki

36 Mobility services in Thessaloniki

37 Mobility services in Thessaloniki

38 Mobility services in Thessaloniki

39 Mobility services in Thessaloniki TrafficThess ( o Visual representation of the current as well as past speeds in Thessaloniki, Greece o notifications Historical raw data export per link in open format o TrafficPaths ( o Descriptive information of the current travel times wherever available (Thessaloniki, Patra, Irakleon, Serres, Kavala) o Mobile friendly web page TrafficThess Reports ( o Visual and descriptive representation of the current traffic conditions (speeds & travel times) on the main roads of Thessaloniki, Greece o Highly customizable notifications

40 Mobility services in Thessaloniki TrafficThess ( Reliable traffic conditions monitoring on a 24/7/365 basis Traffic conditions in the city of Thessaloniki, Greece during snowfall on 10 & 11 Jan 2017: (credits to anmpout for helping out with the video) Keep calm! It s just another congestion on the ring road

41 Mobility services in Thessaloniki

42 Mobility services in Thessaloniki TrafficPaths ( Calculation of travel times on a 24/7/365 basis

43 Mobility services in Thessaloniki TrafficThess Reports ( A personalized single point of access

44 Mobility services in Thessaloniki Datatank (Back office + restapis) CKAN (front end)

45 DR. JOSEP MARIA SALANOVA GRAU Paris, EBDVF 22nd November 2017