News / Outlook / Visions

Transcription

1 News / Outlook / Visions Performance und Scalability Ralf Nörenberg Director

2 Performance und Scalability Topics 1. Sketching Tomorrow 2. Big Data Project A: ODS as a Master 3. Big Data Project B: ODS as a Slave 4. Expectations and Combination of Results 2

3 Todays ODS Infrastructure and Architecture View Application ODS Server Physical Storage 3

4 Today s ODS What are the questions? Give me all measurements of vehicle Ford, type Focus, year 2015 and engine 3.0. Meta data result based on a meta data query. 4

5 Tomorrow s ODS What are the questions? Give me all <vehicles, sensors, > of vehicle Ford, type Focus, year 2015 and engine 3.0 where Fuel Consumption > 10 liters where Speed Peak > 150 km/h where Avg. Temp > 35 C where Diesel Pipe Burning = yes Meta data result based on a combined meta data & mass data query. 5

6 Tomorrow s ODS Sketching the Future Applications Big Data Query (Correlations) Query Interface ODS Server Analysis Server Reduction / Filtering of Information Routing of Information Big Data Technology Adapter A Cache - Level Meta-Data Big Data Technology Adapter B Cache - Level Mass-Data 6

7 Tomorrow s ODS Applications Applications have the benefit of predefined questions: boundaries to limit possible questions / queries that can be asked Streamlining of information flow provision of pre-cached result sets Provision of pre-configured information about mass-data 7

8 Tomorrow s ODS Applications Applications have the benefit of predefined questions: boundaries to limit possible questions / queries that can be asked Streamlining of information flow provision of pre-cached result sets provision of pre-configured information about mass-data Give me all <vehicles, sensors, > of vehicle Ford, type Focus, year 2015 and engine 3.0 where Fuel Consumption > 10 liters where Speed Peak > 150 km/h where Avg. Temp > 35 C where Diesel Pipe Burning = yes = AVG ODS = MAX ODS = AVG ODS = pre-configured ODS No Big Data required. 8

9 Tomorrow s ODS What are the questions? Give me all <vehicles, sensors, > of vehicle Ford, type Focus, year 2015 and engine 3.0 where Fuel Consumption > 10 liters where Speed Peak > 150 km/h where Avg. Temp > 35 C where Diesel Pipe Burning = yes in continuous time window of 30 min within measurement We need new technologies ( big data )! 9

10 Tomorrow s ODS Big Data Queries Applications may but correlation queries surely require new technologies for Analysis of mass data Return of result sets For random queries, there are consequences on the solution architecture Streamlining of information flow is not possible How many experts are using the system? How often are the same / likewise queries used? How will the provision of cached information work (for performance)? 10

11 Tomorrow s ODS Sketching the Future Customer Requirements / Solution Specification Applications Big Data Query (Correlations) Query Interface open Customer Requirements / Solution Specification ODS Server Analysis Server Reduction / Filtering of Information Routing of Information Big Data Technology Adapter A Big Data Technology Adapter B Being smart is key! / Discussion currently focus on the very last part of the process Cache - Level Meta-Data Cache - Level Mass-Data 11

12 Tomorrow s ODS Sketching the Future Applications Big Data Query (Correlations) Query Interface HQL Implementation available ODS Server Analysis Server Reduction / Filtering of Information Routing of Information Big Data Technology Adapter A Cache - Level Meta-Data Big Data Technology Adapter B Cache - Level Mass-Data 12

13 Tomorrow s ODS Sketching the Future Applications Big Data Query (Correlations) Query Interface Scalability available / Avalon Distributor ODS Server Analysis Server Reduction / Filtering of Information Routing of Information Big Data Technology Adapter A Cache - Level Meta-Data Big Data Technology Adapter B Cache - Level Mass-Data 13

14 Tomorrow s ODS Sketching the Future Applications ODS Server Reduction / Filtering of Information Big Data Query (Correlations) Query Interface Analysis Server Merlin 2G Implementation Available / Architecture Design analogous to Avalon Scalability (to be shown) Routing of Information Big Data Technology Adapter A Cache - Level Meta-Data Big Data Technology Adapter B Cache - Level Mass-Data 14

15 Tomorrow s ODS Sketching the Future Applications Big Data Query (Correlations) Query Interface ODS Server Analysis Server Reduction / Filtering of Information Routing of Information Big Data Technology Adapter A Cache - Level Big Data Technology Adapter B Cache - Level Big Data Research Meta-Data Mass-Data 15

16 Performance und Scalability Topics 1. Sketching Tomorrow 2. Big Data Project A: ODS as a Master 3. Big Data Project B: ODS as a Slave 4. Expectations and Combination of Results 16

17 Big Data Project A: ODS as a Master Project Facts Project started April 2015 and is running Our partner is in Ingolstadt (~ employees Tier 1 supplier) Why partner up? a big partner is required for provision of scalable work environments ODS technology and customer IT infrastructure are regarded to still be existent (now: Avalon and Oracle) there is an existing challenge with measurements of big size (~50GB each) there is an existing big data cluster running 17

18 Big Data Project A: ODS as a Master Objectives 1) Performance analysis of regular ODS system with big data use cases (bottlenecks?) 2) Identification of BIG ODS system designs and architectures 3) Performance analysis of BIG ODS openmdm application data access Avalon Server data management Merlin Server integration / analysis Frequent Updates within ASAM US Big Data Workshop Working Prototype for ASAM Big Data Conference 18

19 Big Data Project A: ODS as a Master Planned Steps 1) Setting up a SPARK -Cluster and writing mass-data into it (partly done) 2) Retrieving channel data from cluster (driver development) (working on) 3) Performance Analysis considering huge tables / Solution Identification (to start soon) 4) Integration of measurement meta data into SPARK (purple elements) (planned) Solve the challenge of Joints between Oracle and SPARK 5) Retrieval data out of BIG ODS system (planned) 19

20 Big Data Project A: ODS as a Master System Architecture Spark Cloud / Cluster Files 1 Hadoop SQL XY 20

21 Big Data Project A: ODS as a Master System Architecture Avalon Server Spark Cloud / Cluster 2 Driver 3 5 Master / Worker JOB 4 Oracle Files 4 Driver submits orders (to be generic) Jobs accepts orders (specific to physical storage and ODS data model) Master / Worker enable physical storage access 21

22 Big Data Project A: ODS as a Master Sustantiated Big Data Statements Spark Cloud / Cluster Master / Worker Files JOB JOB Avalon Server Driver Oracle Technology Boundary: ODS Server needs to run outside of cluster Scalability: Multiple (local) clusters may run next to data generators and enable scalability Confirmation: ODS Server / Oracle remain as structuring entities 22

23 Performance und Scalability Topics 1. Sketching Tomorrow 2. Big Data Project A: ODS as a Master 3. Big Data Project B: ODS as a Slave 4. Expectations and Combination of Results 23

24 Big Data Project A: ODS as a Slave Project Facts Project started April 2015 and is running Our partner is Why partner up? a big partner is required for provision of a scalable middle-ware solution Know-how of integration systems to middle-ware Objectives General Case Study and Prototype Specific Project Realization 24

25 Big Data Project A: ODS as a Slave Today s Setup Third Party Analysis Tool MoMa Avalon Server + Physical Storage 25

26 Big Data Project A: ODS as a Slave Introduction of a Middleware Log / Admin Third Party Analysis Tool BUS - Middleware MoMa Avalon Server + Physical Storage 26

27 Big Data Project A: ODS as a Slave Use-Case A: Connecting ODS and ODS Third Party Analysis Tool Combined ODS Web BUS - Middleware Avalon Server + Physical Storage Avalon Server + Physical Storage 27

28 Big Data Project A: ODS as a Slave Use-Case B: Connecting ODS and None-ODS Third Party Analysis Tool None-ODS Third Party Tool BUS - Middleware Avalon Server + Physical Storage None-ODS 28

29 Big Data Project A: ODS as a Slave Use-Case C1: Scalability of ODS Systems by Outsourcing ( Copy ) Third Party Analysis Tool Combined ODS Web BUS - Middleware Avalon Server + Physical Storage Avalon Server + Physical Storage Avalon Server + Physical Storage None-ODS Avalon Server + Physical Storage 29

30 Big Data Project A: ODS as a Slave Use-Case C2: Scalability of ODS Systems by Outsourcing ( Move ) Third Party Analysis Tool Tool XY BUS - Middleware Avalon Server + Physical Storage Avalon Server + Physical Storage Avalon Server + Physical Storage 30

31 Big Data Project A: ODS as a Slave Use-Case D: Connecting Locations Third Party Analysis Tool Tool XY Third Party Analysis Tool Combined ODS Web Tool XY BUS - Middleware Avalon Server + Physical Storage None-ODS 31

32 Big Data Project A: ODS as a Slave Use-Case E: Scalability / Decentralization of indifferent Set-Ups Third Party Analysis Tool Tool XY Third Party Analysis Tool Combined ODS Web Tool XY BUS - Middleware Avalon Server + Physical Storage Avalon Server + Physical Storage Avalon Server + Physical Storage None-ODS Avalon Server + Physical Storage 32

33 Performance und Scalability Topics 1. Sketching Tomorrow 2. Big Data Project A: ODS as a Master 3. Big Data Project B: ODS as a Slave 4. Expectations and Combination of Results 33

34 Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like SPARK are sustainable, the actual physical storage is of secondary interest Both project designs (master & slave) supplement each other Realization of successful BIG ODS systems is realistic This presentation shall encourage discussions throughout the day! Many topics of the overall picture will be discussed and presented today! Our outlook is optimistic! 34