CHALLENGES AND DEVELOPMENTS FOR A DATA CENTRIC AND MICROSERVICE BASED INFRASTRUCTURE FOR BIG DATA AND AI

Transcription

1 CHALLENGES AND DEVELOPMENTS FOR A DATA CENTRIC AND MICROSERVICE BASED INFRASTRUCTURE FOR BIG DATA AND AI Casper van den Broek, casper.vandenbroek@tno.nl NATO IST 160 specialists meeting,

2 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

3 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

4 THE MILITARY BIG DATA WORLD

5 SURVIVAL OF THE FASTEST Situational awareness Situational understanding

6 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

7 FROM APPLICATION CENTRIC TO DATA CENTRIC

8 DATA CENTRIC APPROACH Advantages Interfacing with a relatively low number of standardized data stores interfaces instead of interfacing with many application specific API s Limits the number of interfaces in a big-data world Makes it easier to share data between different parties In the Mobile-Dismounted-on-foot conditions it is easier to determine where the data should reside in order to have it available when necessary It is no longer bound to an application Easier to synchronize

9 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

10 MICROSERVICES SOA done properly? Do one thing and do it properly

11 MULTIPLE PARTIES mqtt-toelasticsearch facebookscanner locationestimator sentimentestimator mqtt KIXS twitterscanner elasticsearch frontend

12 MULTIPLE LANGUAGES mqtt-toelasticsearch facebookscanner locationestimator sentimentestimator mqtt twitterscanner elasticsearch frontend

13 CONFIGURABLE, ADAPTABLE, REPLACEABLE mqtt-toelasticsearch facebookscanner locationestimator sentimentestimator mqtt twitterscanner elasticsearch frontend

14 STATELESS MICROSERVICES: SCALABILITY Multiplier microservice Input: 3 * 3 Output: 9 Stateless makes it easy to scale up when needed. Load Balancer

15 BRINGING CODE TO THE DATA

16 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

17 CONTAINERIZATION Containers provide isolation, but share the same OS and kernel. Virtual machine This results in faster deployments, easiermigration, lessoverhead and faster restarts. Container

18 CONTAINERIZATION Traditional distribution Container Linux Your containers Required software only

19 BRINGING CODE TO THE DATA

20 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

21 CHALLENGES: APPLICABLE OVER STATIC AND TACTICAL CONDITIONS Limited Bandwidth Processing Storage

22 Processing Machine input Processing Machine input Storage Machine output Storage Machine output External comms Internal comms (In vehicle network, Man Packed Network Human input External comms Human input Human output Internal comms (In vehicle network, Man Packed Network Human output Processing Machine input Storage Machine output External comms Human input Coördination Coördination Coördination Internal comms (In vehicle network, Man Packed Network Human output SOLUTION DIRECTION: RAINDROP-TO-CLOUD Maximizes IT resources Requires synchronisation and service execution orchestration Processing Storage External comms Coördination Internal comms (In vehicle network, Man Packed Network Machine input Machine output Human input Human output

23 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

24 CHALLENGES: INTEROPERABILITY IN A NATO MISSION CONTEXT

25 CHALLENGES: INTEROPERABILITY IN A NATO MISSION CONTEXT

26 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

27 CHALLENGES: COMPLYING TO NATO

28 CONTENTS The military big data world Three IT trends for Big Data and AI Data centric Microservices Containerization Challenges in developing a data centric/microservices architecture Applicable in static and tactical conditions Interoperability in a NATO context Complying to NATO Solution direction

29 SOLUTION DIRECTION C2 applications (User facing services) Sensors Actuators Coordination services Processing services Disclosure services Representation services Transport services Human Input services Machine Input/ output services Synchronisation services Data security services Storage services

30 WORK IN PROGRESS - QUESTIONS

31 CONTACT INFO Casper van den Broek, Johan van der Geest, Harrie Bastiaansen, Mike Schenk,