Thales PunchPlatform Agenda

Transcription

1 Thales PunchPlatform Agenda What It Does Building Blocks PunchPlatform team Deployment & Operations Typical Setups Customers and Use Cases RoadMap 1

2 What It Does Compose Arbitrary Industrial Data Processing Channels Data logs, documents, events, Alerting Batch Processing Y Y Y Y Y Searching Visualising Reporting Data logs, documents, events, Speed Processing Y 2

3 Building Blocks DataFlow Collect and Transport Your Data Multi-nodes, -racks, -rooms, -sites Scalable, Resilient, Reliable DF DP DataProcessing Parse, Normalize, Enrich Store, Archive, Index Detect, Raise Alarms Search Reprocess, Replay DataVisualization Create your Dashboards to dig months/years of data Pick what suits best your needs: Kibana, Grafana, Zeppelin Data Extraction and Reporting Multi-Tenant, Secure DV DA DataAnalytics Plug in Arbitrary Processing Storm, Spark-Streaming, Flink Join the Thales Big Data and Analytics Community 3

4 Understanding the (CyberSecurity) Data Pipeline Train/Learn Collect Transport Parse Normalise Enrich Detect Search Report Monitor Correlate DF DP DV DA 4

5 DF DV DP DA Log Management Typical Setup Data Processing Raw Data Parsing Normalising Alerting Indexing Kafka Storm Kafka Storm ElasticSearch Searching Visualising Reporting Alerting Data Analytics: Machine Learning Anomaly Detection 5

6 DF DV DP DA Log Management + Analytics Typical Setup Data Processing Raw Data Parsing Normalising Alerting Indexing Kafka Storm Kafka Storm ElasticSearch Searching Visualising Reporting Alerting Data Analytics: Machine Learning Anomaly Detection 6

7 7 Architecture

8 Stream or Batch Processing Architecture In a nutshell : data processing is designed as follows disk batch/bulk processing acknowledged idempotent exactly once disk if failure then replay This requires : partition identifiers, timestamping, unique identifiers, batch identifiers, smart kafka offset handling, idempotent bulk file writing, on the fly efficient zero-copy compression, on the fly ciphering and of course real time supervision in a way manageable by the user. That is what the PunchPlatform provides. 8

9 Deployment & Operations 9

10 PunchPlatform Benefits : Start from empty servers What you need are plain linux servers, with local disks only. Use dedicated hardware, VMs, Clouds. Whatever. If you build your own infrastructure use the PunchPlatform infrastructure ansible libraries. Otherwise use Amazon, OpenStack, or any Iaas tool you have. 10

11 Deploy your services in minutes Describe your setup in a configuration file. Use the PunchPlatform deployer to set it all. Kafka Storm ElasticSearch Kibana PunchPlatform Deployer Grafana Kafka Spark Platform configuration file Admin Ceph 11

12 Deploy your business logic in minutes Describe your data channel in a configuration file. Use the PunchPlatform channel command to set it all. It can be a log parsing pipeline, a scada metric pipeline, whatever. PunchPlatform Channel Channel configuration file 12

13 Resize Your Platform in minutes Give more power to you platform, to scale up to you needs. Do that without service interruption. PunchPlatform Deployer Platform configuration file 13

14 Resize you Processing in seconds Give more power to you data channel, to scale up to you needs. Do that without service interruption. PunchPlatform Channel Channel configuration file 14

15 Real Example Setups 15

16 Connecting Your Sites CyberSecurity Platforms : Holland, France, Honk Kong Boston Paris Sydney Singapore dns/ldap/etc storm zookeeper kafka elasticsearch ceph 16

17 Multi Sites Replication CyberSecurity & Supervision Platform : Toulouse, Thales Avionics Data logs Git 7 Italy Git France Git Germany dns/ldap/etc storm zookeeper kafka elasticsearch ceph 17

18 Small Scale Deployment Transportation Monitoring Systems : Toronto 7 commit/push pull Git pull dns/ldap/etc storm zookeeper kafka elasticsearch 18

19 Customers and Use Cases Data Agnostic :. Logs, XML, Json, text. Snmp traps, Netflow. metrics, time series IoTalk TDS Elec. War. Pmu Urs Open Platform : powered by. ElasticSearch. Cassandra. OpenTsdb. CEPH. Storm. Spark. Kibana/Grafana/Zeppelin ESA TGS industry military infrastructures monitoring supervision cyber security ThalesIT Soc.nl Wide Range of Business Cases :. CyberSecurity. Industry 4.0 : manufacturing, transportation, energy. Monitoring : IT infrastructures Soc.fr Soc.hk 19

20 Summary & Roadmap 20

21 PunchPlatform Stack Elastic Stack all of it Parsers and Channels Simple, straight, industrial Monitoring end to end Stream Processing Takes care of simple but key stream processing Deployer/Updater Install, Update. Fully documented. Machine Learning By configuration or by coding Archiving Long term storage. Secured 21

22 2017 RoadMap Platform CyberSecurity Supervision Industry Distributed Archiving Service Platform Resource Manager Spark Distributed Runtime ElasticSearch Batch Connectors Spark Connector APIs Elastic 5 Platform Update Storm/Kafka/Spark Update IOC offline and online search CyberSecurity Rule Engine CyberSecurity Standard Rules MachineLearning Modules Equipment Parsers Supervision/Monitoring Rule Engine OpenTsdb/Cassandra Supervision/Monitoring Standard Rules Metric Based Alerting Cassandra SparkConnector APIs jan jun dec 22

23 2018 RoadMap Platform CyberSecurity Spark Distributed Runtime Elastic 6 (7 8..) Platform Update IOC/Replay/Extraction Kibana Plugins Equipment Parsers MachineLearning jan jun dec 23

24 Thanks!