Centinel: Streaming Data Handler. April 20 th, 2016

Transcription

1 Centinel: Streaming Data Handler April 20 th,

2 An Overview 2

3 Centinel: Streaming Data Handler Aim of Centinel is to asssist operators by providing visibility into OpenDaylight management software defined network ecosystem (network via plugins, apps and infrastructure). To accomplish this, Centinel Boron aims at a multi phased approach that includes: Data Collection: IPFIX, OpenFlow( from ODL-OF plugin) and SNMP(from ODL-SNMP plugin) Data Analysis: Using Kafka, Storm/Spark, Flume(replace Graylog) Analytics Dashboard: Inter-op with DLUX Framework and ensure UX/UI (Graphana/Kibana) Machine Learning: leverage Apache Mahout/ SparkMlib to enable machine learning based on above analysis. Additinoaly from usage-deployement, Embedded installation is proposed(support from embedded Flume agent, Support for embedded installation of query engine) Framework enhancement : Enhance Centinel architechture to have two branches. A: Batch Processing System(BPS) :BPS comprises of Flume and hbase B: RealTime analytics system(ras) : RAS comprises of Apache Storm(Embedded and distributed mode) for real time prediction. In embed mode, Storm will be used instead of Graylog server. In distributed mode, user will have both the options for log analysis. 3

4 Centinel: High Level Architecture 4

5 Centinel: Project links Project Main page Project Proposal Release plan Project archives Repository 5

6 Design Approach Incremental updates on design document 6

7 DLU X BORON DDoS Protection OpenSt ack Neutron Lege DOCSIS: Data Over Cable Service PCEP: Path Computation Element AAA: Authentication, Authorization & Interface Specification Protocol PCMM: Packet Cable Accounting nd ALTO: Application Layer FRM: Forwarding Rules Manager MultiMedia Plugin2OC: Plugin To Traffic Optimization AuthN: Authentication BGP: Border Gateway Protocol GBP: Group Based Policy IoTDM: Internet of Things Data Broker OpenContrail SDNI: SDN Interface (Cross-Controller Federation) CAPWAP: Control and Provisioning of LACP: LISP: Locator/Identifier Link Aggregation Separation Control Protocol SFC: Service SNMP: Function Simple Chaining Network Wireless Access Points Protocol MAPLE: Management Maple Protocol SNBI: SXP: Source-Group Secure Network Tag Bootstrapping exchange COPS: Common Open Policy Service Programming Infrastructure Protocol DIDM: Device Identification and Driver management NIC: Network DLUX: OpenDaylight User OVSDB: Open vswitch DataBase Experience Intent Proposal DDoS: Distributed Denial Of Service Protocol OPFLEX: Extensible Policy Protocol SDNI Wrapper AAA- AuthN Filter OpenDaylight APIs (REST) VTN Coordinator TSDR: Time Series Data Repository TTP: Table Type Patterns USC: Unified Secure Channel VTN: Virtual Tenant Network Network Applications Orchestrations and Services ALT O DID M MAPL E GBP Servi ce DOCSIS Abstract ion SF C BG P PCE P MD-SAL / Yangtools SDNI Aggrega tor Plugin2 0C L2 Swit ch TCP - MD 5 VTN Manag er OVSD B Neutr on Servic e LISP Servi ce NI C Reservat ion USC Manager Persiste nce VPN Service LAC P TSD R OpenFl ow NETCO NF SN BI PCMM/ C OPS Plugin2 0C SNM P OVS DB LIS P Io T US C OPFL EX OpenFlow Enabled Devices Open vswitches Additional Virtual & Physical Devices 7

8 Centinel: IPFIX Data Collector IPFIX is the technology that aloows for the capture and analyzing of flow information from layer-3 switches, routers and firewalls to a central IPFIX Collector Software on your network. IPFIX differs from NetFlow in such a way that it allows for different types of information to be captured from the device, including proprietary information that a vendor ID can integrate into the device. Also has the ability to receive information about Syslog and SNMP that would otherwise be sent to those respective services/collectors. 8

9 Flow Diagram The figure alongside shows the data packet flow: The data packets from OVS are passed to Centinel. The collector in Centinel collects the data packets. The data is then parsed into Json Objects which are stored into RealTime Processing system and Batch Processing Engine (HBASE). 9

10 Sequence Diagram The figure alongside shows the sequence diagram for IPFIX Collector. It describes the flow for how the packets of IPFIX are collected, parsed and stored in database. 10

11 Class Diagram The figure alongside shows the class diagram of the IPFIX Collector. It defines the following java classes for handling the flow packets. IpfixCollector class to create a socket connection to a port and collecting packets. IpfixProcessor inner class to get packet data and send it to IpfixPacketParser class. IpfixPacketParser class to parse the packet data and get all the attributes from the packet. PersistEvent class to send a Json Object of data packet to database. 11

12 Centinel: OpenFlow Data Collector OpenFlow is an open standard used to centrally manage and affect packet forwarding decisions in network devices such as routers or switches. OpenFlow controls how packets are forwarded through network switches. OpenFlow is a protocol used in Softwarre Defined Networks for communication between a network device and a controller server. This particular architecture seperates packet forwarding and high level routing decisions whereby packet forwarding is performed by the switch hardware and high level routing decisions are made by a controller server. 12

13 Flow Diagram The figure alongside shows the data packet flow. The data packets from OVS are passed to the ODL Controller. The MD-SAL Container in the Controller receives the incoming data packets and passes them onto the Open Flow Collector. The data is the parsed into a Json Object and stored to RealTime Processing system as well as the Batch Processing Engine (HBASE). 13

14 Sequence Diagram The figure alongside shows the sequence diagram for OpenFlow Collector. It describes the flow for how the packets of OpenFlow are collected, parsed and stored in databse 14

15 Class Diagram The figure alongside shows the class diagram of OpenFlow Collector. It defines the following java classes for handling the flow packets. OpenFlowCollector class to create a socket connection to a port and collecting packets. OpenFlowProcessor inner class to get packet data and send it to OpenFlowPacketParser class. OpenFlowPacketParser class to parse the packet data and get all the attributes from a packet. PersistEvent class to send a JsonObject of data packet to database. 15

16 Thank You 16