Problem Code: #ISR13. College Code :

Transcription

1 Ministry Category : Indian Space Research Organisation (ISRO) Problem statement : Detect security breaches or anomalies in Network traffic using Network log analysis Team Leader Name : Vipin George Problem Code: #ISR13 College Code : To alert security breaches or anomalies in network traffic in a timely and effective manner using network log analysis by leveraging the capabilities of Data Mining on Big Data, and Cloud Computing. We model the normal behavior of a system by analyzing its role and behavior over time. Challenges Existing tools are limited to a pre-defined set of rules gathered from experts An event that is not in the rule base will not be flagged A closer look at the data indicates that The Network logs are generated all the time that the network is operational (Volume) Network logs can be huge. Billions of lines might accumulate into a log file in a day (Velocity) The entries have several fields or variables, depends on network protocol. Each variable has a domain of possible values. (Variety) Network data analysis is a constantly repeated task. Traditional data processing services are in batch mode, and therefore cannot adapt to evolving network data streams in real time, resulting in increased latency for detecting a security breach An attacker might have done enough damages by the time it is detected using traditional approach. Proposed solution, data stream mining techniques reliably detect a security breach reducing the time to detect it.

2 OUR SOLUTION Generates alert/alarm about security breach in near real-time in a responsive manner Complexity of analyzing a high dimensional data is managed by BigData(3Vs) 3Vs (Volume, Variety and Velocity) are three defining properties or dimensions of Big Data. We are leveraging the capabilities of Data Mining on Big Data, and Cloud Computing to handle this huge amount of data. Network logs contain a lot of redundant signs of normal behavior and prevented incidents. If something anomalous happens, it is manifested in between normal entries. By analyzing a known entry sequence pattern in the log we can detect an attack in near real-time During time of attack, normalized entropy values will deviate from its normal behavior, and that is used for detecting the anomaly in the traffic. Threshold is predefined for entropy values to detect deviation from normal behavior i.e. security breach. Data mining helps to find advanced attackers where predefined methods are not available. By combining both static and dynamic methods this Hybrid system can improve detection of an anomaly. Identification by localization can help to predict or evaluate the impact of the breach Whenever the log indicates that the network is deviating from the normal behavior, it should raise an alarm. Depending on the size of the network there can be different levels of alarms: Critical, Severe, Substantial, Moderate and Low Our approach uses Dynamic rule creation enables us to detect new types of breaches without further human intervention. Apache Hadoop(Open Source) technique enables processing of large data sets in a parallel way (Map-Reduce method) Hadoop-based algorithm processes data faster compared to algorithm using tree-based structure Data mining uses statistical and machine learning algorithms on a set of features derived from standard and non-standard behavior to detect unforeseen patterns or properties. It consists of two phases: data collection and application of detection method on collected data.

3 A Hybrid approach is proposed to combine the benefits of entropy and SVM based techniques. Our proposed solution can use contemporary Cloud Computing to speed up the detection of an anomaly or security breach. It can sustain and adapt to discover upcoming real world anomalies as and when they arrive. Can detect slow-paced attacks as Big Data could accommodate longer "connection window" features Technology Stack (completely relying on Open Source software) OS: Red Hat Enterprise Linux/Ubuntu/Cent OS Apache Hadoop To process data in near real-time we use Open Source tools Kafka, Logstash, ElasticSearch and Kibana Apache CloudStack Tcpdump and libpcap library Use Case Latency in traditional approach is high

4 Use Case Classifying Network Traffic on various parameters Rule Creation

5 Dependencies/ Show stopper We need to get access permissions on Network to extract logs Normal profile of the terminals needs to be defined initially by analyzing its role. Adaptive knowledge discovery solution is best suited for current day security breaches Once this system is implemented, it can positively identify victims on terminals who are frequent on Click-bait news so that targeted training can be done to ensure cyber hygiene Can collaborate with other systems to share the behavior of a terminal. With Cloud Computing we can have a very reliable system designed which can scale up or down with minimal additional cost. Access to a private cloud will improve system performance and security, which is of paramount importance. By using Data Analytics we can gain finer insights about the type and nature of security breach Economically feasible as we rely on enterprise-grade FOSS software with community support, security and stability updates

6 Our Team Members Vipin George (Team Lead) with 7 years experience in various Infrastructure Management applications including Network Monitoring application deployed and maintained for a Tier 1 ISP, Web Application projects, Infrastructure Security. An M.Tech Final year student in Cyber Forensics and Information Security Field of interest : Cyber Security, Security Analytics, IoT Security and Data recovery. Aswathi K - M.Tech Final year student in Cyber Forensics and Information Security Field of interest : Embedded systems, Networking and Cloud Security Christina Rajan Thomas - M.Tech Final year student in Cyber Forensics and Information Security. Fields of interest: Web designing, Cloud Security, Database Management Systems, Programming and Data Mining Jishnu Raj T - B.Tech Final year student in Computer Science. Field of interest Embedded Hardware, Networking, and Web programming Roshan Oommen Reji - B.Tech Second year student in Computer Science with interests in Programming, Data processing and Entrepreneurship Sreenath K Sahadevan - B.Tech Final year student in Computer Science. Field of interest: Web, Database and Android programming Arunkumar G (Media promoter) B.Tech Second year student in Computer Science Field of interest: Web programming and popular design software.