Apply Graph and Deep Learning to Recommendation and Network Intrusion Detection

Transcription

1 Apply Graph and Deep Learning to Recommendation and Network Intrusion Detection Zhe Wu, Ph.D. Architect Oracle Spatial and Graph June 22, 2017

2 Outline Introduction and overview of graph technologies Architecture of Oracle s Support Use cases: - Recommender System - Network Intrusion Detection Summary 2

3 Graph Data Model What is a graph? Data model representing entities as vertices and relationships as edges Optionally including attributes Also known as linked data What are typical graphs? C A F B D Social Networks LinkedIn, Facebook, Google+, Twitter,... Physical networks, Supplier networks,... Knowledge Graphs Apple SIRI, Google Knowledge Graph,... E 3

4 Graph Data Model Why are graphs popular? Easy data modeling whiteboard friendly Flexible data model No predefined schema, easily extensible Particularly useful for sparse data Insight from graphical representation Intuitive visualization Enabling new kinds of analysis C A F B E D 4

5 Architecture of Existing Support Graph Analytics Data Access Layer Parallel In-Memory Graph Analytics (PGX) Java APIs Apache Blueprints & Lucene/SolrCloud REST/Web Service Java, Groovy, Python, Oracle Spatial and Graph Oracle Database 12.2 Java APIs/JDBC/SQL/PLSQL Oracle Big Data Spatial and Graph Apache HBase Oracle NoSQL Database Property graph formats supported RDF (RDF/XML, N- GraphML Triples, N-Quads, GML TriG,N3,JSON) Graph-SON Flat Files CSV Relational Data Sources 5

6 Oracle Differentiators -- Graph Complete, Supported, Graph Solution: Storage: NoSQL, Hbase, RDBMS back-ends Data Access: Blueprints, Java, Query Language (PGQL) Rich Graph Analytics: 40 pre-built, in-memory graph algorithms Scalable: Analyze billion edge graph in memory on single BDA node Persist extremely large graphs on disk Security: Secure NoSQL, Kerberos CDH, Label Security on Oracle Database 12c 10-50x Faster than graph analysis competitors

7 The Data Model name= marko age = 29 weight= knows 2 8 name = vadas age = 27 9 weight=0.4 created weight=1.0 knows name= lop lang = java created 5 3 created weight=0.4 weight=1.0 name= ripple lang = java name= josh age = 32 created weight= name= peter age = 35 A set of vertices (or nodes) each vertex has a unique identifier. each vertex has a set of in/out edges. each vertex has a collection of key-value properties. A set of edges (or links) each edge has a unique identifier. each edge has a head/tail vertex. each edge has a label denoting type of relationship between two vertices. each edge has a collection of key-value properties. 7

8 Categories of Graph Wordload Computational Graph Analytics Compute values on vertices and edges Traversing graph or iterating over graph (usually repeatedly) Procedural logic Examples: Shortest Path, PageRank, Weakly Connected Components, Centrality,... Graph Pattern Matching Based on description of pattern Find all matching sub-graphs :Person{100} name = Amber age = 25 :friendof {2513} since = 08/01/2014 :Person{200} name = Paul age = 30 :worksat{1831} startdate = 09/01/2015 :friendof{1173} :knows{2200} :Company{777} name = Oracle location = Redwood City :Person{300} name = Heather age = 27 8

9 Examples for Graph Analysis Community detection and influencer analysis Churn risk analysis/targeted marketing, HR Turnover analysis Product recommendation Collaborative filtering, clustering Anomaly detection Social Network Analysis (spam detection), fraud detection in healthcare Path analysis and reachability Outage analysis in utilities networks, vulnerability analysis in IP networks, Panama Papers Pattern matching Tax fraud detection, data extraction 9

10 Build Recommender System with Graph Technologies

11 Building a Recommender System -- with Oracle Big Data Spatial and Graph Environment Oracle Big Data Lite VM Oracle Big Data Spatial and Graph v SolrCloud 4.10.x A user-item property graph Vertices (items, descriptions, and users) Edges (linking users and items) Recommendation: you may also like

12 Building a Recommender System -- with Oracle Big Data Spatial and Graph BDSG offers multiple approaches and they can be mixed together Content-based filtering Match item description Match user profile Relevancy ranking Collaborative filtering People liked similar items in the past will like similar items in the future Personalized Page Ranking Randomly navigate from a user to a product, then back to a user, Randomly jump to starting point(s) A B C u v w x A u u B B w w C

13 Personalized Page Rank-based Recommender System Random walk with restart Reference:

14 Key API for Personalized Page Rank API: ppr=analyst.personalizedpagerank ( pgxgraph, vertexset, /*max error*/, 0.85 /*damping factor*/, 1000 ); Result: ppr.gettopkvalues() it=ppr.gettopkvalues(9).iterator(); while (it.hasnext()) { entry=it.next(); vid=entry.getkey().getid(); System.out.format("ppr=%.4f vertex=%s\n", entry.getvalue(), opg.getvertex(vid));} ppr= vertex=vertex ID 1 {name:str:john, age:int:10} ppr= vertex=vertex ID 11 {type:str:prod, desc:str:kindle Fire} ppr= vertex=vertex ID 10 {type:str:prod, desc:str:iphone5, released:dat:sat Jan 21 00:02:00 EST 2012} 14

15 Recommendation with C.F. A recursive graph algorithm solves taste signature of both customers and items Matrix factorization Graph intuition A customer s taste signature is defined by what he/she likes An item s taste signature is (recursively) defined by who likes it [ ] [ ] [ ] [ ] [ ] [ ] Customer [ ] Item

16 Demo: Build Recommender System with Graph Technologies

17 Network Intrusion Detection with Deep Learning (Skymind /DL4J) and Big Data Spatial and Graph

18 Network Intrusion Detection (NID) Goal: determine if a network activity is legitimate or fraudulent (malicious) Input Data: a sequence of network activity for a machine on a corporate network NID is similar to some other anomaly detection problems Financial fraud Breakdown detection in Vehicles Manufacturing equipment Datacenter servers

19 Network Intrusion Detection (NID) Some characteristics Most of the input data is legitimate Cannot trust first line of defense for NID: corporate firewalls Corporation may have 10s of thousands of machines Hard to monitor them all Breaches are extremely costly Two basic approaches to NID Signature based: we have a labeled dataset of known attacks (supervised learning) Anomaly based: we don t know what attacks look like Most effective NID systems use a hybrid of them, as well as rules engines

20 Oracle BDSG and DeepLearning4J Integration Achitecture Graph Database (BDSG and Oracle Spatial and Graph) Graph Database (BDSG and Oracle Spatial and Graph)

21 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Labeled Network data set UNSW-NB15 data set for Network Intrusion Detection systems Created by IXIA PerfectStorm tool in Cyber Range Lab of Australian Centre for Cyber Security A mix of Real modern normal activities, and Synthetic contemporary attack behaviors Moustafa, Nour, and Jill Slay. "UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)."military Communications and Information Systems Conference (MilCIS), IEEE, Moustafa, Nour, and Jill Slay. "The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set." Information Security Journal: A Global Perspective (2016): 1-14.

22 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Understand the data Features of UNSW-NB15 data set 49 original features

23 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization One round of clean up. Ports should be all integer based, however, there are Hex values Action: convert them back to decimal

24 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Understand the data & define transformations Categorical to One Hot transformation Service dns becomes

25 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Executed transformations with Scala & Apache Spark using Oracle s Big Data stack Save the RDD back to CSV format

26 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Built a Multi-Layer Perceptron (MLP) Neural Network

27 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Tested the quality of MLP NN After 800 iterations of training Accuracy: Labeled as non-intrusion classified as non-intrusion : 46 times Labeled as intrusion classified as non-intrusion : 1 time Labeled as intrusion classified as intrusion : 6 times ((46+6)/(46+6+1) = ) Long Short-Term Memory (LSTM) NN gave similar accuracy result

28 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization A Single GPU: GTX 970 (1664 CUDA cores, 4GB device RAM) 2-Quad core Intel CPUs (Xeon E GHz) CUDA NN Training Performance Improvement GPU over CPUs 7x x MLP LSTM

29 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Converted CSV to a (Oracle defined flat file.opv/.ope) Model each IP as a vertex Model each record (traffic from a source IP to a destination IP) as an edge 60+ Features become properties of edges Utility provided in BDSG OraclePropertyGraphUtilsBase.convertCSV2OPV OraclePropertyGraphUtilsBase.convertCSV2OPE Example CSV file 1,John,4.2,30 2,Mary,4.3,32 3,"Skywalker, Anakin",5.0,46 4,"Darth Vader",5.0,46 5,"Skywalker, Luke",5.0,53 Example output.opv file 1,name,1,John,, 1,score,4,,4.2, 1,age,2,,30, 2,name,1,Mary,, 2,score,4,,4.3, 2,age,2,,32,

30 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Utilized the built-in parallel graph data loader A single API call to loaddata method OraclePropertyGraphDataLoader opgdl = OraclePropertyGraphDataLoader.getInstance(); opgdl.loaddata(opg, <PATH>/net_intrusion.opv", <PATH>/net_intrusion.ope, 8 // 8 threads ); Oracle Big Data Spatial and Graph Apache HBase Oracle NoSQL Database

31 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Network Intrusion Detection Blue edges: malicious Other edges: normal traffic Many attacks originated from to target Visualization tool: Cytoscape v Big Data Spatial and Graph v2.1

32 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Focused on Attacks graph

33 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Focused on Attacks graph

34 Dataset selection Data Cleansing & preparation Train Neural Network model Generate Load into BDSG Graph Visualization Focused on Attacks graph Applied built-in analytics in BDSG Found top-3 IP addresses with highest Page Rank value

35 Summary Graph capabilities in Oracle Big Data Spatial and Graph and Oracle Spatial and Graph Graph databases are powerful tools, complementing relational databases Especially strong for analysis of graph topology and multi-hop relationships Graph analytics offer new insight Especially relationships, dependencies and behavioural patterns Oracle Big Data Spatial and Graph (BDSG) and Oralce Spatial and Graph (OSG) offers Comprehensive analytics through various APIs, integration with relational database Scaleable, parallel in-memory processing Secure and scaleable graph storage using Oracle NoSQL or Hbase or Oracle Database Runs on commodity hardware or BDA, both on-premise or in the Cloud 35

36 Resources Oracle Spatial and Graph oracle.com/technetwork/database/options/spatialandgraph Oracle Big Data Spatial and Graph oracle.com/database/big-data-spatial-and-graph/index.html