Analysing the Panama Papers with Oracle Big Data Spatial and Graph

Transcription

1 speakerdeck.com/rmoff/ Analysing the Panama Papers with Oracle Big Data Spatial and Graph BIWA Summit 2017 Robin Moffatt, Rittman Mead 1

2 Robin Moffatt! Head of R&D, Rittman Mead Previously OBIEE/DW developer at large UK retailer Previously SQL Server DBA, Business Objects, DB2, COBOL.! Oracle ACE! Frequent blogger : and IRC: rmoff / #obihackers / freenode info@rittmanmead.com 2

3 Rittman Mead! Oracle Gold Partner with offices in the UK and USA! 70+ staff delivering Oracle BI, DW, Big Data and Advanced Analytics projects! Significant web presence with the Rittman Mead Blog ( Hadoop R&D lab for dogfooding solutions developed for customers 3

4 What are the Panama Papers?! Dataset of 11.5 million documents regarding offshore entities, released in 2015! International Consortium of Investigative Journalists (ICIJ) analysed the raw data and made available a curated set of the data! The New York Times and The Guardian among newspapers that investigated the data in depth There are legitimate uses for o"shore companies and trusts. We do not intend to suggest or imply that any persons, companies or other entities included in the ICIJ O!shore Leaks Database have broken the law or otherwise acted improperly. Many people and entities have the same or similar names. 4

5 What is a Property Graph and Why is it So Useful? Graph enables us to answer question that relational would struggle with - Show me all people with a connection to an entity that is connected to at least five other people You could write recursive or procedural SQL but it would be nasty. It would also be impossible to maintain and repeat at scale. Graph-based algorithms (e.g. PageRank) enrich an existing dataset and give us additional insights into it 5

6 Property Graph Terminology Node/Vertex - The What Edge - The How / Relationship (Un)Directed - Does the relationship go in a specific direction - for example, a person is an officer of a company, not the other way around 6

7 Graph Analysis Uses 7

8 Oracle Big Data Spatial and Graph! Store the Property Graph definition in HBase or Oracle NoSQL! API to load/modify data! In-memory analytic engine (PGX) loads graph for analysis, and provides built-in algorithm implementations! Also provides RDF and Spatial capabilities 8

9 New in Oracle Property Graph support! Store Property Graph data in Oracle RDBMS! Query through PGX same way as existing Big Data tool version! 122/SPGDG/using-propertygraphs-oracledatabase.htm#BDSPA191 9

10 Interacting with Property Graphs! First-class Java APIs for programatic access - Gremlin support for interactive work! Nascent Python support (pyopg)! Spark library available! Use Notebooks!! Interactive visualisation and exploration of data with tools like Cytoscape (open source), Tom Sawyer (paid), etc 10

11 Notebooks! Interactive code development & execution environment! Notebooks can be shared for others to run and reproduce findings! Apache Zeppelin and Jupyter two popular options! Working with Spatial and Graph: - pyopg/jupyter - PGX/Zeppelin 11

12 Apache Zeppelin 12

13 Loading the Data Data can be loaded from various formats: - GraphML Data Format - GraphSON Data Format - GML Data Format - Oracle Flat File Format Source data was CSV, which needed wrangling to fit a supported input format - Oracle Flat File Format was chosen - Supports highly-parallised loading in BDSG - BDSG now provides CSV->OPE/V Java API 13

14 Data Wrangling with R panama_opv_ope.r 14

15 importflatfiles() 15

16 Inspecting the Property Graph 16

17 Inspecting the Property Graph 17

18 Analysing the Property Graph 18

19 Analysing the Property Graph - Zeppelin! Native rendering support for resultset objects 19

20 Analysing the Data - Property Graph Query Language (PGQL)! SQL-like language for querying property graph! Same SELECT.. WHERE clause pattern but with syntax for expressing graph relationships! 20

21 Simple PGQL 21

22 PGQL with Predicate 22

23 Built-in Functions 23

24 Native Charting in Zeppelin 24

25 Powerful Predicate support 25

26 PGX Built-In Algorithms 26

27 Exploring the Property Graph with Cytoscape 27

28 Exploring the Property Graph with Cytoscape 28

29 Exploring the Property Graph with Cytoscape 29

30 Layout Algorithms Prefuse Force Directed Layout 30

31 Community Detection 31

32 Community Detection 32

33 Let s walk through an example 33

34 34

35 35

36 36

37 37

38 38

39 39

40 40

41 EOF #EOF ! web! speakerdeck.com/rmoff/ irc! #obihackers 41