Analyzing Blockchain and Bitcoin Transaction Data as Graph

Transcription

1 Analyzing Blockchain and Bitcoin Transaction Data as Graph Xavier Lopez Senior Director Zhe Wu Architect Oracle Code Boston April 17th, 2018 Copyright 2015 Oracle and/or its affiliates. All rights reserved.

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Outline Overview Graph for Blockchain Analytics Understanding bitcoin transactions Graph modeling of bitcoin transactions Analysis of Bitcoin Transaction Graph (BTG) on Oracle Cloud Summary and future work 3

4 Graph Analysis for Business Insight Identify Influencers Discover Graph Patterns in Big Data Generate Recommendations 4

5 Financial Services Applying Graph Analysis To Improve Customer Service Model customer relationship to products, services, people, places. Analyze money customer s flow between non-bank to bank accounts Combine internal CRM data with enterprise and social media content Identify high-value customers across banking divisions Enhance new product/service opportunities 5

6 . Oracle Graph Databases Support Spatial and Graph use cases on every platform Property Graph Property Graph Property Graph Oracle Database 12c Oracle Big Data Spatial and Graph Database Cloud Service (DBCS) Big Data Cloud Service (BDCS)

7 Property Graph Database: Scalability and Performance OLTP and/or OLAP Graph Query Language (PGQL) Graph analytics Standard interfaces Graph visualization Integration with machine learning tools Courtesy Linkurious Courtesy Tom Sawyer Perspectives 7

8 Blockchain: Permissioned, Replicated Shared Ledger 8

9 Terminology Blockchain Bitcoin ICO BTC, ETH, LiteCoin, Satoshi ( BTC) Nonce (number used once) A few important names: - Wei Dai - Satoshi Nakamoto SHA256 (a cryptographic hash/digest that generates 256 bits signature) 9

10 Terminology (2) Mining - Doing the work of finding nonce s.t. SHA256(SHA256(data+nonce)) < difficulty - Difficulty adjusted every 2016 blocks (roughly 2 weeks) - Purpose - Find the next block to append to the chain, maintain consistency - Create new bitcoins (get rewarded) - Mining is computationally expensive - Trivial to validate results though - Presentation of a block with the correct nonce value Proof of Work (POW)

11 Terminology (3) Mining Illustrated set timing on DECLARE vcdata VARCHAR2(2000) := <Transaction Data>'; rawdata RAW(20480); digest varchar2(20480); nonce integer; BEGIN for nonce in loop -- max 32Bits rawdata := utl_raw.cast_to_raw(vcdata nonce); digest := to_char(dbms_crypto.hash(dbms_crypto.hash(rawdata, dbms_crypto.hash_sh256),dbms_crypto.hash_sh256)); if (instr(digest, '000') = 1) then dbms_output.put_line(' nonce: ' nonce); dbms_output.put_line('sh2-256: ' digest); exit; end if; end loop; END; / nonce: 451 SH2-256: 000E9DBD2D39D66F BAD3A826F2FA1FB B3D1FC98C51D380C72 PL/SQL procedure successfully completed. Elapsed: 00:00:

12 Understanding Blockchain Reference: 12

13 Motivations Bitcoins/Blockchain are fun Bitcoins can be used in many ways All bitcoin transactions are in public domain A great resource to go deep into real-world transactions 13

14 Happy Story about Bitcoin Man buys $27 of bitcoin, forgets about them, finds they're now worth $886k Bought in 2009, currency s rise in value saw small investment turn into enough to buy an apartment in a wealthy area of Oslo Kristoffer Koch invested 150 kroner ($26.60) in 5,000 bitcoins in 2009, after discovering them during the course of writing a thesis on encryption. He promptly forgot about them until widespread media coverage of the anonymous, decentralised, peer-to-peer digital currency in April 2013 jogged his memory. Bitcoins are stored in encrypted wallets secured with a private key, something Koch had forgotten. After eventually working out what the password could be, Koch got a pleasant surprise: It said I had 5,000 bitcoins in there. Measuring that in today s rates it s about NOK5m ($886,000), Koch told NRK. 14

15 Sad Story about Bitcoin Bitcoin Pizza Day: Celebrating the Pizzas Bought for 10,000 BTC Today, bitcoiners the world over will celebrate the anniversary of the most expensive pizzas in history. Bought on 22nd May 2010 by Laszlo Hanyecz, the programmer paid a fellow Bitcoin Talk forum user 10,000 BTC for two Papa John s pizzas. Back then when the technology was just over a year old that equated to roughly $25, but is $5.12m by today s exchange rate. At bitcoin s all-time high last December, the pizzas would have been worth an eye-watering $11.47m, making them likely candidates for the most expensive pizzas of all time. Now widely recognised as the first real-world transaction with bitcoin, May 22nd has come to celebrate 'Bitcoin Pizza Day', with cryptocurrency enthusiasts raising a slice to Hanyecz s infamous hunger pangs that paved the way for early merchant adoption. 15

16 Bitcoins Can Be Used in Many Ways 16

17 Bitcoin As an Investment Caution: highly volatile. Not for faint hearted. Risk from quantum computing 17

18 Bitcoin As an Investment Caution: highly volatile. Not for faint hearted. Risk from quantum computing 18

19 Bitcoin for Financial Purpose Purchase things Pay ransomware (Wannacry) Move asset across border Store of value (as opposed to USD & inflation) Free of banking system Wealth privacy The underlying blockchain technology has wider applications 19

20 Bitcoin for Dark Side Purchase drugs Demand ransomware (Wannacry) payment in bitcoin Be sure to use multiple distinct payment addresses Wannacry: 115p7UMMngoj1pMvkpHijcRdfJNXj6LrLn Hide wealth from government/tax department Bribe VIPs 20

21 What Does a Bitcoin Transaction Look Like A transaction has input(s) and output(s) An input comes from an output of a transaction TX hash : 6f7cf9580f1c2dfb3c4d5d043cdbb128c640e3f aa7372e TX outputsum : TX Input from : ff3dc8b461305acc5900d31602f2dafebfc406e5b050b14a352294f0965e0bf6:0 -- TX Input from : 2db d0132d fd20329be9cd590fa5ae2b3c55f58931f42e27f7:0 -- TX Output value TX Output scripubaddr 12higDjoCCNXSA95xZMWUdPvXNmkAduhWv Note: 1,000,000 is 0.01 BTC 21

22 What Does a Bitcoin Transaction Look Like A transaction has input(s) and output(s) An input comes from an output of a transaction TX1 $ $ $ Addr X Addr K TX3 TX8 $ $ Addr Y TX9 $ $ $ Addr L Addr Z 22

23 What Is Graph A set of vertices (entities), edges (relationships), and properties Graph is also known as linked data TX1 $ $ $ Addr X Addr K TX3 TX8 $ $ Addr Y TX9 $ $ $ Addr L Addr Z 23

24 Bitcoin Transaction As a Graph Graph Model 1 Transaction as vertices Address as vertices Transaction references as edges (Trans Trans, Trans Addr) 24

25 Bitcoin Transaction As a Graph Graph Model 2 Transaction as vertices Address as vertices Transaction s indirect reference to Address as edges (Addr Trans Addr) 25

26 Bitcoin Transaction As a Graph Graph Model 3 Address as vertices Address to address payment as edges (Addr Addr) TX1 $ $ $ Addr X Addr K TX3 TX8 $ $ Addr Y TX9 $ $ $ Addr L Note: 1,000,000 is 0.01 BTC Addr Z 26

27 Bitcoin Transaction As a Graph: Money Flow Graph Model 3 Address to address payment as edges (Addr Addr) What is Addr X s contribution to Addr K? TX1 $ $ $ Addr X Addr K TX3 TX8 $ $ Addr Y TX9 $ $ $ Addr L Note: 1,000,000 is 0.01 BTC Addr Z 27

28 Bitcoin Transaction As a Graph: Money Flow Graph Model 3 Address to address payment as edges (Addr Addr) What is Addr X s contribution to Addr K? Given an input address i, output address o Contribution of i to o: Amount i i Amount i Amount o 28

29 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics 29

30 Bitcoin Transactions As Graph Workflow Functions of a Graph Database Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics 30

31 Bitcoin Transactions As Graph Workflow Functions of a Graph Database Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics OCI (BMC) & Oracle Database Cloud Service 12.2 (18.1) 31

32 Overview of Oracle s Graph Database 32

33 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 33

34 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 34

35 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Apache Spark Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 35

36 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Apache Spark Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 36

37 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Apache Spark Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Apache HBase Oracle NoSQL Database 37

38 Property Graph with Oracle Database Cloud Service 38

39 Property Graph with Big Data Cloud Service 39

40 Computational Analytics: Built-in Package Rich set of built-in parallel graph algorithms and parallel graph mutation operations Detecting Components and Communities Tarjan s, Kosaraju s, Weakly Connected Components, Label Propagation (w/ variants), Soman and Narang s Spacification Evaluating Community Structures Link Prediction Conductance, Modularity Clustering Coefficient (Triangle Counting) Adamic-Adar SALSA (Twitter s Who-to-follow) Ranking and Walking Path-Finding Other Classics Pagerank, Personalized Pagerank, Betwenness Centrality (w/ variants), Closeness Centrality, Degree Centrality, Eigenvector Centrality, HITS, Random walking and sampling (w/ variants) Hop-Distance (BFS) Dijkstra s, Bi-directional Dijkstra s Bellman-Ford s Vertex Cover Minimum Spanning-Tree (Prim s) a b e d g c Create Bipartite Graph b d e Filtered Subgraph i i g Left Set: a,b,e The original graph a b c d e Sort-By-Degree (Renumbering) e b d i a f c g h i f g h a b c d e i f g h Create Undirected Graph a b c d e g Simplify Graph i f h 40

41 Text Search through Apache Lucene/SolrCloud Why? Contribute to the performance of graph traversal queries Constrained to be uniform in type among the indexed elements (vertices or edges) Automatic Indexes Automatic update based on a subset of property keys Avoid linear scan to access an element by key/value Manual Indexes Maintained by users Fasten up text searches by a particular key/value pair Sub-graphs based on a set of (existing or temporary) properties 42

42 Visualizing Property Graphs (with Cytoscape) Cytoscape supports Property Graph Connects to Oracle Database, Oracle NoSQL Database, or Apache HBase Runs Page Rank, Clustering, Shortest Path, etc Alternative to command-line for inmemory analytics once base graph created

43 Additional Graph Visualization Partners TomSawyer, Cambridge Intelligence, Linkurios, Vis.js,...

44 Pattern matching using PGQL SQL-like syntax but with graph pattern description and property access Interactive (real-time) analysis Supporting aggregates, comparison, such as max, min, order by, group by Finding a given pattern in graph Fraud detection Anomaly detection Subgraph extraction... Proposed for standardization by Oracle Specification available on-line Open-sourced front-end (i.e. parser)

45 Zeppelin Frontend Apache Zeppelin Multi-purpose notebook for data analysis and visualization Enables to embed interactive execution inside Browsers Renders execution results with plots and tables within Browsers PGX provides a hook (interpreter) for Zeppelin integration 46

46 Interacting with the Graph Access through APIs Implementation of Apache Tinkerpop Blueprints APIs Based on Java, REST plus SolR Cloud/Lucene support for text search SQL/PLSQL for property graph functions in Oracle Database Scripting Groovy, Python, Javascript,... Zeppelin integration, Javascript (Node.js) language binding Graphical UIs Cytoscape, plug-in available for BDSG Commercial Tools such as TomSawyer Perspectives, Ogma

47 Some Details on Applying Graph Database to Bitcoin Transactions 48

48 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Database joins, ID generation, mapping, etc

49 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Database joins, ID generation, mapping, etc

50 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph: RDBMS Table => Flat Files OraclePropertyGraphUtils.convertRDBMSTable2OPV(E) 51

51 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph: RDBMS Table => Flat Files OraclePropertyGraphUtils.convertRDBMSTable2OPV(E) 52

52 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph: RDBMS Table => Flat Files OraclePropertyGraphUtils.convertRDBMSTable2OPV(E) 53

53 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph opgdl=oraclepropertygraphdataloader.getinstance(); vfile="/home/graph/btc.opv" // vertex flat file efile="/home/graph/btc.ope" // edge flat file opgdl.loaddatawithsqlldr(opg, "pg", "pg", "orcl122", vfile, efile, 8/*dop*/, false, "sqlldr", true, "PDML=T,PDDL=T,NO_DUP=T,"); 54

54 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics A subset of all bitcoin transactions was converted into a property graph: - 364K vertices - 751K edges 55

55 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics 3bda fd55775a f4c26d898d5840c7e a05ef0b743d8 56

56 Querying Bitcoin Transactions Find top transactions with SQL 400,000 BTC s! 57

57 Finding Critical Bitcoin Addresses Using Groovy with BDSG 58

58 Finding Bitcoin Addresses with Many Sends Using PGQL opg-nosql> pgxresultset = pgxgraph.querypgql("select n, count(m) WHERE (n) - > (m) GROUP BY n ORDER BY count(m) DESC ") ==>PgqlResultSetImpl[graph=btc,numResults=322783] opg-nosql> pgxresultset.print(10); n count(m) PgxVertex[ID=273095] PgxVertex[ID=40669] 7715 PgxVertex[ID=154277] 7580 PgxVertex[ID=35950] 4352 PgxVertex[ID=127317] 5521 PgxVertex[ID=7367] 2068 PgxVertex[ID=45172] 1830 PgxVertex[ID=218422] 1510 PgxVertex[ID=181060] 1450 PgxVertex[ID=197650] ==>null opg-nosql> pgxresultset.getnumresults() ==> opg-nosql> opg.getvertex(273095) ==>Vertex ID {bt_addr:str:1pjnjo4n2rt5jwturcrr4ink2xmfpxqfc7} 59

59 Are Bitcoin Addresses Totally Anonymous? Yes but it depends 60

60 Some Observations A few very active addresses 61

61 Some Observations (2) Activities with an interesting pattern 62

62 Summary and Future Work This talk covered Blockchain/Bitcoin transactions Oracle Spatial and Graph Property Graph database Techniques used for graph modeling, query, visualizations are easily applicable to other domains Banking, logistics network, asset network, social network, security, Future work Add temporal (timestamp) Merge public addresses Identify true value Maybe convert bitcoin amount into USD? Detect anomaly /fraud Decentralized system, no bank or government to complain about abnormal activities 63

63 Resources on Oracle s Property Graph Database Oracle Spatial and Graph Property Graph Oracle Spatial and Graph Blog Getting Started Creating a Property Graph on Oracle Database by Arthur Dayton from Vlamis Software Solutions Oracle Big Data Spatial and Graph on Oracle.com: OTN product page (white papers, software downloads, documentation, tutorials): Oracle Big Data Lite Virtual Machine - a free sandbox to get started: Hands On Lab for Big Data Spatial: tinyurl.com/bdsg-hol Blog examples, tips & tricks: blogs.oracle.com/bigdataspatialgraph 64

64 Resources Online communities Oracle Spatial & Graph SIG user groups (search Oracle Spatial @alanzwu - follow the S&G team Spatial and Graph AskTOM Office Hours (every month) 65