Analyzing Blockchain and Bitcoin Transaction Data as Graph

Transcription

1 Analyzing Blockchain and Bitcoin Transaction Data as Graph Zhe Wu Ph.D. Architect Oracle Spatial and Graph Feb 2018

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Outline Overview of bitcoin/blockchain Understanding bitcoin transactions Graph modeling of bitcoin transactions Analysis, query, and visualization of Bitcoin Transaction Graph (BTG) - Oracle Spatial and Graph (OSG) Property Graph Summary and future work 3

4 Terminology Blockchain Bitcoin ICO BTC, ETH, LiteCoin, Satoshi ( BTC) Nonce (number used once) A few important names: - Wei Dai - Satoshi Nakamoto SHA256 (a cryptographic hash/digest that generates 256 bits signature) 4

5 Terminology (2) Mining - Doing the work of finding nonce s.t. SHA256(SHA256(data+nonce)) < difficulty - Difficulty adjusted every 2016 blocks (roughly 2 weeks) - Purpose - Find the next block to append to the chain, maintain consistency - Create new bitcoins (get rewarded) - Mining is computationally expensive - Trivial to validate results though - Presentation of a block with the correct nonce value Proof of Work (POW) 5

6 Terminology (3) Mining Illustrated set timing on DECLARE vcdata VARCHAR2(2000) := 'OracleCode LA 2018'; rawdata RAW(20480); digest varchar2(20480); nonce integer; BEGIN for nonce in loop -- max 32Bits rawdata := utl_raw.cast_to_raw(vcdata nonce); digest := to_char(dbms_crypto.hash(dbms_crypto.hash(rawdata, dbms_crypto.hash_sh256),dbms_crypto.hash_sh256)); if (instr(digest, '000') = 1) then dbms_output.put_line(' nonce: ' nonce); dbms_output.put_line('sh2-256: ' digest); exit; end if; end loop; END; / nonce: 451 SH2-256: 000E9DBD2D39D66F BAD3A826F2FA1FB B3D1FC98C51D380C72 PL/SQL procedure successfully completed. Elapsed: 00:00:

7 Understanding Blockchain Reference: 7

8 Motivations Bitcoins/Blockchain are fun Bitcoins can be used in many ways All bitcoin transactions are in public domain A great resource to go deep into real-world transactions 8

9 Happy Story about Bitcoin Man buys 27 of bitcoin, forgets about them, finds they're now worth 886k Bought in 2009, currency s rise in value saw small investment turn into enough to buy an apartment in a wealthy area of Oslo Kristoffer Koch invested 150 kroner (26.60) in 5,000 bitcoins in 2009, after discovering them during the course of writing a thesis on encryption. He promptly forgot about them until widespread media coverage of the anonymous, decentralised, peer-to-peer digital currency in April 2013 jogged his memory. Bitcoins are stored in encrypted wallets secured with a private key, something Koch had forgotten. After eventually working out what the password could be, Koch got a pleasant surprise: It said I had 5,000 bitcoins in there. Measuring that in today s rates it s about NOK5m (886,000), Koch told NRK. 9

10 Sad Story about Bitcoin Bitcoin Pizza Day: Celebrating the Pizzas Bought for 10,000 BTC Today, bitcoiners the world over will celebrate the anniversary of the most expensive pizzas in history. Bought on 22nd May 2010 by Laszlo Hanyecz, the programmer paid a fellow Bitcoin Talk forum user 10,000 BTC for two Papa John s pizzas. Back then when the technology was just over a year old that equated to roughly 25, but is 5.12m by today s exchange rate. At bitcoin s all-time high last December, the pizzas would have been worth an eye-watering 11.47m, making them likely candidates for the most expensive pizzas of all time. Now widely recognised as the first real-world transaction with bitcoin, May 22nd has come to celebrate 'Bitcoin Pizza Day', with cryptocurrency enthusiasts raising a slice to Hanyecz s infamous hunger pangs that paved the way for early merchant adoption. 10

11 Bitcoins Can Be Used in Many Ways 11

12 Bitcoin As an Investment Caution: highly volatile. Not for faint hearted. Risk from quantum computing 12

13 Bitcoin As an Investment Caution: highly volatile. Not for faint hearted. Risk from quantum computing 13

14 Bitcoin for Financial Purpose Purchase things Pay ransomware (Wannacry) Move asset across border Store of value (as opposed to USD & inflation) Free of banking system Wealth privacy The underlying blockchain technology has wider applications 14

15 Bitcoin for Dark Side Purchase drugs Demand ransomware (Wannacry) payment in bitcoin Be sure to use multiple distinct payment addresses Wannacry: 115p7UMMngoj1pMvkpHijcRdfJNXj6LrLn Hide wealth from government/tax department Bribe VIPs 15

16 What Does a Bitcoin Transaction Look Like A transaction has input(s) and output(s) An input comes from an output of a transaction TX hash : 6f7cf9580f1c2dfb3c4d5d043cdbb128c640e3f aa7372e TX outputsum : TX Input from : ff3dc8b461305acc5900d31602f2dafebfc406e5b050b14a352294f0965e0bf6:0 -- TX Input from : 2db d0132d fd20329be9cd590fa5ae2b3c55f58931f42e27f7:0 -- TX Output value TX Output scripubaddr 12higDjoCCNXSA95xZMWUdPvXNmkAduhWv Note: 1,000,000 is 0.01 BTC 16

17 What Does a Bitcoin Transaction Look Like A transaction has input(s) and output(s) An input comes from an output of a transaction TX1 Addr X Addr K TX3 TX8 Addr Y TX9 Addr L Addr Z 17

18 What Is Graph A set of vertices (entities), edges (relationships), and properties Graph is also known as linked data TX1 Addr X Addr K TX3 TX8 Addr Y TX9 Addr L Addr Z 18

19 Bitcoin Transaction As a Graph Graph Model 1 Transaction as vertices Address as vertices Transaction references as edges (Trans Trans, Trans Addr) 19

20 Bitcoin Transaction As a Graph Graph Model 2 Transaction as vertices Address as vertices Transaction s indirect reference to Address as edges (Addr Trans Addr) 20

21 Bitcoin Transaction As a Graph Graph Model 3 Address as vertices Address to address payment as edges (Addr Addr) TX1 Addr X Addr K TX3 TX8 Addr Y TX9 Addr L Note: 1,000,000 is 0.01 BTC Addr Z 21

22 Bitcoin Transaction As a Graph: Money Flow Graph Model 3 Address to address payment as edges (Addr Addr) What is Addr X s contribution to Addr K? TX1 Addr X Addr K TX3 TX8 Addr Y TX9 Addr L Note: 1,000,000 is 0.01 BTC Addr Z 22

23 Bitcoin Transaction As a Graph: Money Flow Graph Model 3 Address to address payment as edges (Addr Addr) What is Addr X s contribution to Addr K? Given an input address i, output address o Contribution of i to o: Amount i i Amount i Amount o 23

24 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics 24

25 Bitcoin Transactions As Graph Workflow Functions of a Graph Database Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics 25

26 Overview of Oracle s Graph Database 26

27 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 27

28 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 28

29 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Apache Spark Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 29

30 Architecture of Property Graph Support Graph Analytics Parallel In-Memory Graph Analytics/Graph Query (PGX) Java APIs Graph Data Access Layer (DAL) Scalable and Persistent Storage Management Apache Spark Blueprints & Lucene/SolrCloud RDF (RDF/XML, N- Triples, N-Quads, TriG,N3,JSON) GML Java APIs/JDBC/SQL/PLSQL REST/Web Service/Notebooks Java, Groovy, Python, Property Graph formats GraphML Graph-SON Flat Files Oracle RDBMS Apache HBase Oracle NoSQL Database 30

31 Computational Analytics: Built-in Package Rich set of built-in parallel graph algorithms and parallel graph mutation operations Detecting Components and Communities Tarjan s, Kosaraju s, Weakly Connected Components, Label Propagation (w/ variants), Soman and Narang s Spacification Evaluating Community Structures Link Prediction Conductance, Modularity Clustering Coefficient (Triangle Counting) Adamic-Adar SALSA (Twitter s Who-to-follow) Ranking and Walking Path-Finding Other Classics Pagerank, Personalized Pagerank, Betwenness Centrality (w/ variants), Closeness Centrality, Degree Centrality, Eigenvector Centrality, HITS, Random walking and sampling (w/ variants) Hop-Distance (BFS) Dijkstra s, Bi-directional Dijkstra s Bellman-Ford s Vertex Cover Minimum Spanning-Tree (Prim s) a b e d g c Create Bipartite Graph b d e Filtered Subgraph i i g Left Set: a,b,e The original graph a b c d e Sort-By-Degree (Renumbering) e b d i a f c g h i f g h a b c d e i f g h Create Undirected Graph a b c d e g Simplify Graph i f h 31

32 Text Search through Apache Lucene/SolrCloud Why? Contribute to the performance of graph traversal queries Constrained to be uniform in type among the indexed elements (vertices or edges) Automatic Indexes Automatic update based on a subset of property keys Avoid linear scan to access an element by key/value Manual Indexes Maintained by users Fasten up text searches by a particular key/value pair Sub-graphs based on a set of (existing or temporary) properties 33

33 Visualizing Property Graphs (with Cytoscape) Cytoscape supports Property Graph Connects to Oracle Database, Oracle NoSQL Database, or Apache HBase Runs Page Rank, Clustering, Shortest Path, etc Alternative to command-line for inmemory analytics once base graph created

34 Additional Graph Visualization Partners TomSawyer, Cambridge Intelligence, Linkurios, Vis.js,...

35 Pattern matching using PGQL SQL-like syntax but with graph pattern description and property access Interactive (real-time) analysis Supporting aggregates, comparison, such as max, min, order by, group by Finding a given pattern in graph Fraud detection Anomaly detection Subgraph extraction... Proposed for standardization by Oracle Specification available on-line Open-sourced front-end (i.e. parser)

36 Zeppelin Frontend Apache Zeppelin Multi-purpose notebook for data analysis and visualization Enables to embed interactive execution inside Browsers Renders execution results with plots and tables within Browsers PGX provides a hook (interpreter) for Zeppelin integration 37

37 Interacting with the Graph Access through APIs Implementation of Apache Tinkerpop Blueprints APIs Based on Java, REST plus SolR Cloud/Lucene support for text search SQL/PLSQL for property graph functions in Oracle Database Scripting Groovy, Python, Javascript,... Zeppelin integration, Javascript (Node.js) language binding Graphical UIs Cytoscape, plug-in available for BDSG Commercial Tools such as TomSawyer Perspectives, Ogma

38 Some Details on Applying Graph Database to Bitcoin Transactions 39

39 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Database joins, ID generation, mapping, etc

40 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Database joins, ID generation, mapping, etc

41 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph: RDBMS Table => Flat Files OraclePropertyGraphUtils.convertRDBMSTable2OPV(E) 42

42 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph: RDBMS Table => Flat Files OraclePropertyGraphUtils.convertRDBMSTable2OPV(E) 43

43 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph: RDBMS Table => Flat Files OraclePropertyGraphUtils.convertRDBMSTable2OPV(E) 44

44 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics Oracle Spatial and Graph Property Graph opgdl=oraclepropertygraphdataloader.getinstance(); vfile="/home/graph/btc.opv" // vertex flat file efile="/home/graph/btc.ope" // edge flat file opgdl.loaddatawithsqlldr(opg, "pg", "pg", "orcl122", vfile, efile, 8/*dop*/, false, "sqlldr", true, "PDML=T,PDDL=T,NO_DUP=T,"); 45

45 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics A subset of all bitcoin transactions was converted into a property graph: - 364K vertices - 751K edges 46

46 Bitcoin Transactions As Graph Workflow Blockchain Block data parsing Data preparation Graph Generation Graph Data loading Graph viz query and analytics 3bda fd55775a f4c26d898d5840c7e a05ef0b743d8 47

47 Querying Bitcoin Transactions Find top transactions with SQL 400,000 BTC s! 48

48 Finding Critical Bitcoin Addresses Using Groovy with BDSG 49

49 Finding Bitcoin Addresses with Many Sends Using PGQL opg-nosql> pgxresultset = pgxgraph.querypgql("select n, count(m) WHERE (n) - > (m) GROUP BY n ORDER BY count(m) DESC ") ==>PgqlResultSetImpl[graph=btc,numResults=322783] opg-nosql> pgxresultset.print(10); n count(m) PgxVertex[ID=273095] PgxVertex[ID=40669] 7715 PgxVertex[ID=154277] 7580 PgxVertex[ID=35950] 4352 PgxVertex[ID=127317] 5521 PgxVertex[ID=7367] 2068 PgxVertex[ID=45172] 1830 PgxVertex[ID=218422] 1510 PgxVertex[ID=181060] 1450 PgxVertex[ID=197650] ==>null opg-nosql> pgxresultset.getnumresults() ==> opg-nosql> opg.getvertex(273095) ==>Vertex ID {bt_addr:str:1pjnjo4n2rt5jwturcrr4ink2xmfpxqfc7} 50

50 Are Bitcoin Addresses Totally Anonymous? Yes but it depends 51

51 Some Observations A few very active addresses 52

52 Some Observations (2) Activities with an interesting pattern 53

53 Summary and Future Work This talk covered Blockchain/Bitcoin transactions Oracle Spatial and Graph Property Graph database Techniques used for graph modeling, query, visualizations are easily applicable to other domains Banking, logistics network, asset network, social network, security, Future work Add temporal (timestamp) Merge public addresses Identify true value Maybe convert bitcoin amount into USD? Detect anomaly /fraud Decentralized system, no bank or government to complain about abnormal activities 54

54 Resources on Oracle s Property Graph Database Oracle Spatial and Graph Property Graph Oracle Spatial and Graph Blog Getting Started Creating a Property Graph on Oracle Database by Arthur Dayton from Vlamis Software Solutions Oracle Big Data Spatial and Graph on Oracle.com: OTN product page (white papers, software downloads, documentation, tutorials): Oracle Big Data Lite Virtual Machine - a free sandbox to get started: Hands On Lab for Big Data Spatial: tinyurl.com/bdsg-hol Blog examples, tips & tricks: blogs.oracle.com/bigdataspatialgraph 55

55 Resources Online communities Oracle Spatial & Graph SIG user groups (search Oracle Spatial @alanzwu - follow the S&G team Spatial and Graph AskTOM Office Hours (every month) 56

56 Oracle HQ, Redwood Shores, CA Spatial + Graph sessions, 10 Graph sessions. From hands on labs/intros to get started, to use cases in finance, manufacturing, blockchain, IoT and more Expert speakers from Oracle development and the user community Copyright 2015 Oracle and/or its affiliates. All rights reserved.

57 Graph Developer Training Day March 19, 2018 at Oracle HQ Conference Center, Redwood Shores, CA A free full-day workshop to help you develop property graph applications using Oracle technologies, delivered by the Oracle product team Targeted toward developers, architects and data scientists View agenda at RSVP to marion.smith@oracle.com by March 6 if interested