Scalable overlay Networks

Transcription

1 overlay Networks Dr. Samu Varjonen 1

2 Lectures MO C122 Introduction. Exercises. Motivation. TH DK117 Unstructured networks I MO C122 Unstructured networks II TH DK117 Bittorrent and evaluation MO C122 Privacy (Freenet etc.) and intro to power-law networks. TH DK117 Consistent hashing. Distributed Hash Tables (DHTs) MO C122 DHTs continued TH DK117 Power-law networks MO C122 Power-law networks and applications. TH DK117 Applications I MO C122 Applications II TH DK117 Advanced topics MO C122 Conclusions and summary TH DK117 Reserved 2

3 Summary 1 / 2 Summary Central index: Napster Flooding: Gnutella Selective flooding: Gnutella with ultra peers Efficiently sharing information: Bittorrent Anonymity/Privacy: Freenode Oneswarm 3

4 Summary 2 / 2 Structured overlays: DHTs: Chord Pastry Tapestry CAN Complex networks Milgram s experiment Random networks Rewiring model Scale free networks Preferential attachment Evolving copying model Strong clustering Error and fault tolerance (Scale free is good for random errors) Navigation Greedy routing Kleinberg s small world Complex networks (strong clustering in small worlds) Mathematics and Internet: A source of Enormous Confusion and Great potential 4

5 Contents What now? Trust 5

6 Byzantine generals problem Commander Commander Attack Attack Attack Retreat Lieutenant 1 Lieutenant 2 He said retreat Lieutenant 1 Lieutenant 2 He said retreat Byzantine refers to the Byzantine Generals' Problem, an agreement problem in which a group of generals, each commanding a portion of the Byzantine army, encircle a city. Described by Leslie Lamport, Robert Shostak and Marshall Pease in their 1982 paper, "The Byzantine Generals Problem" 6

7 Centralized trust One truth Power at the center Can be abused Deniability Alterations Non-resilient attacks Long delays on transactions You have to trust the authority 7

8 Decentralized trust: requirements Decentralized Transactions between anybody No central issuing or verification authority No fraudulent transactions The goods that the transaction refers to exist Network of random nodes but everyone maintains a copy of the chain So everyone knows everything Immutability Resilient Anonymity Fast Cheap Voluntary Micro-transactions A very small financial transaction conducted online 8

9 Bitcoin: A Peer-to-Peer Electronic Cash System We Propose a solution to the double-spending problem using a peer-to-peer network. The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power The network itself requires minimal structure. 9

10 Who? Satoshi Nakamoto 10

11 Bitcoin A protocol Unit of account Based on blockchain Transaction content is to who and how much All transactions are public and by default anonymous Transfer of bitcoins is not moving anything but storing that transaction to the blockchain Transfers secured by miners who create blocks that are easy for the clients to validate Miners get compensation of the work 11

12 Public vs Private Business Asset Supplier Permissionless Blockchain Private Blockchain Permissioned Vendor Product Inventory Slower Public ownership Open & Transparent Trust-free Allegal Faster Managed upkeep Private membership Trusted Legal 12

13 Bitcoin network A peer to peer network (P2P) a decentralised network with no central point of failure or command All the nodes, or computers, that participate in the network are equal there is no hierarchy No topology as the network is flat. A collection of nodes running the Bitcoin P2P protocol With other protocols such as stratum used for lightweight mobile wallets (where the full block chain is not downloaded) or for mining. Although the network is flat, and all nodes are equal, some nodes or computers can choose to perform a different task. 13

14 Node tasks A full node does: Routing holds a full copy of the blockchain database performs mining provides wallet services Any node has to provide the routing function to participate in the network. Routing is just another way of saying that the node validates and passes on transactions and block as well as discovering and maintaining connections to other nodes, whether they are full nodes or not. Other nodes: Mobile wallets however can t hold the entire blockchain through space requirements and are essentially simplified payment verification nodes or lightweight nodes. Mining only nodes 14

15 Transactions Let s make a payment Alice wants to give 1 BTC to Bob Alice: retrieves the Bob s address Generates own Public/Private key Builds a message with: Transaction of 1 BTC to Bob s address Transaction s signature Announces her public key for signature verification Broadcasts the message to the Bitcoin network Each transaction Unique signature Can be verified who signed it Any attempt to manipulate transactions invalidates the signature Input and output of the transaction must be equal 15

16 Transaction as double-entry bookkeeping Input Value (BTC) Ouput Value (BTC) Input Output Input Output Input Output Input Total inputs Inputs 0.55 outputs 0.5 Difference 0.05 The implied transaction fee Paying a transaction fee is optional. Miners can choose which transactions to process and prioritize those that pay higher fees. This means that to get your transaction processed quickly you will have to outbid other users. Fees are based on the storage size of the transaction generated, which in turn is dependent on the number of inputs used to create the transaction. The space available for transactions in a block is currently artificially limited to 1 MB. 16

17 Digital signatures Sender Receiver Msg Msg Digest algorithm Sender s private key Sender s public key Digest algorithm Msg digest Encryption algorithm Encryption algorithm Msg digest Msg digest Did these two match? 17

18 Why Public/Private key? Bob has the Alice s public key Only Alice knows the private key Bob verifies the transaction signature using the Alice s public key Only Bob is authorized because has the private key 18

19 Bitcoin transaction The transaction from B to C is signed with B s private key The B s public key is included in the transaction And so on... With this system, bitcoins are passed from address to address through a chain of transactions. Each step in the chain can be verified to ensure that bitcoins are being spent validly. Transactions can have multiple inputs and outputs in general, so the chain branches out into a tree. 19

20 Problem: ordering of transactions Whenever a user initiates a transaction, the transaction goes to an unconfirmed pool. Once they are confirmed and validated they are added to the Blockchain. This would result in two complications: If there is no particular order to transactions, then there could be several branches of the block as plenty of miners would mine transactions and add it to the block in parallel leading to a speculation of which chain blocks is the right one Double spending 20

21 Double spending problem 1. Alice wants buy stuff from Bob 2. Alice sends Bob some bitcoins 3. Takes the items 4. Alice sends herself some bitcoins referencing same inputs 5. Possibility that Alice s transaction to herself gets confirmed 6. All other users will reject Alice -> Bob transaction 7. Alice get s the stuff for free! 21

22 Solution: Double-Spending Problem Transaction details are sent and forwarded to all Bitcoin nodes A Block-Chain contains all transaction done Each blocks into chains must be valid and must include a proof-of-work Network You wait until validated and distributed 22

23 Transactions to the block Crypto hashes Merkle tree Signatures 23

24 Blockchain A transaction databases shared by all nodes (like a ledger) Each block must contain a Proof-of-Work Contains a reference (hash) to the previous block Blockchain avoids the double spending BUT WAIT: what if someone else solved the proof-of-work berofe me Switch to longest chain The longest chain represents the accumulated computational power of the network 24

25 Block chain vs. Tree 1 / 3 Bitcoin's block chain system is really two quite separate systems, and they are easily confused. The block tree The active chain The block tree consists of all valid blocks whose entire ancestry is known, up to the genesis block. The rules for validness include: No double spending Valid signatures No introduction of more currency than allowed, These are the network rules, and every full Bitcoin node verifies them. 26

26 Block chain vs. Tree 2 / 2 The active chain is one path from genesis block at the top to some leaf node at the bottom of the block tree. Every such path is a valid choice, but nodes are expected to pick the one with the most "work" in it they know about (where work is loosely defined as the sum of the difficulties). Relativity and technological constraints prevent us from doing instant communication across the globe, so two nodes can not be expected to pick the same chain as the active one. This is no problem: the mining mechanism makes sure that the chance two nodes disagree about blocks in the past decreases exponentially as they are older. So no, there is not one "right chain", there are many. Nodes choose for themselves, but the system is designed to make sure consensus arises quickly. 27

27 Block chain vs. Tree 3 / 3 The rules in practice are this: when a new block arrives, and it extends the previous active chain, we just append it to the active chain. If not, it depends on whether the branch it extends now has more work than the currently active branch. If not, we store the block and stop. If it does have more work, we do a so called "reorganisation": deactivating blocks from the old branch, and activating blocks from the new branch. 28

28 Anyone can add a block Yes but with some effort Proof-of-Work Miners Corresponds at time of computation The time is spent computing hash function The Proof-of-Work is built when the hash s output has a specific property A block with a Proof-of-Work about each 10 minutes The difficulty is established by a target The miners spends time to find a Proof-of-Work in order to receive a Block reward (now is 12.5 BTC) The reward reduces 50% every 4 years The mining process will end when 21 million bitcoins will be reached After that time the miners will earns imposing a transaction fee The block reward started at 50 BTC in block #1 and halves every 210,000 blocks. This means every block up until block #210,000 rewards 50 BTC, while block 210,001 rewards 25. Since blocks are mined on average every 10 minutes, 144 blocks are mined per day on average. At 144 blocks per day, 210,000 blocks take on average four years to mine. 29

29 Bitcoin tech The distributed algorithm dynamically adjusts ho much computing power it takes to find a valid block. This limits the block creation rougly block per 10 minutes The Block reward + transaction fees miners earn are compensations for securing the block How do they find this number? By guessing at random. The hash function makes it impossible to predict what the output will be. So, miners guess the mystery number and apply the hash function to the combination of that guessed number and the data in the block and the previous blocks hash. The resulting hash has to start with a pre-established number of zeroes. 30

30 Proof-of-work Easy to validate but hard to find Miner spends CPU cycles/power/time A single computer = years, Bitcoin network = ~10 minutes Incremental number Header of the most recent block New block Nonce Combine and hash these three Increment nonce Mining difficulty Hash Value Determine Target Value Is Hash < target Mining reward 31

31 Difficulty / target Difficulty A measure of how difficult it is to find a hash below a given target The Bitcoin network has a global block difficulty Valid blocks must have a hash below this target Difficulty changes every 2016 blocks difficulty = difficulty_1_target / current_target difficulty_1_target can be different for various ways to measure difficulty. Traditionally, it represents a hash where the leading 32 bits are zero and the rest are one (this is known as "pool difficulty" or "pdiff") The Bitcoin protocol represents targets as a custom floating point type with limited precision; as a result, Bitcoin clients often approximate difficulty based on this (this is known as "bdiff"). Target is a 256 bit number Not a set problem (like doing a million hashes), but more like a lottery 32

32 Bitcoin address 1454A2geTxaJwF8eqry7oLECdomgDSj6Zx Public Key ( Address ) 34 characters starting with 1 or 3 Represents a possible destination for payment 5JHkYd4mYkTsCsF5axnFj573PG6tqpeJ39Rz2M33v wbka4s1hu6 Private Key 51 characters starting with 5 Required to transfer value from the address 33

33 Bitcoin Address A Bitcoin-balance is associated to the Bitcoin address A Bitcoin Address (2^160 address space) is derived by a ECDSA Public Key by using hash functions 512 bit ECDSA Public Key with prefix SHA-256 of Public Key with prefix RIPEM1 60 of SHA-256 output Base58 Encoding (HASH + Checksum) = 1422cPZaPRiqeWL8njn87NjLwgZxxmZmKp 34

34 Bitcoin architecture 35

35 Messages Simple broadcast network to propagate transactions and blocks. All communications are done over TCP. Bitcoin is fully able to use ports other than 8333 via the -port parameter. IPv6 is supported with Bitcoind/Bitcoin-Qt v0.7. Messages version - Information about program version and block count. Exchanged when first connecting. verack - Sent in response to a version message to acknowledge that we are willing to connect. addr - List of one or more IP addresses and ports. inv - "I have these blocks/transactions:..." Normally sent only when a new block or transaction is being relayed. This is only a list, not the actual data. getdata - Request a single block or transaction by hash. getblocks - Request an inv of all blocks in a range. getheaders - Request a headers message containing all block headers in a range. tx - Send a transaction. This is sent only in response to a getdata request. block - Send a block. This is sent only in response to a getdata request. headers - Send up to 2,000 block headers. Non-generators can download the headers of blocks instead of entire blocks. getaddr - Request an addr message containing a bunch of known-active peers (for bootstrapping). submitorder, checkorder, and reply - Used when performing an IP transaction. ping - Does nothing. Used to check that the connection is still online. A TCP error will occur if the connection has died. 36

36 Bootstrapping The number one way that clients learn about other clients is by connecting to another client and issuing the 'getaddr' command The standard client has always had that ability However, there's a problem with this - how do you learn about client #1? IRC seeding This is something that's been in since the first version too It was simple to implement, but ultimately didn't scale The first version connected to freenode Then, bitcoin nodes started getting k-lined (IRC kill line temporary ban) Later versions connected to lfnet Then, lfnet went down, and IRC support was removed entirely DNS was around by this point IP addresses of well known nodes This has been in the client since June 2010 Still in the Bitcoin client, though the exact IP addresses change every so often. DNS seeding This has been in the client since March 2011 This is easier to scale, because DNS is already built to handle tens of thousands of connections 37

37 New node The new node must form new connections to the network As sometimes some nodes go offline from time to time as people switch their computers off Then the new node is meshed into the network and resilience is created 38

38 Connect To connect to a peer, you send a version message containing: your version number block count current time The remote peer will send back a verack message and his own version message if he is accepting connections from your version You will respond with your own verack if you are accepting connections from his version. The time data from all of your peers is collected, and the median is used by Bitcoin for all network tasks that use the time (except for other version messages). You then exchange getaddr and addr messages storing all addresses that you don't know about addr messages often contain only one address, but sometimes contain up to This is most common at the beginning of an exchange. 39

39 Getaddr The Satoshi client discovers the IP address and port of nodes in several different ways. Nodes discover their own external address by various methods. Nodes receive the callback address of remote nodes that connect to them. Nodes makes DNS request to receive IP addresses. Nodes can use addresses hard coded into the software. Nodes exchange addresses with other nodes. Nodes store addresses in a database and read that database on startup. Nodes can be provided addresses as command line arguments Nodes read addresses from a user provided text file on startup A timestamp is kept for each address to keep track of when the node address was last seen. The AddressCurrentlyConnected in net.cpp handles updating the timestamp whenever a message is received from a node. Handling Message "getaddr" When a node receives a "getaddr" request, it first figures out how many addresses it has that have a timestamp in the last 3 hours Then it sends those addresses, but if there are more than 2500 addresses seen in the last 3 hours, it selects around 2500 out of the available recent addresses by using random selection. The specification is unclear in this part 1000 or 2500? 40

40 "addr" advertisements Nodes may receive addresses in an "addr" message after having sent a "getaddr" request, or "addr" message may arrive unsolicited, because nodes advertise addresses gratuitously when they relay addresses when they advertise their own address periodically when a connection is made. If the address is from a really old version, it is ignored if from a not-so-old version it is ignored if we have 1000 addresses already. If the sender sent over 1000 addresses, they are all ignored Addresses received from an "addr" message have a timestamp, but the timestamp is not necessarily honored directly. For every address in the message: Check timestamps Existense of addr record Addition of address that are new and have no timestamp problems 41

41 Initial block download At the start of a connection Send a getblocks message containing the hash of the latest block you know about If the peer doesn't think that this is the latest block it will send an inv that contains up to 500 blocks ahead of the one you listed You will then request all of these blocks with getdata The peer will send them to you with block messages After you have downloaded and processed all of these blocks you will send another getblocks etc., until you have all of the blocks. 43

42 Heartbeat If thirty minutes or more has passed since the client has transmitted any messages it will transmit a message to keep the connection to the peer node alive. If ninety minutes has passed since a peer node has communicated any messages, then the client will assume that connection has closed. 45

43 Complex networks The underlying network structure of the bitcoin network Bitcoins, represented as addresses, form nodes, and edges correspond to transactions between them This makes the bitcoin interaction network into a graph, which is something we can quantify and study D. Kondor, M. Pósfai, I. Csabai, G. Vattay Do the Rich Get Richer? An Empirical Analysis of the Bitcoin Transaction Network PLoS ONE 9(2): e86197, 2014 D. Di Francesco Maesa, A. Marino, L. Ricci Data-driven analysis of Bitcoin properties: exploiting the users graph Int J Data Sci Anal, 2017 Their findings confirm that the bitcoin network is another example of a complex network which are large and evolve over time Examples of complex networks range from the web graph, social networks, and protein interaction networks Two key features of complex networks are: The small world property Power law degree distributions For the small world property The network should have short paths connecting randomly selected nodes High local clustering, so pairs of nodes with a common neighbor are more likely adjacent Power law degree distributions point to the networks having an undemocratic character Most nodes have the low degree, but some have high degree In other words, there is a small group of nodes which have most of the links The power law indegree distribution of the bitcoin network, from Do the Rich Get Richer? An Empirical Analysis of the Bitcoin Transaction Network. These properties and others were uncovered for bitcoin networks and networks of bitcoin users An interesting mathematical question is whether bitcoin networks are best fit by established network models like preferential attachement, or by other, as yet undiscovered models. 46

44 The power law indegree distribution of the bitcoin network Do the Rich Get Richer? An Empirical Analysis of the Bitcoin Transaction Network. 47