Corda Performance To infinity and beyond! James Carlyle Chief Engineer, R3 7 March 2018

Transcription

1 Corda Performance To infinity and beyond! James Carlyle Chief Engineer, R3 7 March 2018

2 Performance Matters! Why does it matter? Ultimate capacity of network Efficiency and costs of infrastructure But. Throughput matters more than latency We measure TPS (transactions per second)

3 Finding bottlenecks MONIAC provided a model of national economy with a hydraulic computer Showed physically how the worst bottleneck in the system determines overall flow We re looking for bottlenecks across the Corda landscape

4 Corda s design affects performance 7. The notary cluster orders and finalizes the transaction NOTARY CLUSTER 6. Alice sends the signed transaction to the notary cluster 1. Alice creates a new transaction proposal and signs it ALICE 5. Alice verifies the transaction and checks Bob s signature 4. Bob sends back the transaction and his signature to Alice 2. Alice sends the proposal and her signature to Bob 3. Bob inspects the proposal, verifies it and then signs it BOB Uninvolved peers do not receive any of Alice s or Bob s transactions

5 Foundations of performance Performance depends on interaction of 4 areas Your application (not in scope) Notary cluster Messaging Node

6 Performance is multi-dimensional Performance varies by contract and flow complexity Notary type (RAFT/BFT) and configuration Infrastructure configuration Load duration Network size Load

7 Making performance a science We need to: Build a framework and repeatable steps Isolate parts of the system under test Change one thing at a time Measure scientifically Record our results Photo credit: David Nadlinger - University of Oxford

8 Node performance Parallel processing allows the CPU to work on other tasks even if one is blocked, and to take advantage of many CPU cores Each Corda flow moves forward independently of each other Flows are not aware of others running at the same time Corda s design removes contention from ledger updates

9 Node performance the journey not just about multithreading Date Optimization steps Issues Rate 8 Jan Multithreading, modest hardware. 4 cores, 8 GB RAM = 4GB JVM heap, 250 DTU Saturated network to DB, due to sender ID and duplicate message check queries. Missing attachment caching. 120 TPS 15 Jan Caching in NetworkMapCache, Optimized message de-duplication fast path Excessive database writes hitting Azure SQL quota limits, due to flow checkpointing 22 Jan Checkpoint optimization -66% Checkpoint write frequency -70% (Artemis replay). Unregistered scheduled states, random number generation profiled to be heavily contentious (45% wall time spent waiting) 200 TPS 29 Jan Attachment caching, replaced RNG. Profiling of EDSA library shows contention. Artemis is deadlocked with many threads 300 TPS 5 Feb EdDSA fork/fix. Reduced RPC threads. Compression support for storage serialization context 24 cores, 2800 DTU. Contention of security providers and factories. Focus moving to messaging TPS

10 Messaging performance We don t want a fast car on a slow road!

11 Gossip-based performance The next-neighbor gossip protocols of Ethereum and Hyperledger Fabric will deliver all messages, but only eventually Finance applications need deterministic performance that is under the control of just the participants directly involved

12 Corda messaging performance Corda uses point to point messaging over the internet In practice, messaging is complex: there are many protocol and security layers, serialisation of data, dynamic setting up of queues and communication sessions

13 Messaging performance the journey Date Optimization steps Issues Rate 8 Jan Multithreading, modest hardware. 4 cores, 8 GB RAM = 4GB JVM heap, 250 DTU Very limited performance initially. Inefficient sender ID and duplicate message checks. 3 TPS 15 Jan Optimized message de-duplication fast path. AMQP bridges in their current form sustain approximately TPS maximum. Output buffer of encrypt/decrypt messages is not big enough. 12 TPS 22 Jan Skip checkpoints on receive, and rely on the Artemis replay. Core bridges exceed AMQP bridges. 37 TPS 29 Jan Deployed OpenSSL to avoid the ShortBufferExceptions No improvement; OpenSSL deemed insecure. Bug in Artemis blocks executors. 5 Feb Manage Artemis to RPC threads. Progress in messaging 128 flow threads and 200 RPC clients, CPU utilization still low. 85 TPS

14 Notary cluster performance Notary-signing marks the finality of many flows We can have lots of parallel notary clusters, but don t want too many because of managing trust We seek to maximize the ratio of nodes to notaries by having very high throughput notaries

15 Notary performance test steps Measure throughput against: node / network size transaction size notarized transaction count Leave running for months (soak test) Take a notary cluster member offline, check resynchronization Introduce new cluster member, check synchronization Recover notary cluster from backup and transaction logs

16 Notary design initial steps Early focus on RAFT-like consensus, backed by different stores Rocks DB 125 TPS Cockroach DB 25 TPS Permazen DB 17 TPS Percona XtraDB 80 TPS

17 The Corda network notary cluster 175 TPS RAFT-like consensus, Percona XtraDB 5 cluster members, 5 regions Disk IO latency >> networking latency

18 Our best notary cluster (so far) Thousands of TPS RAFT consensus, 3 cluster members Kafka distributed log, RocksDB persistent index Uniform performance across members and time Experimental stage

19 The goal : balanced performance End of 2018: nodes at 1000 TPS, messaging that will keep up, a notary cluster to support 200 mid-nodes at full speed A network capacity of millions of transactions a day

20 ありがとう