Network Coordinates in the Wild

Transcription

1 Network Coordinates in the Wild Jonathan Ledlie Margo Seltzer Paul Gardner Harvard University Aelitis / Azureus Hourglass Project Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 1

2 Overview Introduction Azureus Very popular BitTorrent client million current users worldwide! Locality essential! Uses network coordinates (NCs) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 2

3 Overview Introduction Azureus Very popular BitTorrent client million current users worldwide! Locality essential! Uses network coordinates (NCs) Problem: inaccurate and unstable :-( Developers found our work on PlanetLab [Ledlie 06] Worked with them to analyze and tame their NC system Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 3

4 BitTorrent Introduction Protocol for file sharing Peers don t download from seed (content source) Instead: discover other peers and download from them Upside Users can provide content without bandwidth hit Downside Consumes huge portion of Internet bandwidth (18-55%) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 4

5 BitTorrent: Peer Discovery Introduction 1. Request list of peers from tracker tracker Peers? New node Existing swarm Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 5

6 BitTorrent: Peer Discovery Introduction 1. Request list of peers from tracker 2. Tracker records IP of requester tracker New node Existing swarm Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 6

7 BitTorrent: Peer Discovery Introduction 1. Request list of peers from tracker 2. Tracker records IP of requester 3. Tracker responds with list of recent peers tracker A A,B,C New node Existing swarm B C Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 7

8 BitTorrent: Peer Discovery Introduction 1. Request list of peers from tracker 2. Tracker records IP of requester 3. Tracker responds with list of recent peers 4. Contact tracker A New node Expanded swarm B C Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 8

9 BitTorrent: Locality Matters Introduction ISP #1 Unbiased swarms tracker Locally-biased swarms ISP #1 tracker ISP #2 Improved b/w for peers Reduced inter-isp b/w Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 9 ISP #2

10 Contributions Introduction Evidence Use NCs for locality-aware anycast decisions Developed new techniques Minimize overhead Churn Data from Internet-scale system Large latency matrix from end-users Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 10

11 Outline Introduction A Network Coordinate Refresher Methodology Two Problems and our Solutions Limited Horizon Slow DHT Lookups Conclusion Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 11

12 Network Coordinates NC Refresher Embed inter-node latencies into metric space Measure to (small) subset of network Establish coordinate Predict missing measurements Low dimensional space 2-5 dimensions in practice 2d Euclidean embedding Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 12

13 Vivaldi Refinement Process NC Refresher Incremental refinement: minimize global prediction error * Vivaldi [Cox 03,Dabek 04] A B Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 13

14 Measurement NC Refresher Incremental refinement: minimize global prediction error 1. A measures latency to B. (100,80) A Coord? B Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 14

15 Reply NC Refresher 1. A measures latency to B. 2. B replies with its coord. A deduces RTT. (100,80) A RTT=60ms Coord? (70,40) B Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 15

16 Computation NC Refresher 1. A measures latency to B. 2. B replies with its coord. A deduces RTT. 3. A computes estimate and error. (100,80) A RTT=60ms (70,40) B Estimate = (100,80)-(70-40) =50ms Error = (60 - Estimate) = 10ms Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 16

17 Adjustment NC Refresher 1. A measures latency to B. 2. B replies with its coord. A deduces RTT. 3. A computes estimate and error. 4. A moves toward ideal coord, relative to B. B (103,84) A A 60ms Estimate = (100,80)-(70-40) =50ms Error = (60 - Estimate) = 10ms Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 17

18 Repeat NC Refresher 1. A measures latency to B. 2. B replies with its coord. A deduces RTT. 3. A computes estimate and error. C (103,84) A D 4. A moves toward ideal coord, relative to B. 5. Repeat with C, D, E. B E Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 18

19 Predict NC Refresher A has never seen or measured RTT to X 1. A starts measurement to B. 2. B replies with its coord. A deduces RTT. 3. A computes estimate and error. C (103,84) A D 4. A moves toward ideal coord, relative to B. 5. Repeat with C, D, E. B X E (130,50) 6. Predict to X Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 19

20 Predict NC Refresher A can predict locality of X and Y. 1. A starts measurement to B. 2. B replies with its coord. A deduces RTT. 3. A computes estimate and error. 4. A moves toward ideal coord, relative to B. 5. Repeat with C, D, E. C B A X E D Y 6. Predict to X Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 20

21 Outline Introduction A Network Coordinate Refresher Methodology Two Problems and our Solutions Limited Horizon Slow DHT Lookups Conclusion Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 21

22 Goals: Study and Refine NCs Methodology Twin goals: Study and refine Internet-scale network coordinate system Study Observe and understand causes of inaccurate prediction Refine Test new techniques in simulation / local clients / PL Release Repeat Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 22

23 Two Data Collection Processes Methodology Fine: Instrumented Azureus clients run on PlanetLab Logged every update» real RTT, remote coordinate, remote error, per update error Detailed picture + PL-to-non-PL latency matrix Coarse: Added NC statistics message to Azureus Summarizes their coordinates behavior» instantaneous error, neighbor error, stability» other metrics [Lua 05] Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 23

24 Measurement Challenges Methodology ~1.3 million Azureus clients running at any given moment Impossible to acquire all-to-all latency matrix Complicates simulation and testing Simulated with 249x2902 rectangular matrix Impossible to force clients to run latest version Cannot ensure all clients are running algorithm X, parameter Y 10,000 users regularly (and automatically) update to CVS» >50% within one day; >75% within one month. Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 24

25 Outline Introduction A Network Coordinate Refresher Methodology Two Problems and our Solutions Limited Horizon Slow DHT Lookups Conclusion Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 25

26 Zero Maintenance Limited Horizon Azureus maintains DHT routing table (RT) Heartbeat messages» Frequent: several per minute Tests new nodes for entry into RT» Less frequent: one per 3-5 minutes NC maintenance piggybacks on these messages Good: Zero additional messages! Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 26

27 Problem: Limited Horizon Limited Horizon Bad: Limits view of network to routing table RT (+ neighbor set) biased to nearby nodes Local bias damages accuracy [Dabek 04, Ledlie 05] Creates islands: poor estimation beyond horizon Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 27

28 Strawman Limited Horizon Make tug proportional to distance Boost impact of (occasional) long range contacts Problems What s the right weight? Produces instability Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 28

29 Insight: Expand Horizon over time Limited Horizon Optimize coordinate against (recent) whole network Don t boost all at once; extend period of effect Scale push/pull per neighbor by age Staleness of infomation Decaying tug of neighbor over time: neighbor decay In Effect Limits impact of high frequency (nearby) neighbors Extends impact of low frequency (longer-distance) ones Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 29

30 Original Adjustment Step Limited Horizon A optimizes with respect to B 1. A measures latency to B. 2. B replies with its coord. A deduces RTT. 3. A computes estimate and error. 4. A moves toward ideal coord, relative to B. B (103,84) A A 60ms Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 30

31 Neighbor Decay Adjustment Step Limited Horizon As springs age, they get looser Force F = 0 d for (n : neighbors) { Compute force F n Norm F n by age n 4 minutes old C A A D 2 min F += F n } A += F B E 3min 0min Older information gets less weight Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 31

32 Reading the Neighbor Decay Video Limited Horizon No ND: -300 Red Neighbor Decay Comparison Pct. improvement in accuracy (30-50%) ms Two coords - same node With ND: Green -200 Neighbors: Bluems Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 32 0h 51m Error: 0.472% w/nd no ND Elapsed Time Accuracy ms 3d Euclidean ms x ms x ms

33 Neighbor Decay Video Limited Horizon [First half hour in the life of two coordinates on Azureus with and without neighbor decay] Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 33

34 Outline Introduction A Network Coordinate Refresher Methodology Two Problems and our Solutions Limited Horizon Slow DHT Lookups Conclusion Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 34

35 Improved Accuracy, but End-to-end benefit That s nice that accuracy improves, but Do NCs actually improve app-level performance? Aid in decisions involving node locality? Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 35

36 DHT Traversal: Problem End-to-end benefit Problem: Azureus frequently performs DHT lookups. Need to be as fast as possible! E.g.:» Distributed tracker run within DHT» Want to find these trackers faster Can we use latency prediction to speed up its lookups? Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 36

37 Kademlia s Iterative Lookups End-to-end benefit Target key 0xDABD Sort routing table by distance to T» distance (a,b) = a b Options for next hop key 0xDD1... 0xD53... Kademlia [Maymounkov 02] Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 37

38 Kademlia s Iterative Lookups End-to-end benefit Target key 0xDABD Sort RT by distance to T» distance (a,b) = a b 2. Query logically nearest for nodes near T Options for next hop key 0xDD1... 0xD53... Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 38

39 Kademlia s Iterative Lookups End-to-end benefit Target key 0xDABD Sort RT by distance to T» distance (a,b) = a b 2. Query logically nearest for nodes near T 3. Add responses to list Options for next hop key 0xD9A... 0xD xDD1... 0xD53... Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 39

40 Kademlia s Iterative Lookups End-to-end benefit Target key 0xDABD Sort RT by distance to T» distance (a,b) = a b 2. Query logically nearest for nodes near T 3. Add responses to list 4. Repeat Options for next hop key 0xD9A... 0xD xDD1... 0xD53... Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 40

41 DHT Traversal: Solution End-to-end benefit Target key 0xDABD... Find good trade-off between logical progress and delay Similar opt. in Chord [Dabek 04]. Options for next hop key NC dist. 0xD9A xD xD xD9F xDD xD Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 41

42 DHT Traversal: Solution End-to-end benefit Target key 0xDABD... Find good trade-off between logical progress and delay 1. Eliminate logically distant hops» Make similar progress to target Same number of expected hops if fixing same number of bits Options for next hop key NC dist. 0xD9A xD xD xD9F xDD xD Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 42

43 DHT Traversal: Solution End-to-end benefit Target key 0xDABD... Find good trade-off between logical progress and delay 1. Eliminate logically distant hops» Make similar progress to target 2. Break ties in favor of lowest predicted latency Options for next hop key NC dist. 0xD9A xD xD xD9F xDD xD Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 43

44 DHT Traversal: Evaluation End-to-end benefit Compared four methods: XOR Options for next hop key OptNC OrigNC 0xD9A Optimized Original and Random 0xD xD xD9F xDD xD Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 44

45 DHT Traversal: Evaluation End-to-end benefit Experiment Put 250 random keys into Azureus DHT For each key, performed four lookups (once per method)» XOR, OptNC, OrigNC, Random» Permuted method ordering Repeated 10x Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 45

46 DHT Traversal: Evaluation End-to-end benefit Cumulative Distribution of Lookup Delay (faster is better) 0.2 Faster Slower Lookup Delay (ms) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 46

47 DHT Traversal: Evaluation End-to-end benefit XOR Long delays Median delay: 196ms Lookup Delay (ms) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 47

48 DHT Traversal: Evaluation End-to-end benefit XOR Random Logical detours slow random vs. direct XOR path (23% at median) Lookup Delay (ms) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 48

49 DHT Traversal: Evaluation End-to-end benefit OrigNC Random XOR Good choice possible: Unoptimized NCs reduced delay e.g. 24% vs. XOR Lookup Delay (ms) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 49

50 1.0 DHT Traversal: Evaluation End-to-end benefit OptNC OrigNC Random OptNC: higher accuracy, faster lookups (e.g. 33% vs XOR, 12% vs OrigNC) 0.2 XOR Lookup Delay (ms) Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 50

51 DHT Traversal: Evaluation End-to-end benefit OptNC OrigNC XOR Takeaway Coordinates aided locality-aware anycast decisions Lookup Delay (ms) Random Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 51

52 Conclusion Methodology Coarse and fine statistics Limited Horizon Problem Solution: neighbor decay Locality-aware anycast -style decisions DHT traversal Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 52

53 Thanks & Pyxida Check out Pyxida: Same code base Network Coordinate Service for PlanetLab Library Implementation of NC techniques Links to data Thanks. Any Questions? Jonathan Ledlie Jonathan Ledlie - Harvard University - NSDI - April 07 - Slide 53