A Virtual Circuit Multicast Transport Protocol (VCMTP) for Scientific Data Distribution Jie Li and Malathi Veeraraghavan University of Virginia Steve Emmerson University Corporation for Atmospheric Research Robert D. Russell University of New Hampshire April 23, 2013 This work was supported by the NSF grants OCI-1038058 and OCI-1127340, and DOE grants DE-SC002350 and DESC0007341 1
I. Case Study of A Scientific Data Distribution Application 2
Background Internet Data Distribution (IDD) Project Developed by University Corporation for Atmospheric Research (UCAR) Distributes real-time meteorology data 10 GB/hour data generation rate Subscriber base: 170 institutions Software used for distribution: Local Data Manager (LDM) 3
Question Which of these network services is best suited for IDD data? IP routed service + unicast TCP (current mechanism) Static circuits (leased lines) if continuous data flow, is this an option? Scheduled dynamic circuit service (DCS) P2P if data flow is long-lived, option? Multicast 4
To answer this question Per-flow data characteristics insufficient typical classification: loss-sensitive, high throughput delay-sensitive, low latency Instead, need distribution topology consider whole network view 5
CONDUIT data Installed and configured the LDM to receive CONDUIT data from UCAR Parsed and analyzed the log files for received data(9 sample days) Peak throughput: 250 MB/minute (SD: 28.8 MB/minute) Total size of generated data: ~60 GB/day (SD: 0.3 GB/day) 6
Distribution structure Downloaded and parsed real-time statistics of the CONDUIT feed tree Data Distribution Topology of the CONDUIT feedtype For the max fan-out of 104 receivers, the peak bandwidth requirement is 104 * 250 MB/minute 3.5Gbps This is just for a single feedtype of a single application CONDUIT Feed Tree Topology Information Parameter Number # Distinct Hosts 163 # Root Hosts 22 # Intermediate Hosts 35 # Leaf Hosts 106 Max. Fan-out Number 104* * This maximum fan-out number is for the UCAR site (idd.unidata.ucar.edu)
CONDUIT distribution topology http://www.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?conduit 8
Answer to question Different network service types Static unicast VCs: unsuitable Divide NCAR access link bandwidth between 104 subscribers: if 10 Gbps, then ~10 Mbps per subscriber Subscribers would like to receive the data asap (low rate VC will increase latency) Dynamic unicast VCs: unsuitable For the worst-case fanout of 104, the total delay will be greater than with IP service, since for each receiver a new circuit needs to be set up, which can only be done after the transfer to the previous receiver is complete and the circuit to that receiver is released. Multicast: can save bandwidth and computing resource 9
New options: multicast and P2P Multicast P2P Pros: total delay for distributing the data to the receivers will be lower for a given computing capacity of the upstream servers, or conversely, the same transfer delay can be achieved as with IP-routed service or P2P but with smaller upstream server computing capacity. Cons: one or more slow receivers can slow down everyone Pros: scales better with the number of receivers; suitable when files are obtained by different participants at different times Cons: not suitable for real-time or near real-time delivery (which is a key requirement of IDD) 10
II. VCMTP: Design and Prototyping 11
VCMTP Requirements Goal: Design and implement a reliable and scalable transport protocol for data distribution over high-speed multipoint virtual circuits Requirements Reliability: error control, flow control Scalability: support at least hundreds of receivers High-speed multicast: support Gbps transfers 12
VCMTP Operational Overview A Negative Acknowledgment (NACK) based reliable transport protocol Data blocks transmitted over a multicast network service (can be unreliable) Retransmissions carried over a reliable unicast service (e.g., TCP) 13
VCMTP Prototyping A user-level library implemented in C++ for Linux OS environment Asynchronous programming model Simultaneous data multicast and retransmission VCMTP Sender Process Sending Thread Retransmission Thread 1 Retransmission Thread N Coordinator Thread Receiving Thread Retransmission Request Thread Receiving Thread Retransmission Request Thread VCMTP Receiver Process 1 VCMTP Receiver Process N 14
Evaluation Metrics for Continuous File Transfers Metric for fast receivers: Throughput n f : number of fast receivers m: number of continuously sent files F i : size of file i T i,vcmtp : transfer time for file i Metric for slow receivers: Robustness n s : number of slow receivers m: number of continuous files S ij : an indicator variable that is set to 1 if file i was successfully received at receiver j, or 0 otherwise 15
Experimental Evaluation: Throughput Experiments conducted in the Emulab testbed (hosted by Univ. of Utah) 40% slow receivers experienced random packet drops at different rates Rho is the traffic intensity calculated from the average file size (Pareto distribution) and interarrival time (exponential distribution); link rate = 100 Mbps Experiment: 500 files; repeat 5 times 16
Experimental Evaluation: Robustness 17
Key Evaluation Observations Increase in total number of receivers (and hence number of slow receivers) has adverse impact on both robustness and throughput because of resource contention Both robustness and throughput decrease as traffic intensity (Rho) or loss rate increases The sending-side retransmission timeout factor offers a knob for trading off robustness against throughput 18
Summary Multicast VCs are suitable for scientific data distribution applications VCMTP: a reliable multicast transport protocol is designed, prototyped, and evaluated Tradeoff between robustness and throughput for continuous file delivery 19
Thank You! & Questions? 20