Data Center Traffic and Measurements: SoNIC

Transcription

1 Center Traffic and Measurements: SoNIC Hakim Weatherspoon Assistant Professor, Dept of Computer Science CS 5413: High Performance Systems and ing November 12, 2014 Slides from USENIX symposium on ed Systems Design and Implementation (NSDI) 2013 presentation of SoNIC: Precise Realtime Software Access and Control of Wired s,

2 Goals for Today Analysis and Traffic Characteristics of Centers in the wild T. Benson, A. Akella, and D. A. Maltz. In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement (IMC), pp ACM, 2010.

3 Interpacket Delay and Research Link Interpacket gap, spacing, arrival time, Important metric for network research Packet Generation Can be improved with access to the PHY Increasing Throughput IPG Packet i Packet i+1 IPD Detecting timing channel Packet Capture Characterization Estimating bandwidth 11/12/2014 SoNIC NSDI

4 Research enlightened via the PHY Link Valuable information: Idle characters IPG Packet i Packet i+1 IPD Can provide precise timing base for control Each bit is ~97 ps wide 11/12/2014 SoNIC NSDI

5 Research enlightened via the PHY Link Valuable information: 12 /I/s Idle = 100bits characters = 9.7ns IPG Packet i Packet i+1 One Idle character (/I/) = 7~8 bits Can provide precise timing base for control Each bit is ~97 ps wide Packet Generation Detecting timing channel Packet Capture 11/12/2014 SoNIC NSDI

6 Principle #1: Precision Precise network measurements is enabled via access to the physical layer (and the idle characters and bits within interpacket gap) 11/12/2014 SoNIC NSDI

7 How to control the idle characters (bits)? Link Access to the entire stream is required IPG Packet i Packet i+1 Issue1: The PHY is simply a black box No interface from NIC or OS Valuable information is invisible (discarded) Packet i Packet i+1 Packet i+2 Packet i Packet i+1 Issue2: Limited access to hardware We are network systems researchers a.k.a. we like software Packet i+2 11/12/2014 SoNIC NSDI

8 Principle #2: Software Systems researchers need software access to the physical layer 11/12/2014 SoNIC NSDI

9 Precision + Software = Physics equipment??? BiFocals [IMC 10 Freedman, Marian, Lee, Birman, Weatherspoon, Xu] Enabled novel network research Precision + Software = Laser + Oscilloscope + Offline analysis Allowed precise control in software Limitations Offline (not realtime) Limited Buffering Expensive 11/12/2014 SoNIC NSDI

10 Principle #3: Realtime systems researchers need access and control of the physical layer (interpacket gap) continuously in realtime 11/12/2014 SoNIC NSDI

11 Challenge Link Goal: Control every bit in software in realtime IPG Packet i Packet i+1 IPD Enable novel network research Challenge Requires unprecedented software access to the PHY 11/12/2014 SoNIC NSDI

12 Outline Introduction SoNIC: Software-defined Interface Card Background: 10GbE Stack Design Research s Conclusion 11/12/2014 SoNIC NSDI

13 SoNIC: Software-defined Interface Card Implements the PHY in software IPG Link Enabling control and access to every bit in realtime With commodity components Thus, enabling novel network research How? Packet i Packet i+1 IPD Backgrounds: 10 GbE stack Design and implementation Hardware & Software Optimizations 11/12/2014 SoNIC NSDI

14 10GbE Stack L3 Hdr L2 Hdr L3 Hdr Link Preamble Eth Hdr L2 Hdr L3 Hdr CRC Gap 64/66b PCS Encode Decode 64 bit Idle 2 bit characters syncheader (/I/) Gigabits /S/ /D/ /D/ /D/ /D/ /T/ /E/ Scrambler Descrambler 16 bit Gearbox Blocksync PMA PMD /12/2014 SoNIC NSDI

15 10GbE Stack SW L2 Hdr L3 Hdr Packet i Packet i+1 L3 Hdr Link Preamble Eth Hdr L2 Hdr L3 Hdr CRC Gap 64/66b PCS Encode Decode HW /S/ /D/ /D/ /D/ /D/ /T/ /E/ Scrambler Descrambler Packet i Packet i+1 Gearbox Blocksync PMA PMD Commodity NIC 11/12/2014 SoNIC NSDI

16 10GbE Stack L3 Hdr SW Link Preamble Eth Hdr L2 Hdr L2 Hdr L3 Hdr L3 Hdr HW CRC Link Gap 64/66b PCS Encode Decode Scrambler Descrambler SW 64/66b PCS /S/ Packet i /D/ /D/ Packet /D/ i+1 /D/ Encode /T/ Decode /E/ Scrambler Descrambler Gearbox Blocksync PMA PMD HW SoNIC NetFPGA Gearbox Blocksync PMA PMD 11/12/2014 SoNIC NSDI

17 SoNIC Design L3 Hdr L2 Hdr L3 Hdr Link Preamble Eth Hdr L2 Hdr L3 Hdr CRC Gap 64/66b PCS Encode Decode Scrambler Descrambler SW /S/ /D/ /D/ /D/ /D/ /T/ /E/ Gearbox Blocksync PMA PMD HW SoNIC 11/12/2014 SoNIC NSDI

18 SoNIC Design and Architecture L3 Hdr APP Userspace L2 Hdr L3 Hdr APP Kernel Link Preamble Eth Hdr L2 Hdr TX MAC L3 Hdr RX MAC CRC Gap 64/66b PCS Encode Decode Scrambler Descrambler SW /S/ /D/ /D/ /D/ /D/ /T/ /E/ TX PCS RX PCS Gearbox Blocksync PMA PMD HW Transceiver Transceiver SoNIC Gearbox Blocksync Hardware SFP+ 11/12/2014 SoNIC NSDI

19 SoNIC Design: Hardware Link 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA PMD SW HW To deliver every bit from/to software High-speed transceivers PCIe Gen2 (=32Gbps) Optimized DMA engine SFP+ SFP+ FPGA PCIeGen2 11/12/2014 SoNIC NSDI

20 SoNIC Design: Software Port 0 APP Port 1 APP Link TX MAC RX MAC TX MAC RX MAC 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA PMD SW HW TX PCS RX PCS TX PCS RX PCS Dedicated Kernel Threads TX / RX PCS, TX / RX MAC threads APP thread: Interface to userspace Packet i Packet i+1 11/12/2014 SoNIC NSDI

21 SoNIC Design: Synchronization Port 0 APP Low-latency FIFOs Port 1 APP Link TX MAC RX MAC TX MAC RX MAC 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA SW HW TX PCS SFP+ SFP+ RX PCS FPGA TX PCS RX PCS Pointer-polling No Interrupts PMD PCIeGen2 11/12/2014 SoNIC NSDI

22 SoNIC Design: Optimizations 64/66b PCS Encode Scrambler Gearbox Link PMA PMD Decode Descrambler Blocksync Scrambler Naïve Implementation s state d data for i = 0 63 do in (d >> i) & 1 out (in (s >> 38) (s >> 57))&1 s (s << 1) out r r (out << i) state s end for CRC computation DMA engine G( x) = x58+ x Gbps 21 Gbps Optimized Implementation s state d data r (s >> 6) (s >> 25) d r r (r << 39) (r << 58) state r 11/12/2014 SoNIC NSDI

23 SoNIC Design: Interface and Control Hardware control: ioctl syscall I/O : character device interface Sample C code for packet generation and capture 1: #include "sonic.h" 2: 3: struct sonic_pkt_gen_info info = { 4:.mode = 0, 5:.pkt_num = UL, 6:.pkt_len = 1518, 7:.mac_src = "00:11:22:33:44:55", 8:.mac_dst = "aa:bb:cc:dd:ee:ff", 9:.ip_src = " ", 10:.ip_dst = " ", 11:.port_src = 5000, 12:.port_dst = 5000, 13:.idle = 12, 14: }; 15: 16: /* OPEN DEVICE*/ 17: fd1 = open(sonic_control_path, O_RDWR); 18: fd2 = open(sonic_port1_path, O_RDONLY); 19: /* CONFIG SONIC CARD FOR PACKET GEN*/ 20: ioctl(fd1, SONIC_IOC_RESET) 21: ioctl(fd1, SONIC_IOC_SET_MODE, PKT_GEN_CAP) 22: ioctl(fd1, SONIC_IOC_PORT0_INFO_SET, &info) 23 24: /* START EXPERIMENT*/ 25: ioctl(fd1, SONIC_IOC_START) 26: // wait till experiment finishes 27: ioctl(fd1, SONIC_IOC_STOP) 28: 29: /* CAPTURE PACKET */ 30: while ((ret = read(fd2, buf, 65536)) > 0) { 31: // process data 32: } 33: 34: close(fd1); 35: close(fd2); 11/12/2014 SoNIC 24

24 Outline Introduction SoNIC: Software-defined Interface Card Research s Packet Generation Packet Capture Covert timing channel Conclusion 11/12/2014 SoNIC NSDI

25 Research s Link Interpacket delays and gaps IPG Packet i Packet i+1 IPD Packet Generation Detecting timing channel Packet Capture 11/12/2014 SoNIC NSDI

26 Packet Generation and Capture Basic functions for network research Generation: SoNIC allows control of IPGs in # of /I/s Capture: SoNIC captures what was sent with IPGs in bits APP APP TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS TX PCS RX PCS 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 11/12/2014 SoNIC NSDI

27 Packet Generation SoNIC allows precise CDF of control generated of IPGs IPDs APP CDF Specialized NIC Higher variance APP TX MAC TX PCS RX MAC RX PCS SoNIC Zero variance!!! TX MAC TX PCS RX MAC RX PCS Interpacket delays (ns) 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 11/12/2014 SoNIC NSDI

28 Packet Capture SoNIC captures what CDF of is sent captured IPDs CDF APP APP TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS TX PCS RX PCS Interpacket delays (ns) 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 11/12/2014 SoNIC NSDI

29 Covert Timing Channel Embedding signals into interpacket gaps. Large gap: 1 Small gap: 0 Covert timing channel by modulating IPGs at 100ns TX MAC TX PCS APP RX MAC RX PCS Packet i Packet i+1 Packet i Packet i+1 Overt channel at 3 Gbps Covert channel at 250 kbps Over 4-hops with < 1% BER TX MAC TX PCS APP RX MAC RX PCS 11/12/2014 SoNIC NSDI

30 Covert Timing Channel Modulating IPGS at 100ns scale (=128 /I/s) 3562 /I/s CDF /I/s /I/s BER = 0.37% APP APP TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS 0 1 TX PCS RX PCS Interpacket delays (ns) 1 : /I/s 0 : /I/s 1 : a /I/s 0 : 3562 a /I/s 11/12/2014 SoNIC NSDI

31 Contributions Research Unprecedented access to the PHY with commodity hardware A platform for cross-network-layer research Can improve network research applications Engineering Precise control of interpacket gaps (delays) Design and implementation of the PHY in software Novel scalable hardware design Optimizations / Parallelism Status Measurements in large scale: DCN, GENI, 40 GbE 11/12/2014 SoNIC NSDI

32 Conclusion Precise Realtime Software Access to the PHY Commodity components An FPGA development board, Intel architecture applications measurements characterization steganography Webpage: SoNIC is available Open Source. 11/12/2014 SoNIC NSDI

33 Before Next time Project Interim report Due Monday, November 24. And meet with groups, TA, and professor Fractus Upgrade: Should be back online Required review and reading for Friday, November 14 Timing is Everything: Accurate, Minimum Overhead, Available Bandwidth Estimation in High-speed Wired s, H. Wang, K. Lee, E. Li, C. L. Lim, A. Tang, and H. Weatherspoon. ACM SIGCOMM Internet Measurement Conference (IMC), November Check piazza: Check website for updated schedule