SoNIC: Precise Real1me So3ware Access and Control of Wired Networks. Ki Suh Lee, Han Wang, Hakim Weatherspoon Cornell University

Transcription

1 SoNIC: Precise Real1me So3ware Access and Control of Wired s Ki Suh Lee, Han Wang, Hakim Weatherspoon Cornell University 4/11/13 SoNIC NSDI

2 Interpacket Delay and Research Link Interpacket gap, spacing, arrival 1me, Important metric for network research Packet Genera1on Can be improved with access to the PHY Increasing Throughput IPG Packet i Packet i+1 IPD Detec1ng 1ming channel Packet Capture Characteriza1on Es1ma1ng bandwidth 4/11/13 SoNIC NSDI

3 Research enlightened via the PHY Link Valuable informa1on: Idle characters IPG Packet i Packet i+1 IPD Can provide precise 1ming base for control Each bit is ~97 ps wide 4/11/13 SoNIC NSDI

4 Research enlightened via the PHY Link Valuable informa1on: 12 /I/s Idle = 100bits characters = 9.7ns IPG Packet i Packet i+1 One Idle character (/I/) = 7~8 bits Each bit is ~97 ps wide Can provide precise 1ming base for control Packet Genera1on Detec1ng 1ming channel Packet Capture 4/11/13 SoNIC NSDI

5 Principle #1: Precision Precise network measurements is enabled via access to the physical layer (and the idle characters and bits within interpacket gap) 4/11/13 SoNIC NSDI

6 How to control the idle characters (bits)? Link Access to the en1re stream is required IPG Packet i Packet i+1 Issue1: The PHY is simply a black box No interface from NIC or OS Valuable informa1on is invisible (discarded) Packet i Packet i+1 Packet i+2 Packet i Packet i+1 Issue2: Limited access to hardware We are network systems researchers a.k.a. we like so3ware Packet i+2 4/11/13 SoNIC NSDI

7 Principle #2: So3ware Systems researchers need so3ware access to the physical layer 4/11/13 SoNIC NSDI

8 Precision + So3ware = Physics equipment??? BiFocals [IMC 10 Freedman, Marian, Lee, Birman, Weatherspoon, Xu] Enabled novel network research Precision + So3ware = Laser + Oscilloscope + Offline analysis Allowed precise control in so3ware Limita1ons Offline (not real'me) Limited Buffering Expensive 4/11/13 SoNIC NSDI

9 Principle #3: Real1me systems researchers need access and control of the physical layer (interpacket gap) con1nuously in real1me 4/11/13 SoNIC NSDI

10 Challenge Link Goal: Control every bit in so-ware in real'me IPG Packet i Packet i+1 IPD Enable novel network research Challenge Requires unprecedented so3ware access to the PHY 4/11/13 SoNIC NSDI

11 Outline Introduc1on SoNIC: So3ware- defined Interface Card Background: 10GbE Stack Design Research s Conclusion 4/11/13 SoNIC NSDI

12 SoNIC: So3ware- defined Interface Card Implements the PHY in so3ware IPG Link Enabling control and access to every bit in real1me With commodity components Thus, enabling novel network research How? Packet i Packet i+1 IPD Backgrounds: 10 GbE stack Design and implementa1on Hardware & So3ware Op1miza1ons 4/11/13 SoNIC NSDI

13 10GbE Stack L3 Hdr L2 Hdr L3 Hdr Link Preamble Eth Hdr L2 Hdr L3 Hdr CRC Gap 64/66b PCS Encode Decode 64 bit Idle 2 bit characters syncheader (/I/) Gigabits /S/ /D/ /D/ /D/ /D/ /T/ /E/ Scrambler Descrambler 16 bit Gearbox Blocksync PMA PMD /11/13 SoNIC NSDI

14 10GbE Stack SW L2 Hdr L3 Hdr Packet i Packet i+1 L3 Hdr Link Preamble Eth Hdr L2 Hdr L3 Hdr CRC Gap 64/66b PCS Encode Decode HW /S/ /D/ /D/ /D/ /D/ /T/ /E/ Scrambler Descrambler Packet i Packet i+1 Gearbox Blocksync PMA PMD Commodity NIC 4/11/13 SoNIC NSDI

15 10GbE Stack L3 Hdr SW Link Preamble Eth Hdr L2 Hdr L2 Hdr L3 Hdr L3 Hdr HW CRC Link Gap 64/66b PCS Encode Decode Scrambler Descrambler SW 64/66b PCS /S/ Packet i /D/ /D/ Packet /D/ i+1 /D/ Encode /T/ Decode /E/ Scrambler Descrambler Gearbox Blocksync PMA PMD HW SoNIC NetFPGA Gearbox Blocksync PMA PMD 4/11/13 SoNIC NSDI

16 SoNIC Design L3 Hdr L2 Hdr L3 Hdr Link Preamble Eth Hdr L2 Hdr L3 Hdr CRC Gap 64/66b PCS Encode Decode Scrambler Descrambler SW /S/ /D/ /D/ /D/ /D/ /T/ /E/ Gearbox Blocksync PMA PMD HW SoNIC 4/11/13 SoNIC NSDI

17 SoNIC Design and Architecture L3 Hdr APP Userspace L2 Hdr L3 Hdr APP Kernel Link Preamble Eth Hdr L2 Hdr TX MAC L3 Hdr RX MAC CRC Gap 64/66b PCS Encode Decode Scrambler Descrambler SW /S/ /D/ /D/ /D/ /D/ /T/ /E/ TX PCS RX PCS Gearbox Blocksync PMA PMD HW Transceiver Transceiver SoNIC Gearbox Blocksync Hardware SFP+ 4/11/13 SoNIC NSDI

18 SoNIC Design: Hardware 64/66b PCS Encode Scrambler Gearbox Link PMA PMD Decode Descrambler Blocksync SW HW To deliver every bit from/to so3ware High- speed transceivers PCIe Gen2 (=32Gbps) Op1mized DMA engine SFP+ SFP+ FPGA PCIeGen2 4/11/13 SoNIC NSDI

19 SoNIC Design: So3ware Port 0 APP Port 1 APP Link TX MAC RX MAC TX MAC RX MAC 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA PMD SW HW TX PCS RX PCS TX PCS RX PCS Dedicated Kernel Threads TX / RX PCS, TX / RX MAC threads APP thread: Interface to userspace Packet i Packet i+1 4/11/13 SoNIC NSDI

20 SoNIC Design: Synchroniza1on Port 0 APP Low- latency FIFOs Port 1 APP Link TX MAC RX MAC TX MAC RX MAC 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA SW HW TX PCS SFP+ SFP+ RX PCS FPGA TX PCS RX PCS Pointer- polling No Interrupts PMD PCIeGen2 4/11/13 SoNIC NSDI

21 SoNIC Design: Op1miza1ons 64/66b PCS Encode Scrambler Gearbox Link PMA PMD Decode Descrambler Blocksync Scrambler Naïve Implementa1on s ç state d ç data for i = 0 è 63 do in ç (d >> i) & 1 out ç (in (s >> 38) (s >> 57))&1 s ç (s << 1) out r ç r (out << i) state ç s end for CRC computa1on DMA engine G( x) = x58+ x Gbps 21 Gbps Op1mized Implementa1on s ç state d ç data r ç (s >> 6) (s >> 25) d r ç r (r << 39) (r << 58) state ç r 4/11/13 SoNIC NSDI

22 SoNIC Design: Interface and Control Hardware control: ioctl syscall I/O : character device interface Sample C code for packet genera1on and capture 1: #include "sonic.h" 2: 3: struct sonic_pkt_gen_info info = { 4:.mode = 0, 5:.pkt_num = UL, 6:.pkt_len = 1518, 7:.mac_src = "00:11:22:33:44:55", 8:.mac_dst = "aa:bb:cc:dd:ee:ff", 9:.ip_src = " ", 10:.ip_dst = " ", 11:.port_src = 5000, 12:.port_dst = 5000, 13:.idle = 12, 14: }; 15: 16: /* OPEN DEVICE*/ 17: fd1 = open(sonic_control_path, O_RDWR); 18: fd2 = open(sonic_port1_path, O_RDONLY); 19: /* CONFIG SONIC CARD FOR PACKET GEN*/ 20: ioctl(fd1, SONIC_IOC_RESET) 21: ioctl(fd1, SONIC_IOC_SET_MODE, PKT_GEN_CAP) 22: ioctl(fd1, SONIC_IOC_PORT0_INFO_SET, &info) 23 24: /* START EXPERIMENT*/ 25: ioctl(fd1, SONIC_IOC_START) 26: // wait 1ll experiment finishes 27: ioctl(fd1, SONIC_IOC_STOP) 28: 29: /* CAPTURE PACKET */ 30: while ((ret = read(fd2, buf, 65536)) > 0) { 31: // process data 32: } 33: 34: close(fd1); 35: close(fd2); 4/11/13 SoNIC 22

23 Outline Introduc1on SoNIC: So3ware- defined Interface Card Research s Packet Genera1on Packet Capture Covert 1ming channel Conclusion 4/11/13 SoNIC NSDI

24 Research s Link Interpacket delays and gaps IPG Packet i Packet i+1 IPD Packet Genera1on Detec1ng 1ming channel Packet Capture 4/11/13 SoNIC NSDI

25 Packet Genera1on and Capture Basic func1ons for network research Genera1on: SoNIC allows control of IPGs in # of /I/s Capture: SoNIC captures what was sent with IPGs in bits APP APP TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS TX PCS RX PCS 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 4/11/13 SoNIC NSDI

26 Packet Genera1on SoNIC allows precise CDF of control generated of IPGs IPDs 1" TX MAC TX PCS APP RX MAC RX PCS CDF 0.8" 0.6" 0.4" 0.2" Specialized NIC Higher variance SoNIC Zero variance!!! SoNIC" Sniffer"10G" TX MAC TX PCS APP RX MAC RX PCS 0" 1000" 1500" 2000" 2500" 3000" Interpacket delays (ns) 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 4/11/13 SoNIC NSDI

27 Packet Capture SoNIC captures what CDF of is sent captured IPDs 1" TX MAC APP RX MAC CDF 0.8" 0.6" 0.4" SoNIC" Kernel" Userspace" Sniffer"10G" TX MAC APP RX MAC TX PCS RX PCS 0.2" TX PCS RX PCS 0" 0" 1000" 2000" 3000" 4000" 5000" Interpacket delays (ns) 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 4/11/13 SoNIC NSDI

28 Covert Timing Channel Embedding signals into interpacket gaps. Large gap: 1 Small gap: 0 Covert 1ming channel by modula1ng IPGs at 100ns TX MAC TX PCS APP RX MAC RX PCS Packet i Packet i+1 Packet i Packet i+1 Overt channel at 3 Gbps Covert channel at 250 kbps Over 4- hops with < 1% BER TX MAC TX PCS APP RX MAC RX PCS 4/11/13 SoNIC NSDI

29 Covert Timing Channel Modula'ng IPGS at 100ns scale (=128 /I/s) CDF 1" SoNIC" 3562 /I/s 0.8" Kernel" /I/s /I/s 0.6" BER = 0.37% TX MAC APP RX MAC 0.4" TX MAC APP RX MAC TX PCS RX PCS 0.2" 0 1 TX PCS RX PCS 0" 500" 1500" 2500" 3500" Interpacket delays (ns) 4500" 1 : /I/s 0 : /I/s 1 : a /I/s 0 : 3562 a /I/s 4/11/13 SoNIC NSDI

30 Contribu1ons Research Unprecedented access to the PHY with commodity hardware A pla orm for cross- network- layer research Can improve network research applica1ons Engineering Precise control of interpacket gaps (delays) Design and implementa1on of the PHY in so3ware Novel scalable hardware design Op1miza1ons / Parallelism Status Measurements in large scale: DCN, GENI, 40 GbE 4/11/13 SoNIC NSDI

31 Conclusion Precise Real1me So3ware Access to the PHY Commodity components An FPGA development board, Intel architecture applica1ons measurements characteriza1on steganography Webpage: h p://sonic.cs.cornell.edu SoNIC is available Open Source. 4/11/13 SoNIC NSDI

32 Thank you Demo tonight! h p://sonic.cs.cornell.edu