Atacama: An Open Experimental Platform for Mixed-Criticality Networking on Top of Ethernet

Transcription

1 Atacama: An Open Experimental Platform for Mixed-Criticality Networking on Top of Ethernet Gonzalo Carvajal 1,2 and Sebastian Fischmeister 1 1 University of Waterloo, ON, Canada 2 Universidad de Concepcion, Chile 1 st Workshop on Real-Time Ethernet SS

2 Real-Time Ethernet Real Time Station Best Effort Station Switch Guaranteed latency between real-time stations Require coordination mechanisms to prevent competition 2

3 State of the Art on E More than 20 years of research From the real-time community Timing models and analysis of COTS Ethernet devices Proposals for extending the protocol High-performance solutions require hardware support From the hardware design community Unexplored area Misconceptions about real-time systems High-throughput timing correctness 3

4 State of the Art on E More than 20 years of research From the real-time community Timing models and analysis of COTS Ethernet devices Proposals for extending the protocol High-performance solutions require hardware support From the hardware design community Unexplored area Misconceptions about real-time systems High-throughput timing correctness 4

5 State of the Art on E More than 20 years of research From the real-time community Timing models and analysis of COTS Ethernet devices Proposals for extending the protocol High-performance solutions require hardware support From the hardware design community Unexplored area Misconceptions about real-time systems High-throughput timing correctness 5

6 State of the Art on E Research on formal models and analysis Real-world solutions 6

7 THE ATACAMA FRAMEWORK 7

8 Atacama Framework Overview First integral open-source framework for Real-Time Ethernet Coordination model based on time-triggered communication Expressive programming language for dynamic TDMA scheduling Verification and analysis tools Hardware prototypes for implementation and experimental validation 8

9 System Overview Real-Time Domain 1 Standard Ethernet Real-Time Path Real-Time Domain 2 Real-time stations use a custom programmable NIC for TDMA Logical bus for real-time capable stations Segmentation of real-time domains within large networks Seamless integration with COTS components 9

10 Real-Time Stations Programmable custom module executes a predefined schedule Schedule specifies precise points in time to transmit and receive data Real-Time Task Data Buffers Optional Higher-Layers Time-triggered scheduler MAC Arbiter Ethernet MAC PHY Communication medium 10

11 Network Code Language Designed for dynamic TDMA arbitration Data flow Instructions Transmission: create, send Reception: receive Timing Control and Synchronization Instructions Timing: future, halt => wait Reference Broadcast Synchronization (RBS): sync Execution Flow Instructions branch 11

12 Communication Example Assume max. propagation delay of p time units Task 1 Task 2 Task 3 Task 4 Data Buffers A A B B Predefined Schedules L0:sync(master) wait (p, L1) L1: create (A) send() wait(4p, L2) L0: sync(slave) wait(p, L1) L1: receive(a) branch(l0) L0: sync(slave) wait(2p, L1) L1: create(b) send () branch(l0) L0: sync(slave) wait(3p, L2) L1: receive(b) branch(l0) B In Out MAC S A S B In Out A In Out B S A MAC MAC In Out MAC PHY PHY PHY PHY Slot A S B Sync Snd A Rcv A Snd B Rcv B Sync Snd A Rcv A Snd B Rcv B 0 p 2p 3p 4p 5p 6p 7p 8p 9p Round 12 t

13 Communication Example Assume max. propagation delay of p time units Task 1 Task 2 Task 3 Task 4 Data Buffers A A B B Predefined Schedules L0:sync(master) wait (p, L1) L1: create (A) send() wait(4p, L2) L0: sync(slave) wait(p, L1) L1: receive(a) branch(l0) L0: sync(slave) wait(2p, L1) L1: create(b) send () branch(l0) L0: sync(slave) wait(3p, L2) L1: receive(b) branch(l0) In Out In Out In Out In Out MAC MAC MAC MAC PHY PHY PHY PHY Slot B Sync Snd A Rcv A Snd B Rcv B Sync Snd A Rcv A Snd B Rcv B 0 p 2p 3p 4p 5p 6p 7p 8p 9p Round 13 t

14 End-to-End Latency L Tx Sw Rx Processing time in end stations (Tx and Rx) Processing time in the path (Sw and L) Number of switches Type of switches (operation mode, brands, models, etc.) Physical links Effects of best-effort traffic 14

15 Programmable NIC Full hardware implementation of Network Code instructions Fast and precise transactions between buffers and medium Real-Time Tasks Real-Time Task DATA-MEM CFG-MEM Data Buffers PROG Memory Space Status/Guards Scheduler Controller halt() receive() branch() create() MAC Arbiter Ethernet MAC PHY To communication medium Scheduler future() Ethernet Core frame Communication Medium Receive Buffer autoreceiver -Rx buffer Rx-MAC arbiter frame slave Sync() MAC mast Physical Transceiver (PHY) Send Buffer send() -Tx buffer Tx-MAC arbiter frame 15

16 Switching Path Dedicated bus-like path for time-triggered frames COTS Switching Fabric Real-Time Path -Rx -Rx Buffer Buffer Classif. MAC PHY -Tx -Tx Buffer Buffer Tx-arbit MAC PHY -Tx -Tx Buffer Buffer Tx-arbit MAC PHY 16

17 Transmission Arbiter Frame Format 8 B B 6 B 6 B 6B6B 2B 2B B 1500 B 4B 4 Pream Dest Src EthType Payload FCS Pream Dest Src EthType Payload FC COTS Switching Fabric Real-Time Path input B1 input R1 R2 -Rx -Rx Buffer Buffer Classif. MAC PHY -Rx -Tx Buffer Buffer Tx-arbit MAC PHY Port 1 Port 2 -Tx -Tx Buffer Buffer Tx-arbit MAC PHY MAC out R1 interrupted B1 B1 R2 t No processing overhead Fixed latency for real-time frames Propagation of best-effort frames during idle slots 17

18 Cyclic Discovery and Integration Sync Switches detect connected real-time devices during sync slot Only real-time capable devices connect to the logical bus during data exchange 18

19 EXPERIMENTAL RESULTS 19

20 Latency and Robustness Reference stream Rx Reference stream Tx 10 to 90 Mbps Interfering traffic approx 960 Mbps 20

21 Latency and Robustness Reference stream Rx Reference stream Tx 10 to 90 Mbps Interfering traffic approx 960 Mbps Starts dropping frames Cut-through Latency [µs] frames frames (est. error << 1%) Total Traffic [Mbps] 21

22 Buffer-Less Video Streaming Demonstration at 22

23 CLOSING REMARKS 23

24 Summary Integral open-source platform for E Accurate, validated and verifiable latency models Support for multi-hop topologies and segmentation Cost-effective approach Support of specialized hardware components is key It works! 24

25 Future Work and Open Challenges (Disclaimer: academic solutions) Usability: given Ethernet solutions are complicated (unnecessary strive for optimization? Simple RR vs scheduled traffic?) Need to know the topology Need to derive a schedule Compositionality: concurrent streams on large deployments New infrastructure vs infrastructure reuse Multi-objective traffic Latency guarantee Throughput guarantee Dependability (=> spatial and temporal redundancy; checkpointing) Mixed-criticality traffic Multi-mode Specification mistake tolerances Tool-chain support 25

26 The Atacama Desert Thanks!

27 Future Work and Open Challenges (Disclaimer: this applies to mostly academic solutions) Usability: given Ethernet solutions are complicated (unnecessary strive for optimization? Simple RR vs scheduled traffic?) Need to know the topology Need to derive a schedule Compositionality: concurrent streams on large deployments ( New infrastructure vs infrastructure reuse Open-source release aims to promote a collaborative model Encourage developers to explore, deploy, and test Accelerate solutions for real-world applications Technical improvements Support redundant paths for real-time stations Security mechanisms to prevent interference from babbling idiots and faulty stations Feel free to propose and try yourself Tool-chain for model-driven design of the schedules 27

28 More Information Contact information Formal definitions and related documentation Request for tools and source codes Application demos

29 Segmentation (Filter/Gateways) filter Real-Time Domain 1 Real-Time Domain 2 Two port device blocks or converts real-time frames Isolated exchange of real-time messages within domains 29

30 Segmentation (Time-to-Live) Real-Time Domain 1 TTL = 2 TTL = 2 Real-Time Domain 2 Each switch decrements the TTL by 1 Switches only forward frames with TTL > 0 30

31 Modeled v/s Observed Latency Worst-Case Model EEL = Ns vl Measurements with three cascaded switches Vl [words] Model [µs] Observed [µs] Error % % % % 31

32 If You Remember Only One Thing High-throughput timing correctness Fast is good, but not enough Design based on timed behavior => Open challenges

33 Existing Commercial Solutions Ethercat Tailored for industrial control applications Limited to ring topology Limited integration of best-effort traffic Limited to 100 Mbps Time-Triggered Ethernet (TTE) Integrates standard traffic Complex configuration Expensive devices Ethernet AVB Latency bounds based on pessimistic worst-case scenarios

34 Rare Worst Case Applications must handle the worst case, even though it rarely occurs Provide means for efficient resource usage without compromising the worst case

35 Evolution into Tree Schedules Two nodes (N1, N2), two temporal replica (N1*, N2*) Standard TDMA: N 1 N 1* N 2 N 2* t Tree Schedule: N 1* N 1 N 2 N 2* Guard checkn 1 Guard checkn 2 t N 2* Tree Schedule: (defragment) N 1 N 1* N 2 N 2 t

36 Cyclic Topology Discovery Sync frame Ack frame master

37 Cyclic Topology Discovery

38 Reference Broadcast Synchronization One master station sends a synchronization message Usually used to signal the start of communication rounds All other stations execute their corresponding actions for the round after receiving a synchronization message Schedules must consider the processing and propagation latency of sync messages Effective and low cost alternative to distributed clocks E.g. IEEE 1588 No use of timestamps

39 Distributed Real-Time Systems Real Time Station Multiple stations collaborate to perform one application Requires predictable communication delays Specific buses for specific applications 39

40 Automotive Networks Today CAN 1 Mpbs Comfort, Mech. Backup FlexRay 10 Mpbs Drive-by-wire MOST 150 Mpbs Infotainment 40

41 Trends in Automotive Networks CAN 1 Mpbs Comfort, Mech. Backup Enhanced Driving Assistance FlexRay 10 Mpbs Drive-by-wire MOST 150 Mpbs Infotainment 41

42 Ethernet in Real-Time Real Time Station Real-Time Domain 1 Real-Time Domain 2 42

43 Ethernet in Real-Time Real Time Station Best Effort Station Switch Real-Time Domain 1 Real-Time Domain 2 General purpose network High throughput 1Gbps Ethernet 100x (+) faster than current real-time buses By definition, Ethernet does not provide timing guarantees 43

44 Software Implementation Successful transmissions versus throughput Output queue must not overflow. Messages must be received. Slot structure must not be violated. Interrupts preempt instructions, cause jitter, cause delays. Send() instruction: 372 ns (39.47%), 733 ns (99%), ns (99.999%) LinuxPro + direct link (max. 11MB/s) 10% latency increase due to computation time 2.96MB/s with a 99% guarantee of a successful transmission 2.65MB/s with % guarantee 44

45 Programmable NIC Deterministic execution time for all instructions Tight assignment of slots for processing tasks create send branch receive 70 NC clk cycles d [32-bit words] L 45

46 Conclusions First open-source hardware accelerated platform for E Cost-effective solution Accurate, validated, and verifiable latency models Aims to promote a collaborative model Expects to accelerate development of solutions for real-world applications High-throughput timing correctness 46

47 Real-Time Stations Programmable custom module executes predefined schedules Schedules specify precise points in time to transmit and receive data Application Layer Non-Real-time Tasks Optional Higher Layers (TCP/UDP/IP) Real-time Tasks Scheduler NC Data Buffers Communication Layer Queues Queues MAC Arbiter MAC Physical Layer PHY Communication Medium 47

48 Latency in Switched Ethernet P1 in M1 Rx buffer Rx buffer Rx buffer Tx buffer P2 P3 in in M2 M3 L2 L2 MAC PHY Port 1 MAC MAC MAC PHY PHY PHY Port 2 Port 3 Port 4 P4 out Tx buffer M2 M1 M? M? M3 t Stations transmit data of any size at any time Switches are designed to maximize throughput Out of order delivery High jitter Latency is a probabilistic parameter 48

49 Latency in Switched Ethernet P1 in M1 Rx buffer Rx buffer Rx buffer Tx buffer P2 P3 in in M2 M3 L2 L2 MAC PHY Port 1 MAC MAC MAC PHY PHY PHY Port 2 Port 3 Port 4 P4 out Tx buffer M2 M1 M? M? M3 t Stations transmit data of any size at any time Switches are designed to maximize throughput Out of order delivery High jitter Latency is a probabilistic parameter 49

50 Real-Time Ethernet Real Time Station Best Effort Station Switch Guaranteed latency between real-time stations Best-effort communication must not affect real-time traffic Maximize compatibility with COTS components 50

51 Summary of the Enhanced Switch Custom extensions create logical real-time channels Seamless integration with COTS switch architectures Reduced bandwidth for best-effort traffic Configuration free Transparent for best-effort traffic Cost-effective solution

52 Network Code Language Designed for dynamic TDMA arbitration Data flow Instructions Transmission: create, send Reception: receive Timing Control and Synchronization Instructions Timing: future, halt => wait Reference Broadcast Synchronization (RBS): sync Execution Flow Instructions branch 52

53 End-to-End Latency L ASIP Tx Sw Tx ASIP Rx EEL = ASIP Tx + Ns*SW + (Ns+1)L + ASIP Rx Task Based on NetFPGA prototypes Task ASIP Tx create send Sw = 1184 [ns] ASIP Rx receive 480 [ns] Out FIFO MAC PHY MAC PHY Path MAC PHY *vl [ns] Out FIFO MAC PHY L = 384 [ns] 53

54 Device Utilization Based on Virtex2 chip (NetFPGA) Device Flip-Flops LUTs Block RAMs NC-ASIP (full) 4.3% 7.8% 7.8% NC-ASIP (no-tx) 3.4% 6.4% 7.3% NC-ASIP (no-rx) 2.9% 6% 6% Reference COTS switch 25% 40.5% 48.3% Custom Switch 26% 40.7% 59.3% 54