Theory, Concepts and Applications

Transcription

1 Theory, Concepts and Applications ETR 2015 Rennes August, the 27 th Jean-Baptiste Chaudron Copyright TTTech Computertechnik AG. All rights reserved. Page 1

2 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 2

3 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 3

4 Introduction Real-Time Computer System A real-time computer system is a computer system in which the correctness of the system behavior depends not only on the logical results of the computations, but also on the physical time, when these results are produced [Kop97]. The point in time when a certain action must be finished is called deadline. Soft deadlines: If the result has utility after the deadline. Hard deadlines: Missing a deadline can result in a catastrophic event. Computer systems classification Guaranteed Timeliness RT systems Best Effort no timing guarantees no RT systems Copyright TTTech Computertechnik AG. All rights reserved. Page 4

5 Introduction Distributed Real-Time System Reasons for distribution: Scalability single computer systems have limited computing resources Complexity handling through smaller simpler intelligent units Safe wiring from single computer to different sensors/actuators Fault-tolerance avoid single point of failure Control loops: Periodic operation Sensor communicate calculate - actuator Low end-to-end communication latency enables implementation of tighter control Real-time communication sensor actuator node1 node2 real-time bus Copyright TTTech Computertechnik AG. All rights reserved. Page 5

6 Introduction Latency vs. Deadline Relevant input/measurement occurs at Node A min Latency of system response jitter max Deadline for system response Node A processes input Result is communicated with node B Node B acts upon result from A Flow of time Copyright TTTech Computertechnik AG. All rights reserved. Page 6

7 Introduction End to End Latency (1) The time interval between the initiation of transmission from the host computer to other host computer at the receiver depend on many factors: Communication protocol, Media access control (MAC) Transmission speed, cable lengths Network load Node Host computer Communication Controller Node Host computer Communication Controller Time Copyright TTTech Computertechnik AG. All rights reserved. Page 7

8 Introduction End to End Latency (2) Communication can be delayed by: Concurrency of transmissions and the media access strategy e.g., CSMA CSMA access free channel one message in transit (or pending, but with higher priority than own message) two messages in transit / pending three messages in transit t min t msg t msg t msg t msg time Communication can also be delayed by: PAR No error Error handling strategy, e.g. PAR (Positive Acknowledge or Retransmit) Retransmit once Retransmit twice Bus access delays due to EMI (External Memory Interface) - wait for bus idle t min t msg t msg t msg t msg Retransmit three times time Copyright TTTech Computertechnik AG. All rights reserved. Page 8

9 Introduction Peak Load Handling Challenge (1) Peak load handling Peak load situation: all nodes on the shared bus require communication services at the same time, send maximum amount/length of data, highest priority messages Problem: find out in which scenario this happens, and what the actual load and the worst-case message delays are at this time In event driven systems this can be very complex Even more complicated if faults that lead to retransmission of messages must be accounted for Experiments or approximate scheduling can only offer probabilities : Ex: latency for message X less than 500 µs in 99,96%, but no guaranteed worst case latency Copyright TTTech Computertechnik AG. All rights reserved. Page 9

10 Introduction Peak Load Handling Challenge (2) Thrashing: Abruptly decreasing throughput that occurs with an increase of the system load. Cause of trashing: Retry mechanism in PAR protocols (error handling and time-outs) Combined with the waits from the CSMA access throughput 100 % thrashing point Ideal system Real system requested load Copyright TTTech Computertechnik AG. All rights reserved. Page 10

11 Introduction Deterministic Networks Features of deterministic networks: Known (maximum) end-to-end latency Bounded and small jitter Message ordering guarantee Error detection Masquerade protection Copyright TTTech Computertechnik AG. All rights reserved. Page 11

12 Introduction TT-System vs. ET-System Transportation - example Cars and taxis are event-triggered: they go whenever they are needed Trains are time-triggered: they go according to a fixed schedule Advantage of the event-triggered approach: very flexible Advantage of the time-triggered approach: very predictable When would you prefer a time-triggered solution? Copyright TTTech Computertechnik AG. All rights reserved. Page 12

13 Introduction Why Clock Synchronization? (1) In RT systems all layers of functionality in the system must meet the quality of service requirements defined by the application: the application layer must operate timely and predictably, reading the sensors in time, computing correct values, updating actuators reliably etc. the communication layer must meet the specified functionality of transmitting information between the nodes in the system, and must also do this predictably and timely Timely operation: Coordination of the computer nodes in the time domain Clock synchronization Copyright TTTech Computertechnik AG. All rights reserved. Page 13

14 Introduction Why Clock Synchronization? (2) Local clocks, a counter triggered by an oscillator. Oscillators have nominal rate (10 Mhz), and a certain drift rate. Standard drift rates of oscillators in the market: 10-3 s/s to 10-5 s/s Oscillators with small drift rate ~ 10-6 s/s expensive What does 10-3 s/s drift rate mean? 1 microsecond deviation every 1 millisecond, 1 second deviation in 1000 seconds, equals 10 min. deviation per week. Oscillator drift rate can be affected by other factors like: temperature, humidity, Clock synchronization: keeps clocks of distributed computers close to each other Copyright TTTech Computertechnik AG. All rights reserved. Page 14

15 Introduction Global Notion of Time 1. GLOBAL notion of time, built on top of local time Local clocks - free running Local view of global time 2. Activities triggered on basis of global time Copyright TTTech Computertechnik AG. All rights reserved. Page 15

16 Introduction Precision Interval (1) The precision, or precision interval (denoted ), is the upper bound between the slowest and the fastest non-faulty clock in the system. A fast clock and a slow clock will never differ by more than one precision interval. Clock 1 10:45 11:00 11:15 Clock 2 10:45 11:00 11:15 Clock 3 10:45 11:00 11:15 Copyright TTTech Computertechnik AG. All rights reserved. Page 16

17 precision interval Introduction Precision Interval (2) The precision interval in a distributed system depends on the hardware properties of each clock (clock drifts, e.g. 100 ppm) the resynchronization interval (e.g. 5 ms) the resynchronization method used (how efficient does it work) smaller precision interval smaller timeouts more efficient system drift offset resynchronization interval Average clock Copyright TTTech Computertechnik AG. All rights reserved. Page 17

18 Introduction Whole System Synchronization (1) Yes, 15:00 Two different approaches... centralized vs. distributed I want to join, what s the time? It s 15:00! 15:00 14:59 OK, 15:00. I see, 15:00. 15:00 Copyright TTTech Computertechnik AG. All rights reserved. Page 18

19 Introduction Whole System Synchronization (2) Synchronization to external time reference is possible all nodes can apply a bounded correction term to slightly speed up or slow down the local clock the precision window will never be left time will never go backwards this mechanism can be used to broadcast a correction value relative to some external time reference (e.g. GPS time) application of this term to the local node is performed by the host CPU 15:00 15:00 End System with GPS receiver 15:00 Copyright TTTech Computertechnik AG. All rights reserved. Page 19

20 Introduction Time Triggered System Summary (1) Any Time-Triggered System must have two key properties: 1 a global notion of time in case of a distributed system: a GLOBAL notion of time, available to each node in the system 2 a global schedule (when to do what) in case of a distributed system: a GLOBAL schedule or CONSISTENT parts of a GLOBAL schedule available to each node in the system Copyright TTTech Computertechnik AG. All rights reserved. Page 20

21 Introduction Time Triggered System Summary (2) All protocol operations are initiated at a priori known points in time ( action time ), transmission and reception is thus performed during known slots. There is no external (application) control over the protocol progression. Generation of a global time base and common knowledge about the action times reside inside the protocol controllers and cannot be modified by the application CPU. Node 1 send receive receive t 1 t 2 t 3 Node 2 receive send receive t 1 t 2 t 3 Node 3 receive receive send t 1 Slot t 2 t 3 Copyright TTTech Computertechnik AG. All rights reserved. Page 21

22 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 22

23 TTEthernet Basics Ethernet History Ethernet is most popular LAN technology in the world Technology is a well-established open-world standard and very scalable Early version was bus-based and 10 Mbit/s, today 100 Mbit/s and 1 Gbit/s (Gigabit Ethernet) are common Ethernet is specified in OSI layer one and two Found by Xerox Palo Alto Research Center (PARC) in 1975 Original designed as a 2.94 Mbps system to connect 100 computers on a 1 km cable Later, Xerox, Intel and DEC drew up a standard support 10 Mbps Basis for the IEEE s specification Ethernet uses the CSMA/CD media access control Copyright TTTech Computertechnik AG. All rights reserved. Page 23

24 TTEthernet Basics Ethernet - CSMA/CD (1) CSMA/CD (carrier sense multiple access with collision detection) Data is transmitted in the form of packets. Listen/Sense channel before packet transmission. If channel is sensed idle packet is transmitted Else defer the transmission until channel becomes idle Channel is idle, I will start transmission I want to send, but there is an ongoing tranmisison in the bus, I will wait! Copyright TTTech Computertechnik AG. All rights reserved. Page 24

25 TTEthernet Basics Ethernet - CSMA/CD (2) If collision is detected: stop sending frame data send a 32-bit "jam sequence wait for a random time interval start retransmission I will wait for 12 miliseconds before listening to the channel I will wait for 43 miliseconds before listening to the channel Copyright TTTech Computertechnik AG. All rights reserved. Page 25

26 TTEthernet Basics Possible Ethernet Topologies Bus Star / Switched Ring Copyright TTTech Computertechnik AG. All rights reserved. Page 26

27 TTEthernet Basics Switched Star Topology It s modular Independent wires for each end node Independent traffic in each wire A second layer of switches can be added to build a hierarchical network that extends the same two benefits above Copyright TTTech Computertechnik AG. All rights reserved. Page 27

28 TTEthernet Basics Ethernet Half Duplex vs. Full Duplex Half duplex Full duplex Uni-directional Bus based Ethernet 10 Mbit/s Bi-directional Switched Ethernet 100 Mbit/s and above Copyright TTTech Computertechnik AG. All rights reserved. Page 28

29 TTEthernet Basics RT Limitations of Ethernet Bus based Ethernet No timing guarantees Latency not bounded latency because of the CSMA/CD Traffic bursts cause congestion, and packet delays Switched Ethernet Point-to-Point transmission between an switch and an end-system Separate conductors for transmission and reception paths no message collisions Traffic bursts cause congestion in switch and packet delays, buffer overflows and packet loss Message ordering is not maintained Fault handling protocols, like PAR in TCP introduce additional end-to-end communication latency between sender and receiver Additional traffic for ACK, increase network congestion Copyright TTTech Computertechnik AG. All rights reserved. Page 29

30 TTEthernet Basics Why TTEthernet? (1) Ethernet hardware is low cost. Ethernet is a well-established open-world standard and very scalable. The OSI reference model gives a well-structured classification of concepts that can be built on top of Ethernet. Existing tools can be leveraged as cost-efficient diagnosis tools. As all messages in TTEthernet are standard Ethernet compliant, existing tools can be leveraged for time-triggered messages as well. Standard web servers can be leveraged for maintenance and configuration. Engineers learn about Ethernet at school. Ethernet compatibility enables the usage of technology that is already established, tested, and verified. Copyright TTTech Computertechnik AG. All rights reserved. Page 30

31 TTEthernet Basics Why TTEthernet? (2) IEEE addresses the lowest layers of the OSI reference model, some higher layers are represented by other IEEE 802 parts. TTEthernet performs services transparently within the Data Link layer, using all IEEE services without modification and not modifying IEEE services Application Presentation Session Transport Network Data Link Physical OSI layer model architecture, NM, layers above (TCP,UDP,IP) Logical Link Control (IEEE LLC) Media Access Control (IEEE MAC) Physical Layer (IEEE PHY) 10BaseT 100BaseTx 1000BaseCX Copyright TTTech Computertechnik AG. All rights reserved. Page 31

32 TTEthernet Basics TTEthernet Properties Short communication loop times (<100 µs) high-speed controls Minimized loop jitter (<1 µs) deterministic control functions Integration of legacy (non-real-time) traffic in the RT network Utilize the full potential of switched Ethernet high bandwidth, dedicated links with full duplex communication Fault tolerant communication system possible High-performance variant offering full link speed per device Low-cost variant using regular Ethernet components Interoperability between high-performance and low-cost components Receive-compatible with standard Ethernet components (e.g. for traffic monitoring, gateways, diagnosis) Transmit-compatible with standard Ethernet components (e.g. for simulation, low-cost non-rt applications) Copyright TTTech Computertechnik AG. All rights reserved. Page 32

33 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 33

34 Critical Traffic over TTEthernet RT Communications Overview (1) Real-Time Communication Wired Wireless Unidirectional e.g. ARINC429 Single-Master e.g. LIN Multi-Master Switched asynchronous synchronous switched Ethernet CAN (ISO11898) unswitched Ethernet TTP ARINC659 FlexRay ARINC 664 (AFDX ) Copyright TTTech Computertechnik AG. All rights reserved. Page 34

35 Critical Traffic over TTEthernet RT Communications Overview (2) Real-Time Communication Wired Wireless Unidirectional e.g. ARINC429 Single-Master e.g. LIN Multi-Master Switched asynchronous synchronous switched Ethernet CAN (ISO11898) unswitched Ethernet TTP ARINC659 FlexRay ARINC 664 (AFDX ) Copyright TTTech Computertechnik AG. All rights reserved. Page 35

36 Critical Traffic over TTEthernet ARINC 664 traffic ARINC 664 is a standard for an aircraft data network based on Ethernet (IEEE 802.3). AFDX (Avionics Full-Duplex switched Ethernet) is a deterministic data network for safety critical applications based on ARINC 664 and defined in ARINC 664 part 7. Note: AFDX is a registered trademark of Airbus TTEthernet developments are influenced by ARINC 664 p7 because one key use of TTEthernet is to extend such networks with time-triggered communication services. In TTEthernet network, three classes of traffic can co-exist and A664 traffic is named rate-constrained traffic. time-triggered traffic rate-constrained traffic regular ( legacy ) traffic TT RC BE Copyright TTTech Computertechnik AG. All rights reserved. Page 36

37 Critical Traffic over TTEthernet AFDX Planes & Helicopters Airbus A380 Boeing 787 Airbus A400M Airbus A350 Sukhoi Superjet 100 AgustaWestland AW101 Irkut MS-21 Bombardier CSeries Comac ARJ21 AgustaWestland AW149 Copyright TTTech Computertechnik AG. All rights reserved. Page 37

38 Critical Traffic over TTEthernet Virtual Link Concept (1) ARINC 664 p7 traffic is based on Virtual Links (VLs). A Virtual Link is a path between sender and receiver(s) (1:n relation) with a unique VL identifier (VL ID). Switches are configured to know about the VL definitions (up to 4096 VL IDs per switch). End-Systems exchange frames through Virtual Links (VLs). A Virtual Link defines a unidirectional path from one End-System to one or more destination End-Systems. VL ID Sender Receive ES 1 Network VL 1 VL 2 ES 2 ES 3 ES 4 Copyright TTTech Computertechnik AG. All rights reserved. Page 38 r(s) 1 a b,d 2 a c 3 b c,e,f 4 c a,b,d 5 c a,f example Virtual Link definitions

39 Critical Traffic over TTEthernet Virtual Link Concept (2) Switches are configured to know about the VL definitions VL ID Sender Receiver(s) 1 a b,d 2 a c 3 b c,e,f 4 c a,b,d 5 c a,f a g b 3 c d e f Copyright TTTech Computertechnik AG. All rights reserved. Page 39

40 Critical Traffic over TTEthernet Virtual Link Concept (3)...and route frames according to the VL definition. VL ID Sender Receiver(s) 1 a b,d 2 a c 3 b c,e,f 4 c a,b,d 5 c a,f a g b c 3 3 d e 3 f Copyright TTTech Computertechnik AG. All rights reserved. Page 40

41 Critical Traffic over TTEthernet Virtual Link Concept (4) HOST ES Error Propagation Boundaries Static configuration of VLs per port Restricted access for configured VL Traffic filtering policing At switch and ES VL not configured on SW-port Switch VL not configured on ES-port ES HOST ES HOST Copyright TTTech Computertechnik AG. All rights reserved. Page 41

42 Critical Traffic over TTEthernet Virtual Link Address Scheme Virtual Link are encoded in the destination MAC address. Differentiate a critical traffic from a standard Ethernet traffic All VLID have the same CT marker VL ID is encoded in the lower two bytes of the destination MAC address. CT marker VL ID 89 1D Copyright TTTech Computertechnik AG. All rights reserved. Page 42

43 Critical Traffic over TTEthernet Rate Constrained Traffic Definition VL have reserved bandwidth, expressed in terms of: Maximum Frame Size (MFS) Minimum time between two frames - Bandwidth Allocation Gap (BAG) For each VL a bandwidth limit per time is defined. This limit will be enforced by BAG policing in the switch(es): VL traffic exceeding the limit will be silently suppressed RC by itself does not guarantee timing or jitter, or preserve order of transmissions. It only ensures bandwidth limitation. VL ID Sender Receiver(s) Limit/BAG 1 a b,d 25 Kb/20 ms 2 a c 12 Kb/50 ms 3 b c,e,f 17 Kb/10 ms 4 c a,b,d 20 Kb/80 ms example Virtual Link definitions with bandwidth limits RC Copyright TTTech Computertechnik AG. All rights reserved. Page 43

44 Critical Traffic over TTEthernet Traffic Shaping in the End System Traffic shaping to ensure the limitation of bandwidth assigned to a VL Host sends instances of the VL arbitrary ES perform traffic shaping Switches enforce traffic shaping HOST Host sends to End-System ES BAG BAG BAG Traffic on the Network Copyright TTTech Computertechnik AG. All rights reserved. Page 44

45 Critical Traffic over TTEthernet Traffic Policing in the Switch Traffic policing to ensure the limitation of bandwidth for a VL Protection of the network from ES faults (babbling idiot) Switch Filtering & Routing Incoming Port Outgoing Port drop drop BAG BAG Copyright TTTech Computertechnik AG. All rights reserved. Page 45

46 Critical Traffic over TTEthernet Virtual Link Scheduling (1) A664 standard guarantees the maximum allowed jitter At the output of an ES Lmax is the maximum frame length of a VL max_ max_ jitter jitter 40µs 500 µs j { set of VLs} ((20 L max net _ bandwidth j ) 8) Copyright TTTech Computertechnik AG. All rights reserved. Page 46

47 Critical Traffic over TTEthernet Virtual Link Scheduling (2) Virtual Links are defined in terms of: Bandwidth Allocation Gap (BAG) Maximum allowed frame size Maximum jitter Jitter = 0 Jitter < Max Jitter < Max Jitter = Max BAG BAG BAG BAG Copyright TTTech Computertechnik AG. All rights reserved. Page 47

48 Critical Traffic over TTEthernet Virtual Link Scheduling (3) Rate constrained traffic is event-triggered traffic. This means that the assigned bandwith of the VL-IDs will not be constant at any point in time. So what happened in a so called peak load scenario? Different network paths will send to a single link (port of a switch). The switch has to store the messages sent to this link and will work through it according to the priorities of the message. Copyright TTTech Computertechnik AG. All rights reserved. Page 48

49 Critical Traffic over TTEthernet Rate Constrained Traffic (Summary) What is Rate-Constrained traffic? Predefined maximum amount of bytes (or frequency of frames) per VL VL traffic exceeding the limit will be silently dropped (by switch) Overall network can be analyzed for jitter: How much will the messages of a VL be delayed in typical case and in worst case? What happens if all of these VLs generate their traffic at about the same time? Will that overflow the switch buffers? Copyright TTTech Computertechnik AG. All rights reserved. Page 49

50 Critical Traffic over TTEthernet Time Triggered Ethernet How does TTEthernet build upon the concept of VLs? Each TT frame is transmitted by the end system at a certain time the end system can also transmit non-tt frames; they have lower priority than the TT frames The switch expects the frame from the transmitter within a certain time interval (window) this provides an implicit bus guardian functionality: TTE traffic received outside of the expected time interval is discarded, just as rate constrained traffic would be if it exceeds the BAG The switch forwards the frame to the receivers (end systems or other switches) at certain times these times can be different for each port! The receivers receive the frame with well-defined latency and minimal jitter even if they are not TTE nodes TT Copyright TTTech Computertechnik AG. All rights reserved. Page 50

51 Critical Traffic over TTEthernet TTEthernet Scheduling Principle (1) Senders have a defined transmit schedule Switches have an acceptance schedule for incoming data VL ID Sender Receiver(s) 1 [07:25-07:35] 8:20 2 [08:55-09:05] 10:30 3 [09:55-10:05] 10:30 4 c a,b,d 5 c a,f a g VL ID Time 10:00 11:15 b 3 c d e f Copyright TTTech Computertechnik AG. All rights reserved. Page 51

52 Critical Traffic over TTEthernet TTEthernet Scheduling Principle (2) Switches have a forwarding schedule per port Receivers receive the frame with defined latency and minimal jitter VL ID Sender Receiver(s) 1 [07:25-07:35] 8:20 2 [08:55-09:05] 10:30 3 [09:55-10:05] 10:30 4 c a,b,d 5 c a,f a g b 3 c d 3 3 e f Copyright TTTech Computertechnik AG. All rights reserved. Page 52

53 Critical Traffic over TTEthernet TTEthernet Traffic (1) Within a cluster, TTEthernet provides configurable synchronization services to ensure that all time-triggered communication is transmitted according to schedule, with known end-to-end latency and minimal jitter, regardless of other traffic going through the same switches and links. Each link in each direction has a schedule of its own. But all schedules are synchronized to one global clock per sync domain. Copyright TTTech Computertechnik AG. All rights reserved. Page 53

54 Critical Traffic over TTEthernet TTEthernet Traffic (2) Time-Triggered (TT) - scheduled and configured sending instant is triggered by the time constant transmission delay and small and bounded jitter the switch expects the frame from the transmitter within a certain time interval (window) this provides an implicit bus guardian functionality: TTE traffic received outside of the expected time interval is discarded switch forwards the frame to the receivers (end systems or other switches) at certain times - these times can be different for each port! receivers receive the frame with well-defined latency and minimal jitter. Off line scheduling by means of SW-tools (TTE-Tools) Copyright TTTech Computertechnik AG. All rights reserved. Page 54

55 Critical Traffic over TTEthernet Message Conflict (1) No message collision can occur, because of point-to-point connections and the full duplex communication. Conflict resolution among Ethernet Messages is resolved by the switch. Two types of conflicts exists: Simultaneous messages: messages are competing for the same resource at the same point in time. E.g., two messages (m1, m2) received in Switch port 1 and port 2 need to be routed through Switch port 3 at the same point in time Port/Link is occupied: message transmission is on-going and another message need to be transmitted through that port/link. E.g., message m1 received in port 1 is being routed through port 3. Message m2 received in port 2 need to be routed through port 3, m1 is in transmission In 100 Mbit/s Ethernet, a transmission of a frame with maximum length takes 123 µs, meaning that for 123 µs the port and the link are occupied during transmission Copyright TTTech Computertechnik AG. All rights reserved. Page 55

56 Critical Traffic over TTEthernet Message Conflict (2) Mechanisms for conflict resolution: Priorities configurable TT traffic has 1 priority, RC traffic has 4 priority levels TT over all RC priorities, TT over specific RC priority, e.g., TT has the priority of RC with priority 6. Shuffling mechanism configurable increase the receive window at the receiver side for one message transmission delay in one link (12,3 µs in 1 Gbit/s). Resource media reservation configurable Prior to the transmission of TT frames, no transmission is allowed. No transmissions of rate-constraint and best effort Ethernet traffic are initiated that last across the (scheduled) start of transmission point in time of the time triggered message. Copyright TTTech Computertechnik AG. All rights reserved. Page 56

57 Critical Traffic over TTEthernet Message Conflict (3) Currently routed frames will be finished, the following frames will be handled according to there priority. Shuffling Copyright TTTech Computertechnik AG. All rights reserved. Page 57

58 Critical Traffic over TTEthernet Message Conflict (4) According to the schedule of the time-triggered messages a window in front of the tt-frame will be reserved so that no other lower priority frame is able to delay the tt-frame. Media Reservation Copyright TTTech Computertechnik AG. All rights reserved. Page 58

59 Critical Traffic over TTEthernet Message Conflict (5) Simultaneous TT-TT message conflict Correct configuration schedules ensure that such message conflicts do not exist. Simultaneous RC-RC message conflict Priority mechanisms resolves the conflict. Among the same priority: internal priority level based on source port number. Simultaneous TT-RC message conflict Priority mechanisms resolves the conflict. Simultaneous TT-BE or RC-BE message conflict TT or RC over best effort Copyright TTTech Computertechnik AG. All rights reserved. Page 59

60 Critical Traffic over TTEthernet Message Conflict (6) Port/Link is occupied TT-TT Correct configuration schedules ensure that such message conflicts do not exist. Use shuffling mechanism - in case of complex network topologies Port/Link is occupied BE-TT, RC-TT Shuffling mechanism Media reservation Port/Link is occupied BE-RC RC message is delayed for the transmission length of a BE frame. Live with that, jitter is introduced to the RC frame Jitter is part of RC configuration. Copyright TTTech Computertechnik AG. All rights reserved. Page 60

61 AFDX Critical Traffic over TTEthernet Mixed Network Illustration (1) 1. Starting point: Pure AFDX network AFDX AFDX AFDX AFDX AFDX AFDX AFDX AFDX Note: AFDX is a registered trademark of Airbus Copyright TTTech Computertechnik AG. All rights reserved. Page 61

62 Critical Traffic over TTEthernet Mixed Network Illustration (2) 1. Starting point: Pure AFDX network AFDX 2. Change switches to TTEthernet configured as pure AFDX AFDX TTE TTE AFDX AFDX AFDX TTE TTE Note: AFDX is a registered trademark of Airbus Copyright TTTech Computertechnik AG. All rights reserved. Page 62

63 Critical Traffic over TTEthernet Mixed Network Illustration (3) 1. Starting point: Pure AFDX network AFDX 2. Change switches to TTEthernet configured as pure AFDX 3. Add function using time-triggered services (Sync, TT messages ) AFDX AFDX TTE TTE TTE TTE AFDX AFDX TTE TTE TTE The TTEthernet switch is seen as an AFDX switch by the AFDX network!!! No modification of previous AFDX architecture organization (Bags, VL ) as long as there is enough bandwidth Note: AFDX is a registered trademark of Airbus Copyright TTTech Computertechnik AG. All rights reserved. Page 63

64 Critical Traffic over TTEthernet Mixed Network Illustration (4) 1. Starting point: Pure AFDX network AFDX 2. Change switches to TTEthernet configured as pure AFDX AFDX TTE TTE AFDX AFDX 3. Add function using time-triggered services (Sync, TT messages ) 4. Do further changes (e.g., add other AFDX network, BE Ethernet E/S) AFDX ETH TTE TTE TTE AFDX TTE AFDX Ethernet ETH TTE ETH ETH ETH ETH ETH Note: AFDX is a registered trademark of Airbus Copyright TTTech Computertechnik AG. All rights reserved. Page 64

65 Critical Traffic over TTEthernet Summary TTEthernet support Time-Triggered and Rate-Constrained (A664) traffic class (as well as non-critical traffic) Both traffic classes (TT and RC) use the VL concept TTEthernet provides deterministic end-to-end timing (latency) and minimal jitter (jitter << latency) for TT traffic including multicast capability TTEthernet provides bandwidth guaranties for RC traffic including multicast capability Rate constrained and regular ( best-effort ) Ethernet traffic are possible and can utilize any remaining bandwidth without disturbing the TT traffic flows. Copyright TTTech Computertechnik AG. All rights reserved. Page 65

66 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 66

67 Clock Synchronization Principles TTEthernet Timing Time-triggered communication and timing checks in a network with forwarding: multiple schedules and delays apply I ll accept M only between 10:40 and 10:50 I ll accept M only between 10:55 and 11:05 I ll expect M between 11:05 and 11:15 M M M I ll transmit M at 10:45 I ll forward M at 11:00 a switch M I ll forward M at 11:10 Let s see if I can receive M Copyright TTTech Computertechnik AG. All rights reserved. Page 67

68 Clock Synchronization Principles Communication Constraints Each TTEthernet component has a local clock. The clock synchronization service periodically re-synchronizes these clocks so that time-triggered frame transmission is possible. Reception of time-triggered frames and regular Ethernet communication of these components do not require clock synchronization. Can send time-triggered frames Can receive time-triggered frames Can send/receive regular frames Clock Sync inactive no yes yes Clock Sync active yes yes yes Copyright TTTech Computertechnik AG. All rights reserved. Page 68

69 Clock Synchronization Principles Synchronization Services Clock Synchronization Service is executed during normal operation mode to keep the local clocks synchronized to each other. Clock Synchronization Service Startup/Restart Service is executed to reach an initial synchronization of the local clocks in the system. Integration/Reintegration Service is used for components to join an already synchronized system. Clique Detection Services are used to detect loss of synchronization and establishment of disjoint sets of synchronized components. Computer Time Fast Clock Perfect Clock Slow Clock Message Exchange Message Exchange Startup/Restart Service R.int R.int Real Time Copyright TTTech Computertechnik AG. All rights reserved. Page 69

70 Clock Synchronization Principles Periodicity by Cycles Time-triggered systems are always periodic. The periods for TTEthernet are: Integration Cycle and Communication Cycle. integration cycle integration cycle integration cycle integration cycle time communication cycle The Integration Cycle is the period for clock synchronization. Typical duration: 1 millisecond. One Communication Cycle is an integer multiple (often a power-of-two) of an Integration Cycle. Copyright TTTech Computertechnik AG. All rights reserved. Page 70

71 Clock Synchronization Principles Protocol Control Frames (1) TTEthernet communication on the PHY/MAC layer is indistinguishable from regular IEEE Ethernet traffic. All frames/transmissions are fully Ethernet compliant. But some of them are very short (64 byte) Ethernet frames with a special Ethertype field (value 0x891D). These are called Protocol Control Frames (PCFs) and are used by TTEthernet to perform the synchronization among the TTEthernet components. 89 1D Copyright TTTech Computertechnik AG. All rights reserved. Page 71

72 Clock Synchronization Principles Protocol Control Frames (2) The PCF payload contains several fields required for protocol operation. Explanations for each of them are given in the context of the related functionality. payload byte offset 0 4 Integration Cycle Membership Sync Priority - reserved - Sync Domain - reserved - Transparent Clock Type - reserved - Type is a 4 bit field Copyright TTTech Computertechnik AG. All rights reserved. Page 72

73 Clock Synchronization Principles Protocol Control Frames (3) TT TTEthernet Time-triggered Ethernet frames (TT). TT frames can be sent by TTEthernet components, and received by any Ethernet component. It is sent and routed at defined points in time, giving it highly deterministic timing properties and minimal jitter. The path and timing of this frame is determined by the schedules configured in the sender and switches along the channel. PCF TTEthernet Protocol Control Frames (PCF). PCF frames are special kind of TTEthernet frames exchanged between TTEthernet components in order to establish and maintain synchronization. They can be received by regular Ethernet components too, but are meaningless for them (except for diagnostic tools). Copyright TTTech Computertechnik AG. All rights reserved. Page 73

74 Clock Synchronization Principles Synchronization Domain The TTEthernet protocol builds up a common notion of time among the TTEthernet components (end systems and switches) in the cluster. The group of components with a common notion of time is called a Synchronization Domain. Regular Ethernet components cannot be part of a synchronization domain they can communicate with components in the synchronization domain, but cannot synchronize with them using TTEthernet synchronization. Copyright TTTech Computertechnik AG. All rights reserved. Page 74

75 Clock Synchronization Principles Two classes of Masters In each Integration Cycle, the TTEthernet components execute the TTEthernet synchronization protocol. This execution is triggered by two groups of components, called and Synchronization Masters Compression Masters Sync Sync Sync Comp Comp Comp Each group has to contain at least one member; for fault tolerant synchronization, multiple members are needed. Note: Typically, Synchronization Master function is located in end systems, and Compression Master function is located in switches. Copyright TTTech Computertechnik AG. All rights reserved. Page 75

76 Clock Synchronization Principles Synchronization Topology some TTE end systems are synchronization masters, others just synchronization clients some TTE switches are compression masters (and synchronization clients), others just synchronization clients Sync Comp Sync Note: this is an architectural choice, but current implementations currently require this setup. Switches can be configured as compression masters, but not as synchronization masters. Copyright TTTech Computertechnik AG. All rights reserved. Page 76

77 Clock Synchronization Principles Cluster Setup Example 1 Two synchronization masters, one compression master A simple configuration with little fault tolerance Sync Comp Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 77

78 Clock Synchronization Principles Cluster Setup Example 2 four Synchronization Masters, one Compression Masters Sync Sync Comp Sync Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 78

79 Clock Synchronization Principles Multiple Compression Masters If multiple compression masters exist, they will be configured with different synchronization priorities (sync_priority). All active Compression Masters will perform the startup sequence, and the one with the highest sync_priority will prevail. If it fails, then the next-lower active Compression Master will automatically follow. Copyright TTTech Computertechnik AG. All rights reserved. Page 79

80 Clock Synchronization Principles Two-Step Synchronization Protocol Control Frames called Integration Frames are used to perform all synchronization functions. They are transmitted accordingly: Sync Sync IN IN IN Comp Comp The Synchronization Masters send Integration Frames at the beginning of each Integration Cycle. The timing of these frames is used for the voting Sync Comp Sync Sync IN IN IN Comp Comp The Compression Masters send Integration Frames to everybody, timing them in a special way so that everybody can correct their clocks. Sync Comp Copyright TTTech Computertechnik AG. All rights reserved. Page 80

81 Clock Synchronization Principles Synchronization Step 1-A Synchronization masters transmit PCFs to the compression masters Sync Comp Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 81

82 Clock Synchronization Principles Synchronization Step 1-B Compression masters perform the compression function based on the received PCFs (only applied to PCFs) Function overview: 1. collect incoming PCFs, 2. compute an average value of their reception times, and 3. prepare to send out PCFs at a time depending on the result Comp Let s see I got two PCFs, both are valid, so I will take the average of their timing to determine when I transmit back. Copyright TTTech Computertechnik AG. All rights reserved. Page 82

83 Clock Synchronization Principles Synchronization Step 2-A Compression masters transmit PCFs to all synchronization masters and clients Note: compression masters also are synchronization clients Sync Comp Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 83

84 Clock Synchronization Principles Synchronization Step 2-B Synchronization masters and clients (i.e. everybody) collect PCFs from all compression masters and clients and perform the clock correction function clock correction function: compute average of PFCs received and adapt the local clock according to the result OK I got one PCF, so I simply use that frame s arrival time to determine how much my local clock needs to be adapted forward or backward. Copyright TTTech Computertechnik AG. All rights reserved. Page 84

85 Clock Synchronization Principles The Distributed Start-up Challenge To bring a set of nodes, ready for communication but not yet synchronized, to synchronous operation within a guaranteed time limit even in the presence of faulty nodes or links. TTEthernet defines a set of related state machines in the synchronization masters (set or subset of end systems) and compression masters (set or subset of switches) to perform this. We will now look at the principles of cold-start and initial synchronization. Mathematical modeling and model checking have been applied on these mechanisms to ensure their correctness even in complex and malicious fault scenarios (We will not look at these formal proofs in this training). Copyright TTTech Computertechnik AG. All rights reserved. Page 85

86 Clock Synchronization Principles Start-up Procedure 1-2 At least one Synchronization Master sends a Coldstart PCF to the Compression Masters Sync Sync CS Comp Comp Comp I want to get synchronized! Sync The Compressions Masters echo back the Coldstart PCF to all Synchronization Masters. Sync Sync CS Comp Comp Comp OK, I am available! Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 87

87 Clock Synchronization Principles Start-up Procedure 3-4 All Synchronization Masters send Coldstart-Acknowledge PCFs to all Compression Masters Then let s start all together! Sync Sync Sync CA CA CA Comp Comp Comp The Compressions Masters echo back the Coldstart-Acknowledge PCFs to the Synchronization Masters. Sync Sync Sync CA CA CA Comp Comp Comp OK, here is the GO signal! Copyright TTTech Computertechnik AG. All rights reserved. Page 88

88 Clock Synchronization Principles Synchronization Master (1) Integration at Synchronization Masters (End systems): I listen a while to receive a PCF. If I receive an integration PCF, the cluster is already synchronized and I use the PCF to join the cluster. If I receive a coldstart PCF, someone has just initiated the start-up (and this frame was echoed to me by the Compression Master). I will transmit a coldstart-acknowledge PCF, which will be echoed back to me. This allows me to integrate. If nobody is sending any PCFs, I will (after listening for a while) perform a coldstart: I will transmit a coldstart PCF, which will be echoed to everybody. Then everybody will transmit coldstart-acknowledge PCFs, which will be echoed back to everybody, which allows everybody to integrate. Synchronization Clients do not perform coldstart. They wait until they receive Integration PCFs. Copyright TTTech Computertechnik AG. All rights reserved. Page 89

89 Clock Synchronization Principles Synchronization Master (2) Integration at Synchronization Master (simplified): need to integrate Do I see an Integration PCF? n Do I see a Cold-Start PCF? n Send a Cold-Start PCF y y integrated Send a Cold-Start- Acknowledge PCF (an echo of my PCF will go to other nodes, and they will send a Cold-Start- Acknowledge) Receive back a Cold- Start-Acknowledge PCF Copyright TTTech Computertechnik AG. All rights reserved. Page 90

90 Clock Synchronization Principles Sync. Compression Master (1) Integration at Compression Master (TTE Switch): I listen to receive a PCF. If I receive an Integration PCF, the cluster is already synchronized and I use the PCF to join the cluster. If I receive a Coldstart or Coldstart-Acknowledge PCF, someone is just initiating the start-up. I will echo back this frame to all Synchronization Masters, letting them know I am here. Doing this will allow the Synchronization Masters to get into the INTEGRATED state, where they start regular integration cycles and transmit Integration PCFs. Being also a Synchronization Client, the Compression Master will then integrate on the Integration PCFs sent by the Synchronization Masters. Copyright TTTech Computertechnik AG. All rights reserved. Page 91

91 Clock Synchronization Principles Sync. Compression Master (2) Integration at Synchronization Compression Master (simplified): need to integrate Do I see an Integratio n PCF? y n Do I see a CS / CS ACK PCF? y n integrated Echo back the frame to all Sync. Masters Copyright TTTech Computertechnik AG. All rights reserved. Page 92

92 Clock Synchronization Principles Start-up Details (1) A transmits a coldstart PCF Echo back from Compression Master After the CSO, everyone (except A) transmits a coldstartacknowledge Echo back from Compression Master Cluster starts first integration round A Sync CS CS CSO CA CAO IN B Sync CS CA CA IN C Sync CS CA CA IN D Sync CS CA CA IN time CS ColdStart PCF CA Coldstart Acknowledge PCF IN Integration PCF CSO ColdStart Offset timeout CAO Coldstart Acknowledge Offset timeout Copyright TTTech Computertechnik AG. All rights reserved. Page 93

93 Clock Synchronization Principles Transmission Delay Correction And Total Order The previous discussions of fault-tolerant start-up and faulttolerant clock synchronization made some assumptions: no dynamic delays incurred by intermediate switches, network stacks, etc. constant and known transmission delays over all paths through the network consistent frame order seen at all receivers (total frame order) These assumptions are not valid a priori and need justification We justify them by introducing a delay correction mechanism one level below the start-up and clock synchronization services Copyright TTTech Computertechnik AG. All rights reserved. Page 94

94 Clock Synchronization Principles Transmission Delay Definition A transmission sent into a network will be delayed by various physical effects (e.g. propagation of electrons/photons in the wire, electric signal conversions in transceivers) and processing times (e.g. forwarding or queueing in a switch). The overall transmission delay is composed of static elements, i.e. (nearly) constant delays independent of the load in the network, such as wire propagation delays along the frame path or the processing time for a frame in a receiver. For these a (nearly) exact value can be defined. dynamic elements, typically incurred by the prioritization scheme in outgoing queues in switches along the frame path. For these, an upper bound can be derived. In total, an upper bound for the transmission delay of any frame in the whole network can be derived. It is called max_transmission_delay time of transmission time of reception (worst case) transmitter delay wire delay switch receive delay queue-out delay (dynamic) wire delay receiver delay example times: 5 µs 1 µs 2 µs 1-7 µs 1 µs 5 µs time Copyright TTTech Computertechnik AG. All rights reserved. Page 95

95 Clock Synchronization Principles Acceptance Windows (1) An Acceptance Window is a time interval of pre-defined duration around an expected receive point. In time-triggered systems, a schedule defines the receive points (i.e. the moments when transmissions are expected). Due to clock drifts and jitter, these moments are not exactly known and therefore require the definition of acceptance windows. If a transmission occurs within a defined acceptance window, the transmission is in-schedule, otherwise out-of-schedule. Time-triggered communication systems consider out-of-schedule transmissions as invalid. receive point acceptance window time Copyright TTTech Computertechnik AG. All rights reserved. Page 96

96 Clock Synchronization Principles Acceptance Windows (2) When a Synchronization Master (SM) transmits a PCF to a Compression Master (CM) and expects to receive it back in correct timing, the acceptance window is calculated like this: [ t dispatch + 2 * max_transmission_delay + compression_master_delay, t dispatch + 2 * max_transmission_delay + compression_master_delay + acceptance_window_size ] SM transmits the frame frame travels to CM frame gets echoed frame travels back to SM in here it must arrive max_ transmission_ delay compression_ master_ delay max_ transmission_ delay acceptance_ window_ size Copyright TTTech Computertechnik AG. All rights reserved. Page 97

97 Clock Synchronization Principles Transparent Clock TTEthernet components accumulate the transmission delays of each PCF. The accumulated value is called transparent_clock. PCF Generator (sender of the frame): transparent_clock = my static_send_delay + my dynamic_send_delay Each PCF forwarder (Switch): transparent_clock + = wire_delay to me + my static_relay_delay + my dynamic_relay_delay PCF Consumer (receiver): transparent_clock + = wire_delay to me + my static_receive_delay + my dynamic_receive_delay Note: The transparent_clock value in the PCF is modified by each TTE component along the frame s route. The receiver can read the PCF s total transmission delay in the PCF. Copyright TTTech Computertechnik AG. All rights reserved. Page 98

98 Clock Synchronization Principles Permanence Frames in a switched network can have different transmission delays. It is possible that receive order is different to the transmit order. Example: frame F1 is transmitted by node A at 10:00 frame F2 is transmitted by node B at 10:05 frame F1 has a transmission delay A C of 0:20 frame F2 has a transmission delay B C of 0:05 receiver C sees: frame F2 arrives at 10:10, then F1 arrives at 10:20 In a TTEthernet network, frame F2 is said to become permanent when it is certain that no frame F1, which was transmitted at an earlier point in time than F2, will be received anymore. TTEthernet needs to know when certain frames become permanent to run synchronization algorithms. B F1 10:05 A 10:00 F2 10:10 C 10:20 Copyright TTTech Computertechnik AG. All rights reserved. Page 99 Comp

99 Clock Synchronization Principles Permanence of PCFs Using the transparent_clock value, a receiver can determine the earliest safe point in time when a PCF becomes permanent: permanence_delay = max_transmission_delay transparent_clock permanence_point_in_time = receive_point_in_time + permanence_delay Example: max_transmission_delay in this network is 0:30 frame F1 is transmitted by node A at 10:00 frame F2 is transmitted by node B at 10:05 frame F1 has a transmission delay A C of 0:20. This is visible in F1 s transparent_clock frame F2 has a transmission delay B C of 0:05. This is visible in F2 s transparent_clock receiver C sees: F2 arrives at 10:10, becomes permanent at 10:10 + (0:30-0:05) = 10:35 receiver C sees: F1 arrives at 10:20, F1 becomes permanent at 10:20 + (0:30-0:20) = 10:30 F1 becomes permanent before F2 A 10:00 F1 B 10:05 F2 10:20 10:10 C Comp Copyright TTTech Computertechnik AG. All rights reserved. Page 100

100 Clock Synchronization Principles Reestablishing Total Frame Order (1) When discussing timing diagrams, we pretend that PCFs have no relevant transmission delays, and that they are always received simultaneously at all receivers. But in reality, each PCF accumulates the transparent_clock and is only used when the permanence function at the receiver allows. End System 1 End System 2 Switch 1 Switch 2 Switch 3 Dt inverted order! ES1 SW1 SW2 Earliest use at Switch 3 SW3 Comp ES2 Dt dataflow example: send order is restored by permanence function Copyright TTTech Computertechnik AG. All rights reserved. Page 101

101 Clock Synchronization Principles Reestablishing Total Frame Order (2) 302 dispatch 0 ES send 5 SM 1 SM 2 SM 3 ES 106 dispatch send SC 1 SM Switch 201 Switch 202 send receive 5 send receive SC 2 CM 1 SM 5 Switch 203 receive max_transmission_delay (=120) SM 6 permanence_delay ( = 110) Switch 203 permanence max_transmission_delay (=120) permanence_delay ( = 40) Copyright TTTech Computertechnik AG. All rights reserved. Page

102 Clock Synchronization Principles Membership Vector (1) Each Synchronization Master is associated 1:1 with a bit in a 32 bit vector called membership. Only synchronization masters (not compression masters) are represented in the membership. The definition is static and valid network-wide. A Sync B Sync bbbb bbbbbbbb Membership definition: A Bit 7 B Bit 6 C Bit 4 C Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 103

103 Clock Synchronization Principles Membership Vector (2) In each Integration Cycle, a Compression Master sets a bit in the membership vector if it receives an Integration PCF from a Synchronization Master. The membership vector in the Compression Master therefore indicates which Synchronization Masters have successfully contributed to the Integration Round. Sync Sync Sync Sync Comp Sync Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 104

104 Clock Synchronization Principles Membership Vector (3) When the Compression Master echoes back a PCF, it includes the membership in the PCFs. Synchronization Masters and Clients can see how reliable the PCF is: more membership bits set indicates higher number of Synchronization Masters contributed their votes. Sync Sync Sync Sync Comp Sync Sync Copyright TTTech Computertechnik AG. All rights reserved. Page 105

105 Clock Synchronization Principles Membership - Clock Quality In networks with multiple Compression Masters and channels, the Synchronization Clients can select the best (=most reliable) source of synchronization by selecting the PCF with the maximum membership (=most number of bits set) of all PCFs received. This provides immediate fault tolerance in case of failures of complete channels, Synchronization Masters, Compression Masters, or network links between any of these. Copyright TTTech Computertechnik AG. All rights reserved. Page 106

106 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 107

107 Fault Tolerance Fault Tolerance Basics (1) Fault Error Failure Failure: A system fails to deliver its intended service Error: Unintended system state Fault: cause of the error Fault classification Transient faults (EMI, SEU - single event upsets, ) Intermittent faults Permanent fault Copyright TTTech Computertechnik AG. All rights reserved. Page 108

108 Fault Tolerance Fault Tolerance Basics (2) Failure Type: Crash Failure: component stops silently (fail-silent), it does not produce any result at all. Consistent Failure: If there are multiple receivers, all receivers see the same erroneous result. Masquerading Failure: Component communicate with incorrect ID. Byzantine (inconsistent, malicious, asymmetric) Failure: the different receivers see differing results. Copyright TTTech Computertechnik AG. All rights reserved. Page 109

109 Fault Tolerance Fault Tolerance Basics (3) FT System Classification Fail safe systems The System can be set in a safe state at any time Stop all trains, set all train lights to red Fail operational The system must continue operation even if the part of distributed computer system fails Airplane cannot be set in the safe state at any time Copyright TTTech Computertechnik AG. All rights reserved. Page 110

110 Fault Tolerance Fault Tolerance in TTEthernet Redundant channels up to 3 end system channels COM/MON Switches Anti Masquerading Guardian (babbling idiot prevention) Fault tolerant startup mechanism Formal verification Fault tolerant clock synchronization mechanism Formal verification Copyright TTTech Computertechnik AG. All rights reserved. Page 111

111 Fault Tolerance Redundancy Management (1) One/Two/three channels configuration Copyright TTTech Computertechnik AG. All rights reserved. Page 112

112 Fault Tolerance Redundancy Management (2) The Ports, Links and Switches are duplicated for redundancy Frames are concurrently transmitted over both networks On the Receiving End-System, First Valid Frame wins ES Per VL End- System Transmit Frame Frame Network A Network B Frame Frame AFDX End-System ES Per VL End- System Receive Application Layer Task 1 M1 M2 Task 2 Task 3 M3 Communication Layer Channel 0 Channel 1 M1 M1 M2 M2 M3 M3 Copyright TTTech Computertechnik AG. All rights reserved. Page 113

113 Fault Tolerance Redundancy Management (3) If voting over redundant frames is required Endsystem can be configured such that it can deliver all redundant messages to host CPU. Host CPU can implements a SW layer (similar to FTCOM in TTP/C protocol) and perform different voting mechanisms Present it transparently to the host application. Application Layer M2 v Task 2 Task 3 M3 FTCOM Layer M2 M2 Communication Layer Channel 0 Channel 1 M2 M2 M3 M3 Copyright TTTech Computertechnik AG. All rights reserved. Page 114

114 Fault Tolerance Redundancy Management (4) Integrity Checking Integrity checking is done per VL and per Network Checking is based on Sequence Number & MFCL (Maximum Consecutive Frames Lost) All Invalid Frames are discarded Copyright TTTech Computertechnik AG. All rights reserved. Page 115

115 Fault Tolerance COM/MON Approach High integrity design: Self checking pair Two processor that execute same function in parallel Comparator checks output of both processors. If one processor fails (maliciously) and generates wrong data, second processors shuts down. Self-checking pair ensures fail-silence! Fail-silence Enforced in case of disagreement Copyright TTTech Computertechnik AG. All rights reserved. Page 116

116 Fault Tolerance Diagnosis Management (1) Switch and ES can send periodic diagnostic information with the status information Synchronization #TT received, #TT dropped messages. BAG errors, Period of diagnostic messages can be configured. Host CPU in addition can read the status fields of its connected end-systems. Copyright TTTech Computertechnik AG. All rights reserved. Page 117

117 Fault Tolerance Diagnosis Management (2) The TTEthernet devices can be queried to get their current state and additional diagnosis information. The TTEtherent devices provide diagnostic information as well as internal self checking mechanisms (IP Built-In Test results). ES TT Frame TT Frame TT Frame TT Frame ES TT Frame TT Frame Status query Status information Status query Status information TT Frame TT Frame ES TT Frame TT Frame TT Frame TT Frame ES Copyright TTTech Computertechnik AG. All rights reserved. Page 118

118 Fault Tolerance Diagnosis Management (3) The TTEthernet devices can also send periodic status and diagnosis information. This data can be scheduled with the other TT-messages or event-triggered traffic can be used (RC or BE). ES Status information Status information ES ES ES Copyright TTTech Computertechnik AG. All rights reserved. Page 119

119 Fault Tolerance TTEthernet Features Overview Anti Masquerading Based on configuration PORT VLID mapping Can be configured for BE traffic as well (on basis of MAC addresses) Guardian (babbling idiot prevention) Based on TT and RC configuration True only for TT and RC traffic BE traffic can be also controlled to prevent BE flooding Bandwidth allocation per MAC address and per port Copyright TTTech Computertechnik AG. All rights reserved. Page 120

120 AGENDA Introduction TTEthernet Basics Critical Traffic over TTEthernet Clock Synchronization Principles Fault Tolerance TTEthernet Products Overview Copyright TTTech Computertechnik AG. All rights reserved. Page 121

121 TTEthernet Product Overview TTEthernet Hardware Development Equipment Switch TTE Switch Lab 24 Ports E/S TTE PMC Card, TTE PCI Card TTE XMC Card, TTE PCIe Card Development Systems TTE Development System A664 TTE Development System v2.0 VxWorks 653 Chip IP TTE End Systems chip IP Certification Package (RTCA DO 254) Airborne Production Hardware DO-254, DO-178B, DO-160F (DAL A) Certifiable products TTE End System Pro Copyright TTTech Computertechnik AG. All rights reserved. Page 122

122 TTEthernet Product Overview TTEthernet 24 ports switch (Hardware) Switching Capability 18 x 10/100 Mbit/s Ethernet ports 6 x 10/100/1000 Mbit/s Ethernet ports FPGA-based Switch processor supporting 3 configurable traffic classes Best-effort traffic (IEEE 802.3) Rate-constraint traffic (full AFDX) Time-triggered traffic (SAE 6802 Standard) Management Built-in management module (MIP) for data loading and diagnostic Management function 4096 VLs with BAGs from 0.5 to 1600 ms ICMP, IP/UDP, SNMP Traffic shaping Arinc 615A/TFTP Data loading Health monitoring Built-in tests Multiple Configurations, PIN programming Form factors / environmental 19 Rack Copyright TTTech Computertechnik AG. All rights reserved. Page 123

123 TTEthernet Product Overview TTEthernet Software Configuration & Verification Tooling TTE Plan Network Planning TTE Build Network Configuration TTE Load / ARINC 615 Data Loader TTE Verify (certification DO 178C) Partition 1 Partition 2 Task 1A Task 1B Task 2A Task 2B Partition 3 Task 3A Embedded Software TTE Protocol Layer TTE Driver (Linux, Windows) TTE API Library TTE Sync Library TTE COM Layer ARINC 653 Core OS Partition OS TTE COM Layer ARINC 653 TTE API Library TTE PCI Driver TT RC BE Hardware Partition OS Message Channels (sampling and queuing ports) IMA OS Message Channel API Core OS Services TTEthernet middleware layer for time and space partitioned IMA OS (ARINC 653) Partition OS Enables TTEthernet data exchange through message channels (queuing and sampling ports) Copyright TTTech Computertechnik AG. All rights reserved. Page 124

124 TTEthernet Product Overview TTEthernet Tools (1) A Flexible Set of Development and Verification Tools for TTEthernet Networks and Systems Modeling of real-time communication requirements Modeling of network and topology Support for manual and automated design steps Based on open XML databases for flexible exchange with 3rd party tools Specialized editors for each design step TTE Plan: Network Scheduling (for TT) and Analysis Tool (for RC) TTE View: Packet Analyzer TTE Visualize: NC Database Explorer TTE Build Network Configuration TTE Build Device Configuration TTE SLC Wizard: GUI for TTE-Plan TTE Load: Data Loading Solution ARINC 615A Loader TTE Verify: RTCA/DO-178B Qualifiable Verification Tool Copyright TTTech Computertechnik AG. All rights reserved. Page 125 Page 125

125 TTEthernet Product Overview TTEthernet Tools (2) Ecore EMF Scripting Capability TTE Plan Network Configuration (Schedule) Generation Schedulability analysis for RC traffic Ecore EMF Scripting Capability Network Description XML Network Configuration XML High-level communication reqs. Senders, receivers, virtual links, sync domains, fault-tolerance requirements, etc. This stores the schedule (TT, RC, ET configs). Who sends what at what time (TT) at what rate (RC) on what route? TTE Build Network Configuration Device Config. Generation + GUI Editor TTE Build Device Configuration Image Generation Ecore EMF Scripting Capability Device Device Configuration XML XML Image Image This is a truthful, human readable XML representation of the binary tables in the switches and end systems. This is the binary image for a switch or end system, ready for download. Images for multiple devices in the system may be collected in a download database Copyright TTTech Computertechnik AG. All rights reserved. Page 126 Page 126

126 TTEthernet Product Overview TTEthernet Tools (3) TTE Plan TTE Build TTE Load System Spec.xml Net Config.xml Config Image.xml/.bin TTE Verify report TTE Verify report TTE Verify report Verifies correct implementation and traceability between highlevel and low-level requirements Copyright TTTech Computertechnik AG. All rights reserved. Page 127