An Introduction to TTEthernet - PDF Free Download

An Introduction to thernet TU Vienna, Apr/26, 2013 Guest Lecture in Deterministic Networking (DetNet) Wilfried Steiner, Corporate Scientist wilfried.steiner@tttech.com Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 1

Future Markets for Real-Time Fault-Tolerant Communication Requirements on a communication infrastructure for future markets Real-time requirements Fault tolerance requirements Low cost Low power Low weight Low size Consumer acceptance A system failure potentially leads to Loss of life Loss of economic assets Loss of research results Loss of power Loss of quality of service (QoS) Any bad thing we can think of Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 3

Closed and Open World Communication Closed World Communication Performance guarantees: real-time, dependability, safety Standards: ARINC 664, ARINC 429, TTP, MOST, FlexRay, CAN, LIN, Applications: Flight control, powertrain, chassis, passive and active safety,.. Validation & verification: Certification, formal analysis,... Open World Communication No performance guarantees: best efforts Standards: Ethernet, TCP/IP, UDP, FTP, Telnet, SSH,... Applications: Multi-media, audio, video, phones, PDAs, internet, web, Validation & verification: No certification, test, simulation,... High cost Low cost We see a market requirement to use the same physical network for data flows from both worlds. Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 4

Mixed-Criticality Systems Windows PC Windows PC Open Networks How to share system resources and partition critical and non-critical distributed functions? Linux Server Standard IEEE802.3 Ethernet LAN Network thernet F2 F4 F1 F2 F3 F4 F1 F2 F4 Time and space partitioned OS F1 F2 F3 F4 Time and space partitioned OS Time and space partitioned OS Time and space partitioned OS Safety-, Time- or Mission-Critical System Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 5

Traffic Classes thernet provides several traffic classes in parallel: time-triggered, rate-constrained, and best-effort Time-Triggered: dispatch messages according a predefined communication schedule Rate-Constrained: enforce minimum duration between two frames of the same stream Best-Effort: standard Ethernet communication paradigm no temporal guarantees are given Layer 3-7 Application Time-Triggered Extension Ethernet IEEE 802.3 thernet 40 msec 40 msec 40 msec TT1 TT2 RC RC BE TT1 BE RC TT2 BE TT1 RC BE RC TT2 TT1 BE BE RC RC 30 msec 30 msec 30 msec 30 msec TT1 TIME Longest Communication Cycle in this Example: LCM(30,40) = 120msec Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 6

thernet, a Communication Infrastructure Highlight: Flexible Integration and COTS Backward Compatible ETH FX CAN FX CAN FX CAN FX CAN <1 Mbit/sec FX < 10 Mbit/sec FX 1 Gbit/sec TTP TTP TTP TTP 100 Mbit/sec Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 7

The Motivation for Ethernet Ethernet hardware is low cost. Ethernet is a well-established open-world standard and very scaleable. The OSI reference model gives a well-structured classification of concepts that can be built on top of Ethernet. Existing tools can be leveraged as cost-efficient diagnosis tools. As all messages in thernet are standard Ethernet compliant, existing tools can be leveraged for time-triggered messages as well. Standard web servers can be leveraged for maintenance and configuration. Engineers learn about Ethernet at school. Ethernet compatibility enables the usage of technology that is established, tested, and verified. Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 8

Outline Prerequisites for Safe and Deterministic Communication Asynchronous vs. Synchronous Communication Clock Synchronization and Fault-Tolerant Clock Synchronization Formal Verification Activities Utilization of Safe and Deterministic Communication Time-Triggered Communication Constraints in Multi-Hop Networks Integrated communication for mixed-criticality systems Combined Time-Triggered / Rate-Constrained / Best-Effort Communication Tooling Overview Summary Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 9

Ethernet = Asynchronous Communication NIC NIC NIC SWITCH X SWITCH X NIC NIC NIC X NIC NIC Asynchronous Communication Transmission Points in Time are not predictable Transmission Latency and Jitter accumulate Number of Hops has a significant impact Usually solved by High Wire-Speeds & Low Utilization and/or Priorities Problem of ``Indeterminism remains SWITCH NIC NIC NIC Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 11

Adding Clock Synchronization to Ethernet Eth Time Master IN 1 Eth Enabler for Synchronous Operation: Synchronized Global Time 1588 1588 Communication Schedule Eth Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 12

Quality of Clock Synchronization: Precision In an ensemble of clocks, the precision is defined as the maximum distance between any two synchronized nonfaulty clocks at any point in real time. Late Clock Perfect Clock Early Clock Page 13

Time-Triggered Operation Time-Division Multiple-Access Communication Composable network Complexity reduction and faster integration Fault tolerant communication system Node A send receive receive t 1 t 2 t 3 Node B receive send receive t 1 t 2 t 3 Node C receive receive send t 1 Slot t 2 t 3 time Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 14

Synchronous Communication (TT) NIC NIC NIC SWITCH SWITCH NIC NIC NIC X NIC NIC Synchronous Communication SWITCH NIC X Exactly one order of messages M i (in contrast to PERM(M i ) in async. comm) NIC NIC Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 15

Single-Master Synchronization Eth IN 1 IN 1 IN 1 IN 1 IN 1 IN 1 IN 1 IN 1 Time Master Eth constant and/or dynamic IN 1 1588 1588 Eth Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 17

Transparent Clock and Permanence dispatch 302 0 SM 1 SM 2 SM 3 ES 102 302 send 5 dispatch 306 0 SC 1 SM 4 ES 106 Switch 201 Switch 202 send send receive 5 send receive 306 302 5 302 45 302 45 70 SC 2 CM 1 SM 5 SM 6 Switch 203 receive 10 306 302 302 80 max_transmission_delay (=120) permanence_delay (120 10 = 110) Switch 203 permanence max_transmission_delay (=120) permanence_delay (120 80 = 40) 302 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 110 120 130 140 150 105 115 125 135 145 306 Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 18

Synchronization Services Clock Synchronization Service Clock Synchronization Service is executed during normal operation mode to keep the local clocks synchronized to each other. Startup/Restart Service is executed to reach an initial synchronization of the local clocks in the system. Integration/Reintegration Service is used for components to join an already synchronized system. Clique Detection Services are used to detect loss of synchronization and establishment of disjoint sets of synchronized components. Computer Time Fast Clock Perfect Clock Message Exchange Message Exchange Slow Clock Startup/Restart Service R.int R.int Real Time Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 19

Single-Master Clock Synchronization Eth Time Master IN 1 Eth Enabler for Synchronous Comm.: Synchronized Global Time 1588 1588 Communication Schedule Eth Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 20

Fault-Tolerant Clock Synchronization Time Master IN 1 IN 1 Eth IN 1 IN 1 Time Master Time Master IN 1 IN 1 Eth 1588 Fault-tolerant synchronization services are needed for establishing a safe global time base Eth 1588 Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 21

Step 1: ALL Synchronization Master Dispatch IN Frames at the SAME Scheduled Point in Time Compression Master IN 1 Synchronization Master 1 IN 5 Synchronization Master 5 IN 2 IN 3 IN 4 Synchronization Master 2 Synchronization Master 3 Synchronization Master 4 Precision Dispatch SM1 SM2 SM5 Permanence SM1 SM2 SM5 SM4 SM3 SM4 SM3 Acceptance Window (of SM 2/5)... CM CM t_0 t_1, t_2 t_4, t_5 Reference Point Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 22

Step 2: Compression Master Dispatch Compressed IN Frame back to Synchronization Masters/Clients Compression Master IN C Synchronization Master 1 Synchronization Master 5 Synchronization Master 2 Synchronization Master 3 Synchronization Master 4 Precision Dispatch SM1 SM2 SM5 Permanence SM1 SM2 SM5 SM4 SM3 SM4 SM3 Acceptance Window (of SM 2/5)... CM CM t_0 t_1, t_2 t_4, t_5 Reference Point Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 23

Other Synchronization Safety Mechanisms Controlled and autonomous late integration Synchronous operation will be reached when a sufficient number of ECUs is powered-up. Remaining ECUs may power up at arbitrary times and will join synchronous operation. Controlled and autonomous re-integration ECUs that drop out of the synchronous operation will autonomously reintegrate after recovery. Controlled and autonomous system-wide reset In the extremely unlikely event that the synchronous time-base is lost, the system is configurable to automatically execute a controlled system-wide restart. Synchronization robustness against EMI Synchronization is configurable to continue operation without receiving synchronization messages for a parameterized number of re-synchronization intervals. Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 26

Formal Verification Activities thernet Executable Formal Specification Using symbolic and bounded model checkers sal-smc and sal-bmc Focus on Interoperation of Synchronization Services (Startup, Restart, Clique Detection, Clique Resolution, abstract Clock Synchronization) Verification of Lower-Level Synchronization Functions Permanence Function (sal-inf-bmc + k-induction) Compression Function (sal-inf-bmc + k-induction) Formal Verification of Clock Synchronization Algorithm First time by means of Model Checking (sal-inf-bmc + k-induction) Re-use of the Formal Models to prove: Layered clock-rate correction algorithm (sal-inf-bmc + k-induction) Layered clock-diagnosis algorithm (sal-inf-bmc + k-induction) Verification and minor corrections of the Sparse Timebase Concept Distributed computations without explicit coordination (PVS) Work has mostly been done in the context of the Marie Curie CoMMiCS project FP7 (FP7/2007-2013) project no. 236701 CoMMiCS Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 27

References B. Dutertre, A. Easwaran, B. Hall, W. Steiner, Model-based analysis of Timed-Triggered Ethernet, Proceedings of the 31st IEEE/AIAA Digital Avionics Systems Conference (DASC 2012), IEEE 2012, Recipient of Best in Session and Best in Track awards W. Steiner, G. Bauer, B. Hall and M. Paulitsch, Time-Triggered Ethernet: thernet, In Time-Triggered Communication, R. Obermaisser, editor, CRC Press, 2011 W. Steiner and J. Rushby, TTA and PALS: Formally Verified Design Patterns for Distributed Cyber- Physical Systems, Proceedings of the 30th IEEE/AIAA Digital Avionics Systems Conference (DASC 2011), IEEE 2011, Recipient of Best in Session and Best in Track awards W. Steiner and B. Dutertre, Layered Diagnosis and Clock-Rate Correction for the thernet Clock Synchronization Protocol, Proceedings of the 17th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2011), IEEE Computer Society, 2011 W. Steiner and B. Dutertre, Automated Formal Verification of the thernet Synchronization Quality, Proceedings of the 3rd NASA Formal Methods Symposium (NFM 2011), Springer Lecture Notes in Computer Science, 2011 W. Steiner and B. Dutertre, SMT-Based Formal Verification of a thernet Synchronization Function, Proceedings of the 15th International Workshop on Formal Methods for Industrial Critical Systems (FMICS 2010), Lecture Notes in Computer Science 6371 Springer, 2010, pp. 148-163 Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 28

End-To-End (E2E) TT Dataflow offset_de offset_ad offset_ef D F A G B C E H Physical Topology Dataflow Path Virtual Link TT frames can be scheduled on each communication link. The communication schedule needs to satisfy constraints as discussed in the following. Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 31

Contention-Free Constraints i Definition A sender or relaying instance will dispatch a new frame only after the previous frame has been processed. In a pure time-triggered network, the term processed refers to the transmission of the previous frame. In a mixed time-triggered / event-triggered network, the term processed can be relaxed as the previous time-triggered frame may get delayed by an event-triggered frame in transition. Cluster Cycle no overlaps Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 32

End-to-End Constraints Definition The end-to-end transmission constraints are derived from the application and assumed to be provided by the user. They describe the worst-case maximum and optionally also worst-case minimum allowed latency for a frame x. In general we assume that the bounds specified will be the same for all receivers of the frame x. Cluster Cycle Cluster Cycle bound Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 33

Path-Dependent Constraints Definition Within the dataflow path of a frame x the dispatch points in time of two adjacent edges will be well-timed. This means that the dispatch point in time of a succeeding edge will be scheduled only after it was received from the preceding edge. offset_ad A e.g., slot = 5 D F G offset_de B C E H Physical Topology Dataflow Path Virtual Link offset_ef Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 34

Bounded-Memory Constraints Definition The restrictions of switch memory generates another implementationimposed set of constraints. The memory size required to prevent buffer overflows in the switch can also be expressed in terms of time. e.g., slot = 5 e.g., slot < 8 Cluster Cycle Cluster Cycle bound Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 35

Simultaneous Relay Constraints Definition Though, not conceptually a requirement, there may me an implementation-derived requirement in the switches to dispatch a frame x on all ports simultaneously. Cluster Cycle Cluster Cycle ~ same points in time Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 36

Application-Level Constraints i Definition Application-level dependency constraints describe requirements that span multiple frames x_i. E.g. x_1 has to be dispatched 17.3 ms before x_2. That s the main complexity driver! Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 37

Application-Level Constraints ii Physical Part Physical Process Sensor Control Actuator Cyber Part CPU NIC 1 Switch 2 CPU NIC 3 4 Switch CPU NIC 5 Switch 6 4 shall be sent x ms after 3 is received Capture Sensor Value Calculate Control Value Task Schedule Operate Actuator Interrupts can be generated by a synchronized time reaching scheduled points in time. NIC Switch 1 Switch Switch Switch NIC 2 3 a b c d e f g h i 4 5 6 Frame Schedule Scheduled Events on the Timeline In several safety-relevant and safety-critical systems, synchronized time is a fundamental building block. Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 38

Emerging Benefits of using TT Consistent Distributed Computing Base Unification of Interfaces Temporal Firewalls Composability Independent Development of ECUs Stability of Prior Services Constructive Integration Replica Determinism Scalability Transparent Implementation of Fault Tolerance Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 40

thernet for Mixed-Criticality Systems Enables robust partitioning of all computing and networking resources in one system Fault-tolerant distributed clock Hard real time communication (µs jitter, fixed latency) host critical controls, video, audio, LAN, Layer 3-7 In parallel, two types of Ethernet communications: Synchronous (TDMA-style) Communication: TT Application Time-Triggered Extension Asynchronous (event-triggered style): RC + BE Ethernet IEEE 802.3 thernet 40 msec 40 msec 40 msec TT1 TT2 RC RC BE TT1 BE RC TT2 BE TT1 RC BE RC TT2 TT1 BE BE RC RC 30 msec 30 msec 30 msec 30 msec TT1 TIME Longest Communication Cycle in this Example: LCM(30,40) = 120msec Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 44

thernet Dataflow: Rate-Constrained Traffic Rate-Constrained Traffic (RC) Switch/Router Receiver Sender min. duration min. duration min. duration Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 45

Mixed Traffic on Ethernet RC Accumulated Jitter 00:10 2 00:01 1 4a 3a 2a 1a thernet Switch Time Triggered 00:11 00:02 Rate Constrained 4b 3b 2b 1b Best Effort Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 46

Mixed Traffic on an Ethernet RC Accumulated Jitter 00:10 TT is dispatched according synchronized time 00:01 4a thernet Switch Time Triggered 00:11 00:02 Rate Constrained 3a 3b TT is forwarded according synchronized time 2 1a TT has lowest latency and lowest jitter 1b 1 4b 2b Best Effort 2a RC frame delivery is guaranteed, but potentially has high latency and jitter RC potentially queue-up in switch memory Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 47

Mixed Traffic on an Ethernet BE Buffer Overflow thernet Switch Time Triggered Rate-constrained frame delivery (standard Ethernet traffic) is guaranteed! Rate Constrained 3 2 1 Best Effort 4 2 1 3 Best-effort frame delivery (standard Ethernet traffic) is NOT guaranteed! Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 49

Integrated Dataflow Example TT BE TT BE TT BE t 3ms cycle 3ms cycle 3ms cycle Sender 1 Switch/Router Dataflow Integration - Time-Triggered (TT) - Rate-Constrained (RC) - Standard Ethernet (BE) Receiver Sender 2 TT TT RC BE TT TT BE BE TT RC TT TT BE t TT BE BE TT RC TT BE 3ms cycle 3ms cycle 3ms cycle 2ms cycle 2ms cycle 2ms cycle t 2ms cycle 2ms cycle 2ms cycle 2ms cycle 6ms Cluster Cycle thernet Switches are non-preemptive store-and-forward switches using priorities Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 50

Integration Options When two (or more) messages compete for relay to the same outgoing port, the switch has to serialize these messages. Typically, a priority mechanism will be used. Priority is easy, when there is a clear winner in terms of priority. If there are messages of same priority the messages will be serviced according FIFO. What happens if there is a low-priority message (L) in relay, when a high-priority message (H) becomes ready for relay? Implemented in early (academic) versions of TT-Ethernet Contention: Preemption: Timely Block: Shuffling: Implemented in current versions of thernet Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 51 L L H H H H L L real-time

Example: 1,000 Frames (Industrial-Sized) 2 1 Dataflow Links are enumerated on the x-axis 3 4 6 5 1 2 RC/BE frames are also integrated during TT phases. RC TT X RC TT RC TT RC TT Time-Triggered Only Time-Triggered + Event-Triggered Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 52

Tools Requirements Data Flow Overview Plan Network Config. (Schedule) Generation (currently -Demo Scheduler) Build Network Configuration Plug-in Device Config. Generation Build Basic Image Generation System Specification XML Network Configuration XML Device Device Configuration XML XML Image Image High-level communication reqs. Senders, receivers, virtual links, sync domains, fault-tolerance requirements, etc. This stores the schedule (TT, RC, ET configs). Who sends what at what time (TT) at what rate (RC) on what route? This is a truthful, human readable XML representation of the binary tables in the switches and end systems. This is the binary image for a switch or end system, ready for download. Images for multiple devices in the system may be collected in a download database Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 54

Summary and Conclusion Cyber-physical systems become more and more complex with an increasing demand on resources. Determinism is a key concept to manage complexity and to ensure system safety. The integration of applications with mixed-criticality requirements, so that they share resources, allows cost-effective architectures for realtime and safety-critical systems. Ethernet is a good basis for an integrated communication infrastructure. Enabling Ethernet with time-triggered services (thernet) generates a deterministic communication infrastructure for mixedcriticality systems that allows synchronous and asynchronous communication. The synchronized global time protects highly critical dataflows from less critical or uncritical ones. Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 56

E n s u r i n g R e l i a b l e N e t w o r k s w w w. t t t e c h. c o m Wilfried Steiner, Senior Research Engineer wilfried.steiner@tttech.com Copyright TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 58