Scheduling Real-Time Communication in IEEE 802.1Qbv Time Sensitive Networks

Transcription

1 Scheduling Real-Time Communication in IEEE 802.1Qbv Time Sensitive Networks Silviu S. Craciunas, Ramon Serna Oliver, Martin Chmelik, Wilfried Steiner TTTech Computertechnik AG RTNS 2016, Brest, France, October 18-21, 2016

2 Why TSN?

3 Time-Sensitive Networks IEEE TSN task group - collection of sub-standards that enhance 802 Ethernet with fully deterministic real-time capabilities Standard 802.1ASrev 802.1Qbv 802.1Qbu 802.1Qca 802.1Qcc 802.1Qci 802.1CB Description Timing & Synchronization Enhancements for Scheduled Traffic (Timed Gates for Egress Queues) Frame Preemption Path Control and Reservation Central Configuration Management Per-Stream Time-based Ingress Filtering and Policing Redundancy, Frame Replication & Elimination

4 Time-Sensitive Networks IEEE TSN task group - collection of sub-standards that enhance 802 Ethernet with fully deterministic real-time capabilities Standard 802.1ASrev 802.1Qbv 802.1Qbu 802.1Qca 802.1Qcc 802.1Qci 802.1CB Description Timing & Synchronization Enhancements for Scheduled Traffic (Timed Gates for Egress Queues) Frame Preemption Path Control and Reservation Central Configuration Management Per-Stream Time-based Ingress Filtering and Policing Redundancy, Frame Replication & Elimination

5 Time-Sensitive Networks IEEE TSN task group - collection of sub-standards that enhance 802 Ethernet with fully deterministic real-time capabilities Standard 802.1ASrev 802.1Qbv 802.1Qbu 802.1Qca 802.1Qcc 802.1Qci 802.1CB Description Timing & Synchronization Enhancements for Scheduled Traffic (Timed Gates for Egress Queues) Frame Preemption Path Control and Reservation Central Configuration Management Per-Stream Time-based Ingress Filtering and Policing Redundancy, Frame Replication & Elimination

6 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1

7 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1

8 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1

9 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1

10 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1

11 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1

12 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T=0

13 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T=0

14 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T=1

15 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T=1

16 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T= 2

17 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T= 2

18 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T= 3

19 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T= 3

20 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T= 4

21 IEEE 802.1Qbv Priority 8 Port A (ingress) Priority 7 Priority 6 Switching fabric Priority filter Port C (egress) Port B (ingress) Priority 1 T= 5

22 Functional parameters Device capabilities Queue configuration Scheduled Es Scheduled Sw Scheduled Es+Sw

23 Functional parameters Device capabilities Queue configuration Scheduled Es Scheduled Sw Scheduled Es+Sw

24 Functional parameters Device capabilities Queue configuration Scheduled Es Scheduled Sw Scheduled Es+Sw

25 Functional parameters Device capabilities Queue configuration Scheduled Es Scheduled Sw Scheduled Es+Sw Critical traffic assigned to the scheduled queues Non-critical traffic assigned to priority queues (post-analysis through network calculus Isolation: non-critical streams may interfere with each other in priority queues, but not with critical streams (isolated in the scheduled queues)

26 Network & traffic model multi-hop layer 2 switched network via full-duplex multi-speed links (multicast) TSN streams with multiple frames per stream synchronised time (<1 usec precision) wire and device delays d- epasks. traints, er to nt TTE A TTE B TTE-Switch 1 TTE C TTE-Switch 2 TTE E TTE D physical link communication path igure1: A TTEthernet networkwith5end-syste 2 switches. respectively. For a CPU link, the sp o allow heterogeneous CPUs wit in di erent WCETsfor th onstraint represents ny size can resid e (switcho in t Scheduled 802.1Qbv-compatible devices (Sw + Es) Scheduled (mutually exclusive) & priority queues Guaranteed delivery of critical traffic with known latency, small & bounded jitter

27 Deterministic Ethernet Constraints a d Frame 0 8 Stream Period 0 8 Period (1, 4) (1, 6) Link see also [Steiner@RTSS10] or [Craciunas@RTNS14]

28 Stream and e2e latency constraints Link Link Link maximum allowed e2e latency see also [Steiner@RTSS10] or [Craciunas@RTNS14]

29 Queue Interleaving

30 Queue Interleaving

31 Queue Interleaving In order to maintain jitter and latency requirements we expect at each device a certain timely order of frames

32 Queue Interleaving In order to maintain jitter and latency requirements we expect at each device a certain timely order of frames

33 Queue Interleaving

34 Queue Interleaving

35 Queue Interleaving expected

36 Queue Interleaving expected

37 Queue Interleaving expected

38 Queue Interleaving expected synchronization errors, frame loss, time-based ingress policing (e.g. IEEE 802.1Qci) may lead to non-deterministic placement in queues during runtime timed gates control events on the egress port, not the order of frames in the queue placing of frames in the scheduled queues at runtime may be non-deterministic Timely behaviour of streams may oscillate, accumulating jitter for the overall end-to-end transmission

39 Queue Isolation

40 Queue Isolation

41 Queue Isolation expected

42 Queue Isolation expected

43 Queue Isolation expected

44 Queue Isolation expected Solves the non-determinism problem but reduces the solution space

45 Stream (Flow) isolation

46 Stream (Flow) isolation

47 Stream (Flow) isolation expected

48 Stream (Flow) isolation expected

49 Stream (Flow) isolation expected

50 Stream (Flow) isolation expected

51 Stream (Flow) isolation expected

52 Stream (Flow) isolation expected Once a flow has arrived, no other flow can arrive in the same queue until the first flow has been completely sent Better than queue isolation but still restrictive

53 Stream (Flow) isolation expected Once a flow has arrived, no other flow can arrive in the same queue until the first flow has been completely sent Better than queue isolation but still restrictive 0 t1 t1 t1 0 t2

54 Stream (Flow) isolation expected Once a flow has arrived, no other flow can arrive in the same queue until the first flow has been completely sent Better than queue isolation but still restrictive 0 t1 t1 t1 0 t2

55 Frame isolation

56 Frame isolation

57 Frame isolation expected

58 Frame isolation expected

59 Frame isolation expected

60 Frame isolation expected

61 Frame isolation expected

62 Frame isolation expected

63 Frame isolation expected

64 Frame isolation expected

65 Frame isolation expected

66 Frame isolation expected Ensure that there are only frames of one flow in the queue at a time Frames from another flow may only enter the queue if the already queued frames of the initial flow have been serviced Less performant than stream isolation since the solver has to consider at all frame interleavings

67 802.1Qbv scheduling constraint The constraint for minimum jitter scheduling of critical traffic for 802.1Qbv networks is: isolate frames/streams in the time domain OR isolate streams in different queues

68 802.1Qbv configurations Only critical traffic (serialized similar to bus systems) Legacy AVB systems that require a few additional highcriticality flows [Specht@ECRTS16] Maximize solution space for critical traffic, non-critical traffic can be scheduled by inverting the cumulated schedule of scheduled queues High-criticality applications that feature both scheduled and non-scheduled traffic, trade-off between schedulability of critical traffic and timeliness properties and flexibility for non-scheduled traffic Standard AVB (IEEE 802.1BA) network in which flows are serviced according to the priority

69 Scheduling problem

70 Scheduling problem Find offsets and queue assignments for individual frames of TSN streams along the route that conform to the constraints

71 Scheduling problem Find offsets and queue assignments for individual frames of TSN streams along the route that conform to the constraints Reduces to finding a solution for a set of inequalities resulting from frame constraints link constraints stream constraints end-to-end latency constraints stream or frame isolation constraints 802.1Qbv

72 Scheduling problem Find offsets and queue assignments for individual frames of TSN streams along the route that conform to the constraints Reduces to finding a solution for a set of inequalities resulting from frame constraints link constraints stream constraints end-to-end latency constraints stream or frame isolation constraints 802.1Qbv NP-complete

73 Satisfiability Modulo Theories satisfiability of logical formulas in first-order formulation background theories variables logical symbols non-logical symbols quantifiers x 1,x 2,...,x n 9, 8 _, ^,, (, ) +, =, %, apple LA(Z) BV optimization (OMT) [Bjørner@TACAS15] A lot of solvers and a very active community OpenSMT [Bruttomesso@TACAS10] CVC4 [Barrett@CAV11] Yices [Dutertre@CAV14] Z3 [de Moura@TACAS08]

74 Satisfiability Modulo Theories satisfiability of logical formulas in first-order formulation background theories variables logical symbols non-logical symbols quantifiers x 1,x 2,...,x n 9, 8 _, ^,, (, ) +, =, %, apple LA(Z) BV optimization (OMT) [Bjørner@TACAS15] A lot of solvers and a very active community OpenSMT [Bruttomesso@TACAS10] CVC4 [Barrett@CAV11] Yices [Dutertre@CAV14] Z3 [de Moura@TACAS08]

75 Optimization Optimize schedule with respect to certain properties of the system (e.g. minimize end-to-end latency of selected streams) 802.1Qbv-specific optimizations: QoS properties: minimize required scheduled queues in order to increase QoS properties of non-critical traffic Design space exploration in case of infeasible use-cases, i.e. find the minimal number of queues required for scheduled traffic such that a schedule is found Many more optimization opportunities in combination with other TSN sub-standards (e.g. frame preemption)

76 Experiments Z3 v4.4.1 solver (64bit) (Yices v2.4.2 with quantifier-free linear integer arithmetic) 64bit 4-core 3.40GHz Intel Core-i7 PC with 4GB memory 3 predefined topologies ranging from 3 end-systems connected to one switch to 7 end-systems connected through 5 switches via 1Gbit/s links with a 1usec macrotick granularity (generate high utilization on the links) Time-out value for a run to 5 hours System configuration: Scalability and schedulability experiments

77 Scalability Experiments Frame isolation method (using an incremental backtracking algorithm with step size of 1) Vary the problem set in 3 dimensions: 1. topology size, 2. number of flows, 3. flow periods (chosen randomly from 3 sets of predefined periods) Data size uniformly between 2 and 8 MTU-sized frames Senders and receivers are chosen randomly

78 Scalability Experiments 4 h 30 min 1 min 1 sec runtime 10 ms P1={10, 20}[ms] P2={10, 25, 50, 100}[ms] P3={5, 10, 200, 500}[ms] 5(40) 15(144) 25(238) Number of flows (Average number of frame instances)

79 Scalability Experiments 4 h 30 min 1 min 1 sec runtime 10 ms P1={10, 20}[ms] P2={10, 25, 50, 100}[ms] P3={5, 10, 200, 500}[ms] 5(52) 25(310) 50(641) Number of flows (Average number of frame instances)

80 Scalability Experiments 4 h 30 min 1 min 1 sec runtime 10 ms P1={10, 20}[ms] P2={10, 25, 50, 100}[ms] P3={5, 10, 200, 500}[ms] 10(125) 50(738) 100(1490) Number of flows (Average number of frame instances)

81 Frame vs. Stream Isolation 381 randomly generated test cases with up to 1000 streams 17 reached the time-out Stream isolation was on average 13% faster with a median of 8.03% 36.7h for stream isolation and 59h for frame isolation % improvement M1 over M2 reduction [%] complex/p3/100 complex/p2/100 complex/p1/100 complex/p3/50 complex/p2/50 complex/p1/50 complex/p3/10 complex/p2/10 complex/p1/10 medium/p3/50 medium/p2/50 medium/p1/50 medium/p3/25 medium/p2/25 medium/p1/25 medium/p3/5 medium/p2/5 medium/p1/5 simple/p3/25 simple/p2/25 simple/p1/25 simple/p3/15 simple/p2/15 simple/p1/15 simple/p3/5 simple/p2/5 simple/p1/5 M1 runtime as percentage of M2 runtime Runtime reduction through M1

82 Schedulability Experiments Generated inputs that force streams to interleave if scheduled in the same egress queue Runs w/ and w/o optimization objectives using both stream and frame isolation methods Minimize accrued sum of the number of queues used per egress port No incremental steps for optimization runs

83 Schedulability Experiments 10 min Flow isolation Frame isolation Optimized Flow isolation Optimized Frame isolation 1 min 10 sec 1 sec runtime 100 ms T20 T19 T18 T17 T16 T15 T14 T13 T12 T11 T10 T09 T08 T07 T06 T05 T04 T03 T02 T01 Flow isolation Frame isolation queues

84 Conclusions

85 Conclusions Scheduling problem arising from the IEEE 802.1Qbv extension on multi-hop fully switched TSN networks

86 Conclusions Scheduling problem arising from the IEEE 802.1Qbv extension on multi-hop fully switched TSN networks key functional parameters affecting the behaviour of 802.1Qbv networks

87 Conclusions Scheduling problem arising from the IEEE 802.1Qbv extension on multi-hop fully switched TSN networks key functional parameters affecting the behaviour of 802.1Qbv networks 802.1Qbv-specific constraints for creating correct offline schedules guaranteeing low and bounded jitter as well as deterministic end-to-end latencies for critical traffic

88 Conclusions Scheduling problem arising from the IEEE 802.1Qbv extension on multi-hop fully switched TSN networks key functional parameters affecting the behaviour of 802.1Qbv networks 802.1Qbv-specific constraints for creating correct offline schedules guaranteeing low and bounded jitter as well as deterministic end-to-end latencies for critical traffic use SMT/OMT solvers to create static schedules for 802.1Qbv devices

89 Conclusions Scheduling problem arising from the IEEE 802.1Qbv extension on multi-hop fully switched TSN networks key functional parameters affecting the behaviour of 802.1Qbv networks 802.1Qbv-specific constraints for creating correct offline schedules guaranteeing low and bounded jitter as well as deterministic end-to-end latencies for critical traffic use SMT/OMT solvers to create static schedules for 802.1Qbv devices optimization directions & system configurations and their trade-offs

90 Conclusions Scheduling problem arising from the IEEE 802.1Qbv extension on multi-hop fully switched TSN networks key functional parameters affecting the behaviour of 802.1Qbv networks 802.1Qbv-specific constraints for creating correct offline schedules guaranteeing low and bounded jitter as well as deterministic end-to-end latencies for critical traffic use SMT/OMT solvers to create static schedules for 802.1Qbv devices optimization directions & system configurations and their trade-offs evaluation in terms of scalability and schedulability

91 Thank you!