Hybrid EDF Packet Scheduling for Real-Time Distributed Systems

Transcription

1 Hybrid EDF Packet Scheduling for Real-Time Distributed Systems Tao Qian 1, Frank Mueller 1, Yufeng Xin 2 1 North Carolina State University, USA 2 RENCI, University of North Carolina at Chapel Hill This work was supported in part by NSF grants , , and Aranya Chakrabortty helped to scope the problem in discussions.

2 Motivation Build up distributed systems to support real-time tasks that require multiple resources over networks. E.g., real-time power grid state monitor and control. 2

3 Motivation Build up distributed systems to support real-time tasks that require multiple resources over networks. R2 Messages R3 Network R1 R4 R1->R2->R3->R4->R1 3

4 Problem Resource scheduler needs to schedule two parts of a distributed task l Initial (on R1) Fixed period l Consecutive (on R2, R3, R4, R1) Released to handle messages Large release jitter due to network transmission time variation Unpredictable interruption to the scheduler How to increase predictability of the system? 4

5 Idea l Utilize periodic tasks to transmit messages/release jobs To eradicate the randomness of interruption l Extend the functionality of the system network layer To prioritize messages according to their urgency (deadlines of corresponding real-time tasks) l Enforce bandwidth limitation strategy To reduce network traffic burstiness/contention To reduce message transmission time variation 5

6 Idea We regulate distributed tasks, so theory of uniprocessor system can be adopted. Utilization-based schedulability test Acceptance test for sporadic tasks Response time based schedulability test 6

7 Solution On each node, l l New: Combine periodic transmission tasks with EDF scheduler New: Add EDF packet scheduler/bandwidth limitation on network layer l New: Support utilization-based schedulability test EDF Scheduler Periodic Transmission Tasks EDF Packet Scheduler Bandwidth Limitation Messages Network R3 R4 R1 7

8 Periodic Transmission Tasks l Messages/jobs are released by periodic receiving task l Messages are sent by periodic sending task l Interruption times are controlled Sender Network Receiver τ Sending Task Receiving Task τ' 8

9 Periodic Transmission Tasks Let k = max. number of jobs of a task. Worst case happens when k jobs are released in one frame. k is upper bounded by the range of network variation (δ + - δ - ). T r : period of receiving task T : minimal inter-transmission time guaranteed by the sender 9

10 Periodic Transmission Tasks l Relative deadline of these tasks can be calculated (0 s < k): l Worst case system utilization is calculated on every node to check the schelulability (sufficient test). ρ 1 10

11 EDF Packet Scheduler l EDF task scheduler may not schedule earlier tasks strictly first since only receiving job can accept messages and release corresponding jobs. E.g., on node R: Message Deadline Arrival Time on R Reception Time on R Message Sending Time on R A B l Task scheduler passes messages into network layer in reversed order. 11

12 EDF Packet Scheduler Prioritize the messages in the network sending queue so that messages with earlier deadlines have high priority to be sent. 12

13 EDF Packet Scheduler We extend the Linux network stack l Let application pass message deadlines to the network stack in kernel l Add EDF queue to transmit message of earlier deadline first 13

14 EDF Packet Scheduler Left: configure deadline for following messages Right: transmit messages in EDF order Application setsockopt (SO_DEADLINE) write sendto socket deadline copy message deadline (sk_buff) EDF queue (data link layer) each socket Transmit in EDF order (physical layer) each interface Linux Network Stack Extension 14

15 Bandwidth Limitation Each node limits the size of data to send in a period of time, with the intention to Reduce burstiness/contention of the underlying network Reduce the network variation Reduce the number of jobs from the same task released by one accepting job 15

16 Bandwidth Limitation l Passive strategy: cannot control the network resource directly l Used in implementation: hierarchical token bucket queue of data link layer l Experimental results show it actually reduces network variation 16

17 Evaluation l Multiple nodes connected with network send ping-pong messages with deadlines to each other l Workload is quantified by (mps, burst): mps: number of frames in one second burst: number of messages sent in one frame Each message has 400 bytes l To prove: Bandwidth limitation reduces network variation EDF packet scheduler reduces deadline miss rate 17

18 Experimental Setup l 8 nodes connected with one switch (Gigabit Ethernet) l 2 cores of 16 cores (in total) on each node are utilized One core is used to send ping messages to 8 nodes, and process the corresponding pong messages. The other core is used to send pong messages. l Each node runs the modified version of Linux , which includes the EDF packet scheduler 18

19 Bandwidth Limitation Result l Workload mps=100, burst is chosen from [5, 10] in a round robin fashion l Blue plots are organized into different clusters, which remained in the queue for different times Network Delay (ms) No Bandwith Sharing Bandwith Sharing Network Delay Variance Bandwidth limitation reduces network variation from 0.97ms to 0.55ms. 19

20 Bandwidth Limitation Result l Network variation are reduced on all 8 nodes No Bandwith Sharing Bandwith Sharing Network Delay (ms) Node-1 Node-2 Node-3 Node-4 Node-5 Node-6 Node-7 Node-8 20

21 EDF Packet Scheduler Result l Experiments of different (mps, burst) are conducted without EDF packet scheduler and then replayed with EDF packet scheduler. l Relative deadline of messages are random uniformly drawn from [D, 2D]. D is configurable. l No bandwidth limitation. 21

22 EDF Packet Scheduler Result l Deadline miss rate is reduced more by the EDF packet scheduler when the network has more congestion. Deadline Miss Rate (%) NoEDF 0.8-EDF 1.0-NoEDF 1.0-EDF , , , , , , , , , , , , , , , 200 Workload (mps, burst) 22

23 More in the Paper l Extended the distributed task model from a pair of nodes to multiple nodes on the path. l Implemented our hybrid packet scheduler in a real-time distributed storage system. l Experimentally proved the formulae to calculate the number of jobs released in one frame for one task in the worst case (the basis for our schedulability test). 23

24 Conclusion l Designed and implemented a system for distributed real-time tasks EDF task scheduler + periodic message transmission tasks EDF packet scheduler Bandwidth limitation l Experiments demonstrate the effectiveness of preserving EDF order + bandwidth reservation through the network stack 24

25 Promising Future Work l How to derive the bandwidth demand from the task set? l How to actively manage network resources to further reduce network variation, when network topology gets more complicated? Thanks! Questions? 25

26 References l l l l l l Heechul Yun, Gang Yao, Rodolfo Pellizzoni, Marco Caccamo, and Lui Sha. Memguard: Memory bandwidth reservation system for efficient performance isolation in multi-core platforms. In Real-Time and Embedded Technology and Applications Symposium (RTAS), 2013 IEEE 19th, pages IEEE, Benoˆıt Dupont de Dinechin, Duco van Amstel, Marc Poulhies, and Guillaume Lager. Timecritical computing on a single-chip massively parallel processor. In Design, Automation and Test in Europe Confer- ence and Exhibition (DATE), 2014, pages 1 6. IEEE, Jakob Rosen, Alexandru Andrei, Petru Eles, and Zebo Peng. Bus access optimization for predictable implementation of real-time applications on multiprocessor systems-on-chip. In Real-Time Systems Symposium, RTSS th IEEE International, pages IEEE, Juan M Rivas, J Javier Gutie ŕrez, J Carlos Palencia, and M Gonza ĺez Harbour. Optimized deadline assignment for tasks and messages in distributed real-time systems. In Proceedings of the 8th International Conference on Embedded Systems and Applications, ESA. Citeseer, Juan Rivas, J Gutierrez, J Palencia, and M Gonzalez Harbour. Deadline assignment in edf schedulers for real-time distributed systems. T. Qian, F. Mueller, and Y. Xin. A real-time distributed hash table. In Embedded and Real-Time Computing Systems and Applications (RTCSA), 2014 IEEE 20th International Conference on, pages IEEE,