Real-Time Internet of Things

Transcription

1 Real-Time Internet of Things Chenyang Lu Cyber-Physical Systems Laboratory h7p://

2 Internet of Things Ø Convergence of q Miniaturized devices: integrate processor, sensors and radios. q Low-power wireless: connect millions of devices to the Internet. q Data analytics: make sense of sensor data. q Cloud: scalable computing. Ø Large-scale IoT-driven control q Smart manufacturing, transportation, power grid, healthcare q Real killer apps of IoT! q Closed-loop control requires real-time performance!

3 Clinical Warning Rapid Response R. Dor, G. Hackmann, Z. Yang, C. Lu, Y. Chen, M. Kollef and T.C. Bailey, Experiences with an End-To-End Wireless Clinical Monitoring System, Conference on Wireless Health (WH'12), October

4 Industrial IoT Closed-loop industrial automation à latency bounds Ø Frontend: wireless sensor-actuator networks Ø Backend: edge and central clouds Offshore Onshore WirelessHART in Process Industries [Courtesy: Emerson Process Management] 4

5 Real-Time IoT à End-to-End Real-Time Performance Ø Miniaturized devices à real-time embedded systems Ø Low-power wireless à real-time wireless Ø Data analytics à real-time analytics Ø Cloud à real-time data processing 5

6 Real-Time IoT à End-to-End Real-Time Performance Ø Miniaturized devices à real-time embedded systems Ø Low-power wireless à real-time wireless Ø Data analytics à real-time analytics Ø Cloud à real-time data processing 6

7 Industrial IoT Closed-loop industrial automation à latency bounds Ø Frontend: wireless sensor-actuator networks Ø Backend: edge and central clouds Offshore Onshore WirelessHART in Process Industries [Courtesy: Emerson Process Management] 7

8 WirelessHART Ø Reliability and predictability q Multi-channel TDMA MAC q One transmission per channel q Redundant routes q Over IEEE PHY Ø Centralized network manager q Collect topology information q Generate routes and schedule q Change when devices/links break Industrial wireless standard for process automation 8

9 The Control Challenge Most of today s industrial wireless networks are for monitoring. Sensor Actuator sensor data control command Controller Dependable control requires real-time resilience to loss control performance 9

10 The Real-Time Problem Ø A feedback control loop incurs a flow F i q Route: sensor à à controller à à actuator q Generate packet every period P i q Multiple control loops share a network Ø Each flow must meet deadline D i ( P i ) q Stability and predictable control performance Ø Research problems q Real-time transmission scheduling à meet end-to-end deadlines q Fast schedulability analysis à adapt to wireless dynamics through admission control and rate adaptation 10

11 Delays in WirelessHART A transmission is delayed by Ø channel contention: all channels are assigned to other transmissions Ø transmission conflict over a same node q contributes significantly to latency! and 5 conflict 4 and 5 conflict 3 and 4 do not 11

12 Fast Delay Analysis Ø Compute upper bound of the delay for each flow q Sufficient condition for real-time guarantees Ø Channel contention à multiprocessor task scheduling q q q A channel à a processor Flow F i à a task with period P i, deadline D i, execution time C i Leverage existing response time analysis for multiprocessors Ø Account for delays due to transmission conflicts A. Saifullah, Y. Xu, C. Lu and Y. Chen, End-to-End Communication Delay Analysis in Industrial Wireless Networks, IEEE Transactions on Computers, 64(5): , May

13 Delay due to Conflict Ø Low-priority flow F l and highpriority flow F h, conflict à delay F l )*$%+*, F l delayed by 2 slots Ø Q(I,h): #transmissions of F h sharing nodes with F l q In the worst case, F h can delay F l by Q(l,h) slots Ø Conflicts contributes significantly to delays q Must be considered in algorithms and analysis! F l delayed by 2 slots F l delayed by 1 slot!"#$%&'"(!" # &!"#$%&'"(!" $ 13

14 Real-Time Wireless Networking Ø WirelessHART stack in TinyOS [EWSN 2015] q Implementation on a 69-node testbed q Network manager (scheduler + routing) Ø Fast delay analyses q Fixed priority [RTAS 2011, TC 2015, RTSS 2015] q Earliest Deadline First [IWQoS 2014] Ø Conflict-aware real-time scheduling q Fixed priority [ECRTS 2011, RTSS 2015] q Dynamic priority [RTSS 2010, IWQoS 2014] Ø Energy-efficient routing [IoTDI 2016] Ø Emergency communication [ICCPS 2015] Ø Channel selection [INFOCOM 2017] C. Lu, A. Saifullah, B. Li, M. Sha, H. Gonzalez, D. Gunatilaka, C. Wu, L. Nie and Y. Chen, Real-Time Wireless Sensor- Actuator Networks for Industrial Cyber-Physical Systems, Proceedings of the IEEE, 104(5): , May

15 Real-Time IoT à End-to-End Real-Time Performance Ø Miniaturized hardware à real-time embedded systems Ø Low-power wireless à real-time wireless Ø Data analytics à real-time analytics Ø Cloud à real-time data processing 15

16 Cloud is real-dme today Ø Virtualization platforms provide no guarantee on latency q Xen: credit scheduler, [credit, cap] q VMware ESXi: [reservation, share, limitation] q Microsoft Hyper-V: [reserve, weight, limit] Ø Clouds lack service level agreement on latency q Amazon, Google, Microsoft cloud services: #VCPUs Current clouds provision resources, not latency! 16

17 Towards Real-Time Cloud Ø Support real-time applications in the cloud. q Latency guarantees for tasks running in virtual machines (VMs). q Real-time performance isolation between VMs. q Resource sharing between real-time and non-real-time VMs. Ø Multi-level real-time performance provisioning. q RT-Xen à real-time VM scheduling on a virtualized host. q VATC à real-time network I/O on a virtualized host. q RT-OpenStack à real-time cloud resource management. RT-OpenStack VATC: Real-Time Communication 17

18 Xen Ø Xen: type-1, baremetal hypervisor q Domain-0: drivers, tool stack to control VMs. q Guest Domain: para-virtualized or fully virtualized OS. Ø Scheduling hierarchy q Xen schedules VCPUs on PCPUs. q Guest OS schedules threads on VCPUs. q Xen credit scheduler: round-robin with proportional share. Real-Time Task VCPU OS Sched OS Sched OS Sched Xen Scheduler PCPUs 18

19 RT-Xen Ø Real-time schedulers in the Xen hypervisor. Ø Provide real-time guarantees to tasks in VMs. Ø Incorporated in Xen 4.5 as the real-time scheduler. RT-Xen h7ps://sites.google.com/site/real^mexen/ S. Xi, M. Xu, C. Lu, L. Phan, C. Gill, O. Sokolsky and I. Lee, Real-Time Multi-Core Virtual Machine Scheduling in Xen, ACM International Conference on Embedded Software (EMSOFT'14), October

20 ComposiDonal Scheduling Ø Analytical real-time guarantees to tasks running in VMs. Ø VM resource interfaces q A set of VCPUs each with resource demand <period, budget > q Hides task-specific information q Computed based on compositional scheduling analysis Hypervisor Resource Interface Scheduler Resource Interface Resource Interface Scheduler Scheduler Workload Workload Virtual Machines 20

21 Real-Time Scheduler Design Ø Global Scheduling q Allow VCPU migration across cores q Work conserving utilize any available cores q Migration overhead and cache penalty Ø Partitioned Scheduling q Assign and bind VCPUs to cores q Cores may idle when others have work pending q No migration overhead or cache penalty Ø Enforce resource interface through budget management q Periodic Server vs. Deferrable Server Ø Priority: Earliest Deadline First vs. Deadline Monotonic 21

22 Real-Time Deferrable Server (RTDS) in Xen Ø Global Scheduling q Allow VCPU migration across cores q Work conserving utilize any available cores q Migration overhead and cache penalty Ø Partitioned Scheduling q Assign and bind VCPUs to cores q Cores may idle when others have work pending q No migration overhead or cache penalty Ø Enforce resource interface through budget management q Periodic Server vs. Deferrable Server Ø Priority: Earliest Deadline First vs. Deadline Monotonic 22

23 RT-Xen vs. Xen Xen (credit scheduler) misses deadlines at 22% of CPU capacity. RT-Xen delivers real-time performance at 78% of CPU capacity. 23

24 Virtualized Network I/O Ø Xen handles all network traffic through Dom0 Ø Real-time and non-real-time traffic share Dom0 q CPU and network contention Ø Long delays for real-time traffic in virtualized hosts Dom0 Dom1 Dom2 Network Components Real-Time App Non Real-Time App Xen Hypervisor NIC Memory CPU Storage 24

25 Network I/O in Virtualized Hosts Ø Linux Queueing Discipline q Rate-limit and shape flows q Prioritization or fair packet scheduling Dom1 Real- Time App Dom2 Non- Real- Time App Ø Priority inversion in virtualization components q between transmissions q between transmission and reception VirtualizaDon Components Dom0 Ø VATC: Virtualization-Aware Traffic Control q Process packets in prioritized kernel threads q Dedicated packet queues per priority C. Li, S. Xi, C. Lu, C. Gill and R. Guerin, Prioritizing Soft Real-Time Network Traffic in Virtualized Hosts Based on Xen, RTAS Queueing Discipline NIC 25

26 Real-Time Traffic Latency VATC reduces priority inversion à lower latency for real-time traffic. Round trip Latency ( ms) Prio, Dom FQ_CoDel, Dom VATC Dyn Cons Dyn Interrupt Interval (µs) Median round-trip latency of real-time traffic. CPU contention from two small-packet interfering streams. 26

27 Virtualized Host à Cloud Ø Provide real-time performance to real-time VMs Ø Achieve high resource utilization 27

28 OpenStack LimitaDons Ø Popular open-source cloud management system Ø VM resource interface q Number of VCPUs q Not real-time Manager Host Host Host Ø VM-to-host mapping q Filtering (admission control) VCPU-to-PCPU ratio (16:1), max VMs per host (50) Coarse-grained admission control for CPU resources q Ranking (VM allocation) Balance memory usage No consideration of CPU resources VM VM VM VM VM 28

29 RT-OpenStack Ø Co-hosting real-time VMs with non-real-time VMs Ø Deliver real-time performance q Support RT-Xen resource interface q Real-time-aware VM-to-host mapping Ø Achieve high resource utilization q Co-locate non-real-time VMs with real-time VMs q Non-real-time VMs consume remaining resources without affecting the real-time performance of real-time VMs S. Xi, C. Li, C. Lu, C. Gill, M. Xu, L. Phan, I. Lee, O. Sokolsky, RT-OpenStack: CPU Resource Management for Real- Time Cloud Computing, IEEE International Conference on Cloud Computing (CLOUD'15), June

30 RT-OpenStack: VM-to-Host Mapping Ø Admission control: RT-Filter q Accept real-time VMs based on schedulability and memory q Consider only accepted real-time VMs Ø VM allocation: RT-Weigher q Balance CPU utilization q Consider only accepted real-time VMs Resource Interface Admission Control VM AllocaDon Real-Time VMs {<period, budget>} Schedulability + Memory CPU U^liza^on Non-Real-Time VMs Best Effort Memory Memory 30

31 OpenStack Ø Overload four hosts with real-time VMs à deadline misses. Ø Two hosts running non-real-time VMs only. Ø Unbalanced distribution of real-time domains. Hadoop finish ^me: 314 seconds 47% 32% 73% 36% 29% 30% 75% 13% 31% 61% 37% 31

32 RT-OpenStack Ø Schedulability guarantees for real-time VMs à no deadline miss. Ø Distribute real-time VMs across hosts. Ø Hadoop makes progress using remaining CPU resources. Hadoop finish ^me: 435 seconds 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 32

33 Real-Dme Cloud Stack Ø RT-Xen: real-time VM scheduling in virtualized hosts. Ø VATC: real-time network I/O in virtualized hosts. Ø RT-OpenStack: real-time cloud resource management. RT-OpenStack VATC: Real-Time Communication Latency guarantees 33

34 IoT: New Horizon for Real-Time! Ø Real killer apps : large-scale IoT-driven control q Smart manufacturing, transportation, grid Ø From best effort to real-time IoT q Devices, wireless, cloud, analytics Ø Grand challenges and opportunities for real-time! q End-to-end latency across multi-tier architecture q Scalability with regard to devices and computational load q Dependable control over IoT 34