Development of Real-Time Systems with Embedded Linux Brandon Shibley Senior Solutions Architect Toradex Inc.
Overview Toradex ARM-based System-on-Modules Pin-Compatible SoM Families In-house HW and SW Development 10+ Year Product Life Cycle Free Engineering Support Direct Sales Global Reach with Local Offices 2
Introduction to Real-Time Overview of Solutions Real-Time Linux Xenomai Heterogenous Multicore Conclusion Q & A
Real-Time Requirements Firmness of Real-Time Requirements Hard: Missed deadline results in system failure Soft: Missed deadline reduces system effectiveness Examples Periodic Deadline is periodic Often streaming/buffer related, must avoid buffer overflow/underflow Event Driven Event triggered, may have event-specific deadline Often spontaneous Control Both measurement & response deadlines Often periodic with feedback or other input 4
The Problem / Opportunity 5
Real-time Terms & Concepts Latency The system s response time from the moment of stimulus Jitter The variability of timing & latency Interrupt An event trigger which can interrupt the system Context Switch The switching of processor control from one thread/process to another Preemption The act of context switching a thread/process off of the processor to allow processing a higher priority task 6
Real-Time Operating System Concerns Determinism Predictable Scheduler Bounded latencies Minimal jitter Prioritization Priority-base scheduling of tasks & interrupt requests Preemption Preempt tasks for higher priority tasks Priority inheritance 7
Real-Time Operating System Concerns Priority Inversion Source: http://www.toulibre.org/pub/2010-11-20-ubuntuparty/slides/petazzoni-linux-temps-reel.pdf 8
Real-Time Evaluation Test Periodic Real-Time Task Toggle memory mapped GPIO every 200µs Stress processor, bus & interrupt controller Measure time between rising/falling edges Metrics: Jitter/Latency Jitter 400 µs 9
Colibri VF50 Colibri VF61 IT Colibri imx7s Colibri imx7d Colibri imx6s Colibri imx6dl A5 @400MHz A5 @500MHz M4 @167MHz A7 @800MHz M4 @200MHz A7 A7 @1GHz @1.0GHz M4 @200MHz A9 @1.0GHz GPU Vivante GC880 A7 A9 @1GHz @1.0GHz GPU Vivante GC880 RAM: 128MB RAM: 256MB RAM: 256MB RAM: 512MB RAM: 256MB RAM: 512MB FLASH: 128MB FLASH: 512MB FLASH: 512MB FLASH: 512MB emmc: 4GB emmc: 4GB US 42 @ 1 US 33 @ 1k US 62 @ 1 US 47 @ 1k US 60 @ 1 US 49 @ 1k US 65 @ 1 US 54 @ 1k US 75 @ 1 US 59 @ 1k 10 US 80 @ 1 US 67 @ 1k
Standard Linux Standard Linux Kernel Priority-based scheduling with preemption Recent kernels support threadable IRQs & IRQ priority setting Standard Preemption Levels: CONFIG_PREEMPT_NONE: No Forced Preemption (Server) CONFIG_PREEMPT_VOLUNTARY: Voluntary Kernel Preemption (Desktop) [DEFAULT] CONFIG_PREEMPT: Preemptible Kernel (Low-Latency Desktop) Test Configuration 3.14.52 kernel on Colibri imx6 colibri_imx6_defconfig CONFIG_PREEMPT_VOLUNTARY=y 11
Worst Case: -0.377 Quantity of samples Worst Case: 15.181 Standard Linux Samples Histogram 6% of samples 92% of samples 2% of samples Error (ms) 2µs bin size 12
Real-Time Linux Solutions Real-Time Linux Kernel Make the Linux kernel more Real-Time More Preemptable PREEMPT_RT Real-Time Co-Kernel Assign real-time tasks to higher priority co-kernel Xenomai Real-Time Co-Processor Assign real-time tasks to dedicated processor core NXP i.mx7 - Heterogenous Multicore Processing 13
PREEMPT_RT Linux Real-Time Linux Kernel PREEMPT_RT patch Minimizes the amount of kernel code that is nonpreemptable Sleeping spinlocks rt_mutex Threaded interrupt handler PREEMPT_RT_FULL: Fully Preemptible Kernel (RT) Real-time Linux Project at The Linux Foundation: https://wiki.linuxfoundation.org/realtime/start PREEMPT_RT patch source: https://www.kernel.org/pub/linux/kernel/projects/rt/ 14
PREEMPT_RT Linux Real-Time Linux Kernel Toradex Real-Time imx6 kernel recipe: http://git.toradex.com/cgit/meta-toradex-nxp.git/tree/recipes-kernel/linux/linuxtoradex-rt_3.14.52.bb Real-Time Linux article at Toradex Developer site: http://developer.toradex.com/knowledge-base/realtime-linux Codesys Software PLC http://developer.toradex.com/knowledgebase/codesys-software-plc-solution 15
PREEMPT_RT Linux Real-Time Linux Kernel Cyclictest Tests for Apalis imx6 with 3.14.28 kernel CONFIG_PREEMPT_VOLUNTARY CONFIG_PREEMPT CONFIG_PREEMPT_RT_FULL # cyclictest -n -p 80 -t -D30m policy: fifo: loadavg: 4.31 4.29 3.71 5/114 6286 T: 0 ( 1845) P:80 I:1000 C:1799993 Min: 10 Act: 19 Avg: 44 Max: 23350 T: 1 ( 1846) P:80 I:1500 C:1199995 Min: 10 Act: 35 Avg: 37 Max: 13460 T: 2 ( 1847) P:80 I:2000 C: 899996 Min: 10 Act: 35 Avg: 41 Max: 13631 T: 3 ( 1848) P:80 I:2500 C: 719997 Min: 11 Act: 38 Avg: 46 Max: 10856 # cyclictest -n -p 80 -t -D30m policy: fifo: loadavg: 4.98 5.07 4.37 5/115 11803 T: 0 ( 2133) P:80 I:1000 C:1799998 Min: 11 Act: 29 Avg: 35 Max: 821 T: 1 ( 2134) P:80 I:1500 C:1199997 Min: 11 Act: 35 Avg: 40 Max: 597 T: 2 ( 2135) P:80 I:2000 C: 899998 Min: 12 Act: 37 Avg: 44 Max: 886 T: 3 ( 2136) P:80 I:2500 C: 719998 Min: 12 Act: 60 Avg: 42 Max: 721 # cyclictest -n -p 80 -t -D30m policy: fifo: loadavg: 4.62 4.52 3.98 4/163 27123 T: 0 (12907) P:80 I:1000 C:1799985 Min: 10 Act: 24 Avg: 26 Max: 166 T: 1 (12908) P:80 I:1500 C:1199994 Min: 9 Act: 27 Avg: 27 Max: 187 T: 2 (12909) P:80 I:2000 C: 899996 Min: 11 Act: 21 Avg: 25 Max: 135 T: 3 (12910) P:80 I:2500 C: 719996 Min: 10 Act: 30 Avg: 28 Max: 129 16
Quantity of samples Worst Case: 0.040 Worst Case: 0.106 PREEMPT_RT Linux Samples Histogram 99.998% of samples 0.002% of samples Error (ms) 2µs bin size
Xenomai Xenomai Supplements Linux with a real-time co-kernel (Cobalt) Built into Linux kernel Linux runs as a low priority task within Linux Userspace real-time API Also a native Linux implementation of real-time framework (Mercury) May utilize PREEMPT_RT Toradex doesn t currently provide a Xenomai build for download 19
Xenomai Xenomai Source: http://elinux.org/images/7/76/kiszka.pdf. Siemens AG 2015, et al. Creative Commons BY-SA 3.0 license 20
Xenomai Xenomai API http://xenomai.org/start-here/ http://xenomai.org/api-reference/ 21
22
Quantity of samples Worst Case: 0.0175 Worst Case: 0.0165 Xenomai Samples Histogram 100% of samples Error (ms) 2µs bin size
i.mx7 HMP NXP i.mx7 - Heterogenous Multicore Processing i.mx 7 implements a Heterogeneous Asymmetric Multi- Processing Architecture Two independent CPU cores on a single chip: 2x ARM Cortex-A7 @ 1GHz 1x ARM Cortex-M4 @ 200MHz Shared Bus Topology Linux on Cortex-A7 FreeRTOS on Cortex-M4 24
i.mx7 HMP NXP i.mx7 HMP Architecture 25
i.mx7 HMP NXP i.mx7 Software Linux Kernel Linux 4.1.15 Rpmsg VirtIO-based TTY driver FreeRTOS FreeRTOS v8.0.0 kernel Various build systems supported (DS-5, CMake + gcc, IAR) Modified version of OpenAMP Drivers: ADC, GPIO, I2C, UART, WDOG, ECSPI, FlexCAN Some Mainline Support (starting with 4.4) Toradex Developer Website article: http://developer.toradex.com/knowledge-base/freertos-on-the-cortex-m4-of-a-colibri-imx7 26
27
Worst Case: 0.0005 Quantity of samples i.mx7 HMP Samples Histogram 100% of samples (< 0.25% of period) 100% of samples Error (ms) 2µs bin size 28 12/15/2016
i.mx7 HMP TAQ Balancing Robot 29
Conclusion Multiple Linux-based solutions with a range of real-time capability Standard Linux Real-Time Linux Xenomai Heterogenous Multicore All solutions have optimization potential All solutions require careful design beyond the OS selection Requirements, Hardware, Drivers, Application 30