LINUX AND REALTIME 1

Transcription

1 LINUX AND REALTIME 1

2 PRESENTATION Pierre Morel - MNIS Paris and Toulouse - France pmorel@mnis.fr Linux Port on new architectures, Realtime and Virtualization OCERA european project on Realtime components in cooperation with UPVLC, Santa-Anna and Prague UWEP mexican/franco project for the development of an ARM Hypervisor in cooperation with Siteldi LHA IBM research project for application virtualization in Linux 2

3 LINUX AND REALTIME Linux is not a Realtime Operating System "...and will never be" Linus Torwald Possible solutions Modification of Linux core Interrupt virtualization and co-kernel Complete Virtualization and co-domain 3

4 WHY IS LINUX NOT REALTIME 4

5 RECALL: LINUX 2.6 / 3.X Interrupt latency Context switch latency Scheduling Memory handling Spinlocks and preemption 5

6 INTERRUPT LATENCY This is the time between receiving an interrupt and starting associated action In realtime system the action is done by a realtime thread not the driver Let see what is happening in Linux kernel 6

7 INTERRUPT LATENCY 7

8 INTERRUPT LATENCY Tasklets and soft-interrupts may differ thread without any determinism including kernel threads Network implementation is particularly bad for realtime Even with the NAPI, it makes heavy use of softinterrupt and interrupt disabling 8

9 CONTEXT SWITCH LATENCY With Linux 2.6 we have better preemptions politics A good choice is essential to lower the latency and that is even for co-kernel adaptions as we will see later 9

10 NO PREEMPTION 10

11 VOLUNTARY PREEMPTION 11

12 PREEMPTIBLE KERNEL 12

13 SPINLOCKS Spinlocks are used to synchronize data access from concurrent processor in SMP/SMT. Due to kernel preemption the Linux implementation also disable preemption when a spinlock is locked This of course increase context switch latency The biggest trouble is caused by the Big Kernel Lock which only disappear after

14 LINUX SCHEDULER Linux scheduler is O1 This means equal time to choose a task whatever the load is Sadly, available scheduling policies are very primitive Rate Monotonic FIFO and RR (Round Robin) with 99 priority levels This is POSIX b compliant 14

15 O1 SCHEDULER 15

16 PRIORITY INVERSION "Handling of priority inversion may be solved by a good application programming" Victor Yodaiken (RTLinux) There are also possibilities to handle this in the kernel itself or in the libraries. What is it exactly? What really happened on mars? 16

17 PRIORITY INVERSION 17

18 PRIORITY INVERSION 18

19 PRIORITY INVERSION 19

20 PRIORITY INVERSION Two measures Priority ceiling priority inheritance In the Linux kernel, only rt-mutex(9) handle priority inheritance. They are only used in the i2c driver... In user space, GLIBC handle P.I. using priority ceiling: pthread_mutex_setprioceiling(3p) 20

21 MODIFYING LINUX 21

22 UNIVERSITY PROJECTS Linux-RT (Thomas Gleixner) Free community project is open and GPL ART-Linux Japanese project but is not open source 22

23 COMMERCIAL PROJECTS SYSLinux (Thomas Gleixner) is partially open source and working in the same direction as Linux-RT. Montavista Took also the same direction. 23

24 LINUX-RT Tasklet and interrupt routines are re-implemented as threads Spinlocks are modified to allow preemption A new EDF scheduler is actually developed 24

25 THREADED INTERRUPTS Interrupt routines are reimplemented as threads You may modify the priorities of the threads with tools like chrt(8) or sched_setscheduler(2) Some good examples for the optimization of audio threads: irqpriorities.html on linux-rt wiki 25

26 SPINLOCKS MODIFICATIONS Linux vanilla spinlocks disable preemption when holding the lock. This increase context switch latency from this duration Linux-RT replace this by having more dedicated spinlocks and disabling migration in for SMP BKL has been eliminated 26

27 EDF SCHEDULER The Earliest Deadline First scheduler makes RT thread programming easier It also allow a better CPU usage for periodic tasks The implementation is done by University Santa Anna - Pisa (ReTIS Lab) and the Evidence spin-off Not yet in the kernel because priority inversion is not handled (but encouraged by T. Gleixner) 27

28 ART-LINUX ART-Linux is used in japanese robotic projects Major change is done by changing standard interrupt handling implementation with interrupt poling. Priority inheritance on kernel mutexes has been implemented 28

29 MEMORY HANDLING Take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2) You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. 29

30 CO-KERNEL AND INTERRUPT VIRTUALIZATION 30

31 CO-KERNEL RT-Linux: Victor Yodaiken / Barbanov (FSMLabs) RTAI: Paolo Mantegazza (Aero-Polimiti Milano) Xenomai: Philippe Gerum 31

32 INTERRUPT VIRTUALIZATION 32

33 INTERRUPT VIRTUALIZATION 33

34 INTERRUPT VIRTUALIZATION 34

35 SECURITY AND MEMORY ACCESS In this implementation both co-kernel shared the same memory space Realtime applications also share the same memory space No protection against faults from Linux or from RTOS 35

36 ADEOS Adaptive Domain Environment for Operating Systems Karim Yagmour - Opersys who also done L.T.T. Virtualization of events - especially interruptions 36

37 ADEOS 37

38 ADEOS 38

39 ADEOS 39

40 ADEOS 40

41 ADEOS 41

42 XENOMAI/RTAI LXR 42

43 SECURITY AND MEMORY ACCESS At least in this implementation, realtime applications and standard Linux applications are protected against faults. But.. both kernel shared the same space 43

44 SCHEDULING By design, Xenomai scheduling must be full compatible with Linux scheduling Only SCHED_FIFO and SCHED_RR are supported 44

45 SCHEDULING LINUX 45

46 SCHEDULING XENOMAI 46

47 SCHEDULING LATENCY Scheduling latency is minimal with the usage of in kernel realtime tasks. For users space threads, the maximal scheduling latency is obtain when a Xenomai thread entered Linux scheduler until it reaches a preemption point. 47

48 MEMORY HANDLING In kernel memory allocation using standard Linux memory must on start. For example in _init() function of the RTLinux module You must use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. TLSF is implemented by Universidad Politecnica de Valencia 48

49 MEMORY HANDLING In user space take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2) You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. 49

50 USEFUL FEATURES Emulation of different realtime operating systems VxWorks, PSOS+, VRTX, UITRON, POSIX, RTAI Comedi, a librairy of Analogic Digital converters Analogy, a library of sensors drivers RTNET, real time ethernet using TDMA 50

51 CONCLUSION Pro responsiveness, low latency Integration of RTNET RTOS emulations Contra: No good scheduler, only Linux schedulers 51

52 MEASURE AND TESTS 52

53 REAL-TIME HYPERVISORS 53

54 HYPERVISORS 54

55 HYPERVISORS Two types type 1: bare metal (xen, XtratuM, L4, ESX...) type 2: OS based (qemu, vmware workstation) Three families micro-kernels (L4) complete virtualization (xen) para-virtualization (xen, ESX, XtratuM) 55

56 HYPERVISORS Main features Spatial isolation, MMU virtualization CPU and Interrupt virtualization Peripherals virtualization Inter domain Communication Applications Security, cryptographic applications isolation System mobility, HAC, HPC 56

57 MMU 57

58 MMU VIRTUALIZATION 58

59 CPU VIRTUALIZATION end of 60th: IBM S360 first virtual machine Popeck and Goldberg define a virtualizable CPU CPU has two modes: normal and privileged "sensitive control instructions and sensitive behaving instruction must be privileged instructions" Privileged instructions in normal mode are trapped Most CPU respect this more or less but Intel CPU 2005: with VTX intel gains CPU virtualization 59

60 PRE VIRTUALIZATION 60

61 WHAT IF THE CPU CAN NOT BE VIRTUALIZED Para virtualization The source code is modified Pre virtualization The binary of the program is modified Binary rewriting The binary of the process is modified 61

62 PARA VIRTUALIZATION 62

63 PRE VIRTUALIZATION 63

64 GENERAL PURPOSE HYPERVISORS Qemu Qemu is also an emulator supporting ARM, MIPS, SH4, PowerPC... KVM, Virtualbox and Xen use Qemu Vmware, HyperV IBM, SUN and HP have their own solutions used for High Availabilities and High Performance Computing 64

65 SPATIAL ISOLATION 65

66 REAL-TIME HYPERVISORS Temporal isolation a domain can not delay the execution of another domain Hypervisor scheduler adds determinism, EDF, ARINC May or may not allow more than one RTOS 66

67 TEMPORAL ISOLATION 67

68 REAL-TIME HYPERVISORS VLX (Virtual Logix / Red Bend) L4 is a german project from Dresden and Karlsruhe OKL4 and Sysgo hypervisor are based on L4 XtratuM is a spanish project from Valencia Other projects: WindRiver, Freescale, Xen, Trangon/Vmware... 68

69 XEN AS A REAL-TIME HYPERVISORS Corean Project - Chuck Yoo If all domains are scheduled by EDF schedulers and the Hypervisor uses EDF algorithm The tasks within each domain will be scheduled in deterministic manner. 69

70 DRIVER IMPLEMENTATION A domain has access to his own peripheral (VLX, XtratuM) A domain is dedicated as driver (L4, Xtratum) A domain is dedicated for all drivers (Xen) 70

71 DRIVER DOMAIN 71

72 COMPARING THE SOLUTIONS Next slides shows a comparison between the different solutions and where the different services are implemented: shared or private in user space, kernel space, domain or hypervisor 72

73 Linux Task Linux Task syscall interface Linux Scheduler IO Logic Tasklets IRQ routine Do_IRQ IRQ Vectors CPU Device PIC Device 73

74 Linux Task syscall interface RT Task Linux-RT IRQ thread Tasklets thread Scheduler IO Logic IRQ Vectors CPU Device PIC Device 74

75 Linux Task Scheduler syscall interface Tasklets RT Task RT-Linux IO Logic IRQ routine Scheduler IRQ routine IO Logic Do_IRQ IRQ logic IRQ Vectors CPU Device PIC Device 75

76 Linux Task syscall interface RT Task RTAI / Xenomai Scheduler Mem Mgt Scheduler Tasklets RT Task IO Logic IRQ routine IRQ routine Do_IRQ IRQ Vectors IRQ logic IO Logic ADEOS - ipipe IRQ logic IRQ Vectors Device CPU PIC Device 76

77 Hypervisor Linux Task RT Task syscall interface Scheduler Driver Scheduler Mem Mgt Mem Mgt Tasklets IO Logic IRQ routine IO Logic IRQ logic IO Logic IRQ routine IRQ logic hypercall interface Syscall Management Memory Management Scheduler IRQ logic Device MMU CPU PIC 77

78 Micro-kernel Driver RT Task Scheduler Scheduler Mem Mgt Tasklets Linux Task IO Logic Mem Mgt IRQ routine IO Logic IRQ logic IO Logic IRQ routine IRQ logic hypercall interface syscall interface Syscall Management Memory Mem. Management Scheduler IRQ logic Device MMU CPU PIC 78

79 THE END Thank you! Any questions? 79