Windows NT Real-Time Extensions better or worse?

Transcription

1 By Martin Timmerman, Chief-Editor of Real-Time Magazine, Bart Van Beneden Project Manager, Laurent Uhres, Software Engineer, Real-Time Consult. RTOS EVALUATION Windows NT Real-Time Extensions better or worse? This article presents an overview of the capabilities of the real-time extensions for Windows NT 4.0 as a real-time operating system (RTOS). The extensions that are covered are RTX 4.1 from VenturCom, INTime 1.20 from Radisys and Hyperkernel 4.3 from Imagination Systems. This paper is part of an evaluation project concerning real-time extensions for Windows NT 4.0. INTRODUCTION W indows NT 4.0 (NT) has been primarily designed as a general purpose operating system (GPOS). Still, there is a clear tendency in the industry to use Microsoft Windows OS technology in the dedicated systems market field, to get openness at every level. There are several reasons for this: The Win32 application programmer interface (API) is becoming a de facto standard for programmers. The graphic user interface is (GUI) popular, other operating systems tend to have a similar interface. A huge amount of third-party device drivers are available. Many integrated development interfaces (IDE) are available. These IDEs are very powerful. WINDOWS NT 4.0 AS A RTOS General requirements A RTOS can be defined as an operating system that responds in a predictable way to unpredictable external events. A detailed description of what makes a good RTOS is available on our web site (.be) ( To qualify as an RTOS, an operating system needs to have a minimum set of requirements. These requirements are necessary but nut sufficient: The operating system needs to be multithreaded and pre-emptable. The notion of thread priority has to exist. The operating system has to support predictable thread synchronization mechanisms. A mechanism for priority inheritance must be available. The OS behavior should be known and predictable (interrupt latencies, task switch latencies, driver latencies, etc). This means that there should be a maximum response time under all system load scenarios. NT obviously meets the first requirement. The second criteria is also met, but only a few priority levels are usable for real-time. The number of priority levels is too low to allow for a well designed real-time application (it is impossible to use methods like rate-monotonic scheduling with only a few thread priorities available). NT also doesn't have a mechanism for priority inheritance. Interrupt handling To minimize the time spent in the interrupt service routine (ISR), NT introduces the concept of deferred procedure calls (DPC). The priority of these DPCs is higher than the priority of user and system threads, but all DPCs run at the same priority. This makes that all DPCs are queued in FIFO order, and that the DPC of a high priority interrupt will only be executed when all the DPCs that were queued before it are finished executing. This leads to variable and unpredictable response times, which is incompatible with the fifth requirement. Memory management In NT, memory management is based on virtual memory mechanism. This implies memory space protection, address translation and swapping. Swapping is unacceptable for real-time applications. Memory pages can be locked in memory. However, Jeffrey Richter claims in his book [Richter95] that NT can unlock the pages of a process and swap them from memory to disk if it is inactive (we were not able to reproduce this). Conclusion Windows NT is not suitable for real-time applications. WINDOWS NT 4.0 REAL-TIME EXTENSIONS General solutions In the previous paragraph, we discussed briefly why NT is not appropriate for the use in real-time applications. There are four types of solutions to keep the advantages offered by NT and add real-time capabilities to the system. First of all, NT could be used as is. Although this may sound contradictory to the conclusion of the previous paragraph, several workarounds allow people to build a real-time system on NT. One way is to enhance the interrupt handling mechanism by intercepting interrupts. One vendor, LP-Elektronik, uses this method and intercepts the non-maskable interrupt (NMI). We do not encourage this procedure since it is not without risks. All the interrupts are disable during the NMI which means that all device drivers are delayed and no system call can be executed in the interrupt handler of the NMI. 11

2 MECHANISM RICHNESS INTIME 1.20 RTX 4.1 HYPERKERNEL 4.3 Task or thread management 41% 35% 41% Clock 71% 14% 29% Interval Timer 33% 83% 50% Fixed block size partition 0% 0% 0% Non-fixed block size partition 38% 54% 17% Interrupt Handling 63% 38% 38% Counting Semaphore 80% 80% 0% Binary Semaphore 0% 0% 0% Mutex 50% 75% 33% Conditional Variable 0% 0% 0% Event flags 0% 38% 0% Signals 0% 0% 0% Queue 44% 0% 19% Mailbox 0% 0% 0% AVERAGE PERCENTAGE: 30% 30% 16% Table 1 : API richness The second solution would be to implement a Win32 API library on a commercial RTOS. The drawbacks of this solutions is that standard NT applications cannot be executed anymore (unless a complete virtual NT machine is implemented) and the huge set of device drivers can't be used. Future expandability will also be difficult. The third idea is to make coexist NT and a RTOS on one processor. This is a solution pursued by many vendors, including those of the products investigated in this article. The final idea is to extend the previous to a multiprocessor environment. Here each operating system can run on a processor. A more detailed study on the different solutions can be found in the article [1] and [2]. Solutions discussed in this article The real-time extensions discussed in this article are based on the third category of solutions, i.e. NT and a RTOS (or at least part of it) coexist on one processor. There are two ways to accomplish this: To modify the Hardware Abstraction Layer (HAL) by intercepting interrupts and including a small scheduler or RTOS. To run NT as one of the tasks on top of a (supervisor) RTOS. The HAL has to be modified or at least intercepted. These HAL modifications such as manipulating the clock or rewriting the interrupt processing mechanism represent an unprecedented and unproved use of the HAL. Interception of the HAL is also possible via Intel processor tricks. These implementations are therefore only available on Intel based machines. RTX 4.1 from VenturCom, Inc. RTX 4.1 is very close to NT. The architecture is based on a nucleus integrated in the NT kernel. This way, every RTX process runs as an NT kernel device driver, which makes that the processes are not protected from each other. This may result in faster context switches, but in terms of security, great care is needed. To debug kernel mode applications, (primitive) device driver debugging tools are needed. INTime 1.20 from Radisys. INTime 1.20 is loosely coupled to NT. The architecture is based on the hardware task mechanism of the Intel processor. It is exactly as two kernels running on the same hardware. Because they share the hardware, some modifications had to be made to the NT HAL. The INTime 1.20 application processes run in user mode, which makes that they receive the same protection provided by the Intel x86 architecture as the NT processes do. Classical thread debugging tools can be used. Hyperkernel 4.3 from Imagination Systems, Inc. Hyperkernel 4.3 is implemented as an extra subsystem. It is conceptually similar to other NT subsystems in that it is a self-contained execution environment. 12

3 However, the Hyperkernel subsystem has its own scheduler, its own set of services and its own kernel. Also here, the applications run in kernel mode, which means that debugging will be "kernel style" debugging. Hyperkernel 4.3 makes an attempt to avoid "NT starvation". NT starvation would occur if the real-time application would consume all available CPU power and keep NT from running. This would make the system crash. Therefore, a time slice for NT and Hyperkernel 4.3 was introduced. This means that NT and Hyperkernel run alternate. The length of this time slice is configurable and is between 25µs and 250µs. INSTALLATION AND CONFIGURA- TION Installation and configuration of RTX 4.1, INTime 1.20 and Hyperkernel 4.3 is pretty straightforward. Installation is based on the installation wizard and uses the standard windows graphical user interface. Since the three products are all extensions to NT, rather than a standalone operating system, surprises are less likely to happen during installation than when installing a complete operating system from the ground up. RTOS RICHNESS Application Programmer Interface (API) It is a known fact that the more system calls an operating system provides, the shorter the development time for a particular application will be. To rate the RTOS API richness, we have created a list of features for the most common system calls and compare it with the system calls that are available in INTime 1.20, RTX 4.1 and Hyperkernel 4.3. Table 1 presents an overview of the results of the RTOS API richness list. It gives the percentage of systems calls available in Hyperkernel 4.3 for the different features listed in the table. As Table 1 shows, the APIs for all 3 products are pretty poor. Some basic features are missing, e.g. none of the APIs support event flags and the ability to synchronize threads on multiple events. The Hyperkernel 4.3 API seems to be the most limited of all. However, this table should not be misunderstood. Every product's API has system calls that are not in our list, and are therefore not represented in the table above. Standard device drivers It is important that a RTOS supports a wide variety of device drivers, especially in this day and age where connectivity is in the center of the attention. Since these products are extensions to Windows, all the device drivers running on a NT machine will run on INTime 1.20, RTX 4.1 and Hyperkernel 4.3 machines. However, these NT device drivers do not always offer real-time deterministic behavior. If predictable responses for a particular device are required a driver specifically for the real-time extension has to be developed. PERFORMANCE EVALUATION Test platform All the tests described in the following sections were executed using the following hardware. Motherboard: Chaintech 5TTDM with a 33MHz PCI bus BIOS: Award BIOS v4.51pg CPU: Pentium 200Mhz MMX RAM: 32 Mb DIMM Hard drive: Western Digital Caviar 34300, primary master, UDMA 2 Graphic adapter: S3 Trio 64V2/DX on PCI slot 1 PCI bus analyzer All tests were performed with cache enabled. Measuring method The main problem is the use of a time reference for the measurements. Software timers often do not have enough precision. Their use adds large and unpredictable overhead to the measurements. Moreover, the measurements are synchronous with the system clock. Instead of a software timer, we use external hardware to carry out the measurements. During the execution of a test, tracing data (double word) is written at a valid PCI bus address before and after the evaluated event. Writing the trace to a double word aligned address on a 33MHz PCI bus consumes six to seven bus cycles, i.e. 180ns to 210ns. The PCI bus analyzer stores the data written on the PCI bus into local memory and adds a timestamp to it. When the test is completed, the data is downloaded from the PCI bus analyzer to a PC where it is further processed. Task switch latency A real-time application consists of several individual tasks running concurrently. The time it takes for the system to switch from one task to another is called the task switch latency. It is important that this latency is as small as possible to keep the task-switching overhead in the application to a minimum. Following sections present the results of tests with 10 different threads all running at the same priority and voluntarily yielding control to each other. All of this happens when the NT side is not loaded. INTime 1.20 Figure 1: Task switch latency- INTime

4 RTX 4.1 Figure 2: Task switch latency- RTX 4.1 Hyperkernel 4.3 Figure 3 : task switch latency- Hyperkernel 4.3 (time slice mechanism not bypassed) The spikes in the graph are the result of the time slicing mechanism for NT and Hyperkernel 4.3 as described above. For this test, the time slice was configured to be 250 µs. This is reflected in the results: roughly every 250 µs, the same amount is added to the average task switch latency. The spikes indicate where the Hyperkernel 4.3 time slice ends and that of NT begins. The NT time slice is fully pre-emptible by Hyperkernel 4.3 interrupts. But when there are no interrupts, the Hyperkernel 4.3 application shares CPU time with NT if it needs it. As a result, the designer of an Hyperkernel 4.3 application has to take into account that at all times the execution of his application can be suspended for the length of the time slice as it is configured (25 µs to 250 µs). This will make the execution of the application almost unpredictable. This time slice feature seems to be enabled permanently, but can be bypassed using the system calls "hkseizeprocessor" and "hkreleaseprocessor". The documentation is vague on the semantics of this system call, but it seems that in between these 2 calls, the Hyperkernel 4.3 application will not be pre-empted and NT will not run. Executing the same tests by bypassing the time slice mechanism leads to the more conventional results in figure 4. Interrupt handling Interrupts are a way to deal with external events and are therefore very important in real-time systems. For Figure 4: Task switch latency- Hyperkernel 4.3 (no time slicing) real-time systems the interrupt handling mechanism needs to be fast and predictable. Two latencies are used to rate the performance of the interrupt mechanism: Interrupt Latency: the time elapsed between the execution of the last instruction of the interrupted task and the first instruction in the interrupt handler. This is an indication of how fast the system reacts to an external event. Interrupt Dispatch Latency: the time interval to go from the last instruction in the interrupt handler to the next task scheduled to run. This indicates the time needed to go from interrupt level to task level. This section displays the results of a test that measured the interrupt latency and dispatch latency. The test was constructed in such way that the interrupted task is an NT task. This makes that since the interrupt handler is executed in the real-time extension, the system has to switch from NT to the extension in order to execute the interrupt handler, and then switch back from extension to NT to continue executing the task that was interrupted. An interrupt was generated roughly every 50 µs. Our measuring method did not permit us to take more than a few hundred interrupt (dispatch) latency samples, as opposed to several thousand samples we were able to obtain in the other tests. Therefore, the connecting lines in the graphs were omitted and only the dots were plotted. INTime 1.20 We observed an average interrupt latency of about 11 Figure 5 : Interrupt latency- INTime 1.20 (task to ISR) 15

5 Figure 6: Interrupt dispatch latency- INTime 1.20 (ISR to task) Figure 9. Interrupt latency - Hyperkernel 4.3 (task to ISR) µs, which is still acceptable. However, under these test conditions, the average interrupt dispatch latency was about 25 µs. Adding those 2 together results in a task to task latency of 36 µs. This is longer than in the other extensions we have tested. RTX 4.1 Both graphs are showing isolated measurements where the latency is a substantially bigger than the average, but this is probably due to higher priority interrupts occurring at the same time or higher priority NT tasks pre-empting our NT task. Figure 10. Interrupt dispatch latency - Hyperkernel 4.3 (ISR to task) Synchronization Synchronization objects are used to synchronize the execution of tasks. Common synchronization objects are semaphores, critical sections and signals. Following sections present the results of our extensions on a test that measured the time it takes to acquire and release a semaphore when no other tasks were pending on that semaphore (no contention). Figure 7 : Interrupt latency- RTX 4.1 (task to ISR) INTime 1.20 Figure 8 : Interrupt dispatch latency- RTX 4.1 (ISR to task) Figure 11. Acquisition time semaphore - INTime 1.20) Hyperkernel 4.3 The test results show that the interrupt mechanism is fast. It is obvious that the NT time slice is fully preemptible by a Hyperkernel 4.3 interrupt. 16

6 Figure 12. Release time semaphore - INTime 1.20) RTX 4.1 Figure 16. Release time semaphore - Hyperkernel 4.3 (no time slicing) Figure 17 displays the results of the test when the time slice mechanism is not bypassed. Again, the impact of the time slice mechanism becomes apparent. Under these circumstances, it has to be assumed that any action in Hyperkernel 4.3 can take longer than the length of the time slice (here configured as 250 µs), since at any time the Hyperkernel 4.3 application can be interrupted to let less time critical NT tasks run. Figure 13. Acquisition time semaphore - RTX 4.1 Figure 14. Release time semaphore - RTX 4.1 Hyperkernel 4.3 The results in figures 15 and 16 were obtained by bypassing the NT / Hyperkernel 4.3 time slice mechanism. Figure 15. Acquisition time semaphore - Hyperkernel 4.3 (no time slicing) Figure 17. Acquisition time semaphore - Hyperkernel 4.3 (time slice mechanism not bypassed) TOOLS Editors All the editors available in Windows NT 4.0 are available to the extensions. Debuggers Of the 3 real-time extensions discussed in this article, INTime 1.20 is the only one that provides its own debugger. The debugger is called "Soft Scope", which can communicate with the INTime 1.20 kernel. It can be used in 2 ways: the user can start the application and connect to Soft Scope afterwards, or the real-time application can be launched from within Soft Scope. The non real-time processes that are part of the application can be simultaneously debugged with the standard Windows development tools. It is no surprise that INTime 1.20 is the only product that provides its own debugger. In both Hyperkernel 4.3 and RTX 4.1, the applications run in kernel mode. Debugging such applications is much like debugging device drivers. No conventional thread level debugging tools can be used. Compilers Applications for INTime 1.20, RTX 4.1 and Hyperkernel 18

7 4.3 can be developed using the Microsoft Visual C++ compiler, an industry standard. All that needs to be done is to change some settings so that the real-time extension libraries are linked with the application. Other tools Hyperkernel 4.3 comes with an application called "Run Time Manager". Run time manager is a completely integrated environment that can be used to: Start and stop application. Monitor output messages generated by Hyperkernel applications. Monitor the status of all the threads in the Hyperkernel application. Monitor and modify the current contents of the Hyperkernel shared memory. Some of these features are available in INTime 1.20 and RTX 4.1 too, but no completely integrated environment is provided where all these features are handy and easy to use. USER FRIENDLINESS Documentation The documentation of RTX 4.1 is complete, clear and well presented. The INTime 1.20 documentation is pretty complete too, but is fragmented and less efficient organized. The Hyperkernel 4.3 documentation overall is too limited. Several architectural issues are not clear and the API documentation does not provide a detailed enough description of the system calls and it's parameters, which can lead to misunderstandings and wrong designs. GUI For all 3 products, the GUI is the standard Microsoft Windows graphical user interface. CONCLUSION In this article, real-time extensions for Windows NT 4.0 from 3 different vendors were discussed. It seems that during the tests performed for this article, all 3 real-time extensions exhibited predictable behavior. The results show typical interrupt latencies in the range of µs and typical task switch latencies in the range of 5-20 µs (on a 200 MHz Intel Pentium processor). Hyperkernel 4.3 and RTX 4.1 seem to have the best score on these 2 tests. However, only a limited amount of tests were performed for this article and it remains to be seen if the extensions will behave predictable under all circumstances. More extensive tests will be performed and published in the evaluation reports available later this year. The real-time extensions were clearly developed with a different design philosophy in mind. On one hand there are Hyperkernel 4.3 and RTX 4.1, where the applications run in kernel mode, and on the other hand there is INTime 1.20 where the application runs in user mode. Having the application code execute in kernel mode may result in faster task switching and interrupt latencies, but this comes with the price of security and stability. Bugs in applications running in kernel mode may have disastrous consequences. Hyperkernel 4.3 has introduced a time slicing mechanism between NT and Hyperkernel. The NT time slice is pre-empted when a Hyperkernel interrupt occurs, but during normal execution the Hyperkernel 4.3 application shares CPU time with NT. This prevents NT from crashing easily, but it does add somewhat unpredictable behavior to the execution of the application. However, the user can use API system calls to create a sort of "critical section" which will be executed without NT intervention. Without knowing your application, we cannot advice on which product would be best to use. This service can be provided upon request. The final evaluation reports are available in November. REFERENCES [1] M. Timmerman & J.C. Monfret, Windows NT as a real-time OS. Real-Time Magazine, issue [2] M. Timmerman & J.C. Monfret, Windows NT realtime extensions: an overview. Real-Time Magazine, issue Dr. Ir. Martin Timmerman graduated in Telecommunications Engineering from the Royal Military Academy (RMA) Brussels and received his Doctorate in Applied Science from the Gent State University (1982). He became the director of the System Development Center (SDC) at RMA, which he created in 1983, and converted himself to a Computer Science Engineer. Presently, he gives general courses on Computer Platforms and more specific courses on System Development Methodologies at the RMA. Outside the RMA, Martin is known for his audits, reviews, seminars, evaluation reports and feasibility studies he performs with Real- Time Consult. Real-Time Consult is also known for Real-Time Magazine and the Real-Time Encyclopaedia website ( His second company, Real-Time User Support International (RTUSI) provides hardware and software support services and is involved in project engineering for Real-Time Systems. Bart Van Beneden has been with Real-Time Consult since 1997 where he is involved in the RTOS evaluation program of Real-Time Magazine as a project manager. He received his degree in computer science at the Free University of Brussels. Before joining Real-Time Consult, he designed multi-media applications with LaserMedia Inc. Laurent Uhres graduated in 1997 as an analyst-programmer from the Haute Ecole Rennequin Sualem, Belgium. He has an additional diploma in Object Oriented Development from the Instito Politécnico do Porto, Portugal. Laurent joined Real-Time Consult mid 1997 to work as a software engineer where he has specialized himself in evaluating RTOSs. * Although all care has been taken to obtain correct information and accurate test results, Real-Time Consult and Real-Time Magazine cannot be liable for any incidental or consequential damages (including damages for loss of business, profits or the like) arising out of the use of the information provided in this report, even if Real-Time Consult and Real-Time Magazine have been advised of the possibility of such damages.. 19