Comparing the Real-time Performance of Windows NT to an NT Real-time Extension

Transcription

1 Comparing the Real-time Performance of Windows NT to an NT Real-time Extension Kevin M. Obenland, Tiffany Frazier, Jin S. Kim, John Kowalik The MITRE Corporation, 1820 Dolley Madison Blvd. McLean, VA Contact: Abstract Because of the dominance of Microsoft Windows in the PC market there is a strong interest in using Windows NT as a platform for real-time process and control systems. This type of solution is very cost effective because applications and development tools are widely available. However, Windows NT was designed as a general purpose operating system and optimizes average not worst case performance. In this paper we investigate two methods for bring real-time process and control systems to NT based platforms. We first evaluate NT as-is, using a series of realtime benchmarks, and show that NT use in real-time systems is limited to soft real-time systems where there is low system load. The second approach for developing NT based real-time systems is to add a real-time extension to NT. We evaluate one such product, INtime from RadiSys and conclude that, even under a heavy system load, hard real-time determinism is possible. 1 Introduction The Windows suite of operating systems, Windows 95 /98 and Windows NT, dominate the personal computer operating system market[1]. Because of this large market there is a vast amount of affordable Commercial off the Shelf (COTS) software products available for these operating systems. The popularity of Windows also makes it desirable to use it in other types of systems like real-time process and control systems. Windows NT is the most robust of all the Windows operating systems and it is therefore the most likely candidate for use in real-time systems. Using Windows NT allows these systems to utilize low cost COTS software and increases the user friendliness of the system because they look and feel more like the personal computer environment. Because of these market forces, the use of Windows has also been mandated by various agencies within the Department of Defense[2][3][4]. Unfortunately NT was designed as a general purpose operating system not a real-time one[5]. It was designed to optimize average performance and not worst case performance as is required of any real-time operating system. Despite its shortcomings there are several efforts underway to use NT as-is for real-time systems[6][7][8]. These efforts are motivated by the low cost of using NT based systems as well as the stability of the Windows Application Programming Interface (API)[9]. In this paper we quantify factors that determine under what conditions it makes sense to use NT for real-time processing. Another alternative, that still allows the use of NT COTS, is to add a real-time extension to NT[10]. This extension is essentially a separate real-time operating system. In this Dual-OS architecture all real-time processing is performed under the real-time OS, and NT handles the non time critical processing. In this system all NT COTS still run out-of-the-box. Therefore development time and cost are still kept to a minimum. In the last several years a number of vendors have produced products that follow this approach. These include the products: INtime from Radi- Sys[11], RTXTM from VenturCom[12][13], and HyperkernelTM from Imagination Systems[14]. In this paper we evaluate one of these products, INtime, and show that it is significantly more deterministic than NT alone. There have been several previous studies which evaluated the real-time performance of NT and/or NT extension products[6][7][8][15][16][10][17]. However, in this paper we provide a direct and unbiased comparison between the two approaches. Many of the previous performance evaluations were performed by the manufacturers themselves. Another key factor, that we examine in detail, is the issue of system load. We find that the determinism of NT is significantly worse when considering a load. However, the determinism of processes running under the INtime realtime extension are relatively unaffected by a system load. 2 Evaluating real-time OS behavior The correctness of a calculation performed by a realtime system is determined by its ability to satisfy time constraints as well as its ability to operate error free[18]. In this paper we consider only real-time systems that are based on COTS operating systems which means that it is impossible to analyze all possible paths in an application. The use of

2 COTS also means that our analysis must be stochastic, based on data gathered empirically, rather than mathematical. We note, however, for complex hard real-time systems (such as those deployed by the Department of Defense) stochastic analysis is the technique generally used to verify critical timing requirements. 2.1 Real-time operating systems In order to construct a real-time computer system the underlining operating system must abide by several requirements[5]. These requirements are outlined below: Support for multiple preemptable threads - Individual threads are typically used to process different hardware devices. It is important that the threads are preemptable so that a scheduler can be used to control the allocation of system resources. Thread priority - A priority scheme is essential to ensure that time critical processing takes precedence over less critical processing. Synchronization between threads - Threads that share data require a means of synchronization so that data is read and written in the proper order. Deterministic timing of the operating system - This usually requires that the operating system be preemptable. A non-preemptable operating system could impose unbounded delays. Support to prevent priority inversion - Priority inversion occurs when a high priority task is waiting on a resource that is held by a lower priority task. The high priority task could be blocked indefinitely if a task with intermediate priority preempts the low priority task. The typical solution for priority inversion is to increase the priority of the low priority task until it relinquishes the resource. Using these five requirements one can evaluate the ability of an operating system to support real-time processing. For example, NT provides multiple preemptable threads but allows only 32 levels of priority. NT also does not provide a mechanism to prevent priority inversion for the highest priority threads. The design choices used in NT and how this relates to real-time systems are discussed in more detail in Section Testing a real-time operating system We used the six different types of benchmarks shown in Table 1 to evaluate the real-time capabilities of NT and the Dual-OS system. The Jitter benchmark measures the determinism of timers of different duration and resolution. The benchmark uses a sleep function to sleep for the specified amount of time and then calculates the actual sleep time using the Pentium performance counter. The jitter is the difference between the actual and desired sleep durations. The response benchmark tests the ability of a system to perform a fixed amount of processing in a deterministic amount of time. The test repeats simple operations such as floating point additions, block memory copies, or disk write operations for a base case duration of approximately 10 milliseconds. The Bintime benchmark tests to what degree the clock granularity is available to a real-time application. It records the latency between successive calls to a time of day clock function[19]. To support deterministic response to external events an OS must process interrupts in a timely fashion. The ISR latency benchmark measures the responsiveness of the OS to the assertion of the interrupt and the ISR release latency benchmark measures the delay seen when transferring control from the ISR to the interrupt thread that completes the processing of the interrupt. The final type of benchmark tests inter-thread/process communication by measuring the time it takes one thread to signal another. Benchmark Jitter Bintime ISR Latency ISR Release Latency Communication 2.3 Hardware setup TABLE 1. Real-time benchmarks Description Measures the difference between the observed and desired delay times of various OS timer functions Measures the amount of time required to perform a fixed amount of processing Tests the responsiveness of an operating system by measuring the time between calls to a time of day clock function Measures the amount of time from the assertion of an interrupt to the beginning of its ISR Measures the time it takes to switch from an ISR to an interrupt thread Tests various inter-thread/process communication mechanisms Table 2 describes the hardware and software configuration of the prototype systems used in our evaluation studies. Everything used is commercially available as standard COTS. All timings in our studies were gathered by the benchmarks themselves, we did not use any special purpose hardware to stimulate the system or gather statistics. We did not install INtime when evaluating the NT only system because INtime modifies the Hardware Abstraction

3 Layer (HAL) of NT. The HAL is the portion of NT that isolates the low level hardware details from the kernel and the device drivers. TABLE 2. Prototype hardware and software setup used in benchmark tests Component PC CPU Graphics Memory Description Dell Dimension XPS D megahertz Pentium II Matrox Millennium II with PCI bus retries disabled 64 Megabytes OS Windows NT Workstation 4.0 with Service pack 3 Real-time Extension INtime version 1.2 (only installed when performing Dual-OS testing) The behavior of device drivers may affect the real-time performance of the Dual-OS system. In particular we found that the Matrox graphics driver locked the PCI bus for extended periods of time. This problem can be eliminated by disabling the PCI retry feature of the Matrox device driver. See Section for a discussion of how device drivers can impact real-time performance.system Loading The main utility of using NT in a real-time system, with or with out a real-time extension, is the ability to run nonreal-time applications as-is. Therefore we expect real-time and non-real-time processes to be running simultaneously, making it important to include a system load while evaluating the prototype systems. Table 3 describes the different types of system load introduced in the benchmark testing. We also ran without a load, signified by the load None, to measure the affect an idle NT has on the real-time performance. To generate the Moderate load we used an application that moves an image around in the graphics frame buffer. This application stresses the CPU and graphics sub-systems. The Heavy load adds a full text search across all files in the hard disk and a playback of an MPEG video to the Moderate load. The WinBench suite is a set of benchmarks that are traditionally used to test the performance of different hardware platforms[20]. The benchmark suite is made up of several different benchmarks, each targeting a specific hardware sub-system. Each benchmark runs a set of popular Windows application programs and produces a performance metric. We use the WinBench suite, not to test system performance, but to determine the impact of different types of loading on real-time performance. Table 4 shows the different WinBench98 benchmarks used in our testing. The CPU test consists of two different tests, one targeting the CPU itself and one targeting the FPU. Both the Disk and Graphics benchmarks give results for a set of Business applications and a set of High-end applications. The business applications consists of programs such as: Microsoft (Access, Excel, Powerpoint, and Word) as well as Lotus The High-end applications include programs such as: Microsoft Frontpage and Visual C++ and Adobe Photoshop. Load None Moderate Heavy WinBench98 Sub-system CPU CDROM Disk Graphics 2.4 Testing methodology TABLE 3. Application loads Description No load other than the OS and the benchmark test A CPU intensive application Three applications: a CPU intensive application, a string search across all files on the hard disk, and playback of a MPEG video from the CDROM Benchmarks that target specific hardware sub-systems (cpu,disk,cdrom,graphics) TABLE 4. WinBench benchmarks Benchmark CPUmark32 FPUwinmark CD-ROM Winmark Business Disk Winmark High-end Disk Winmark Business Graphics Winmark High-end Graphics Winmark The number of tests that must be run, so that a representative maximum is seen, is a major consideration when testing these operating systems. The benchmark results consist of the results from a number of trials, where each trial consists of a series of 4,000 samples. We ran at least 40 trials (i.e. 160,000 samples) of each benchmark. To determine if more trials were needed we calculated a confidence interval over the means of the maximums from each trial. This formulation assumes that the maximums are normally distributed. For the NT only tests, additional trials were run until the confidence interval of the maximums

4 was less than 20% of the mean with a certainty of 95%. For the INtime tests we were able to achieve a confidence interval of less than 5% of the mean with the same level of certainty. Close to 1,000 trials were run for several of the tests to achieve this bound. Extra trials were needed for the benchmarks where the variance was the greatest. 3 Using Windows NT as a real-time OS Using NT stand alone as a real-time OS is attractive for several reasons. First it is relatively inexpensive. All that is needed is a standard PC, NT itself and program development tools like Visual C++. Portability and longevity is the second advantage of using Windows NT. The WIN32 API is a defacto standard and programs written using it should be compatible with future upgrades of NT[9]. However Windows NT was designed as a general purpose operating system and several design decisions limit its use as a real-time operating system. In this section we discuss these design issues and use our benchmark tests to determine under what conditions NT might be suitable for use as a real-time OS. 3.1 Scheduling and priority in NT NT uses a priority based scheduling scheme with 32 levels of priority. These 32 levels are divided into four different priority classes: IDLE, NORMAL, HIGH and REALTIME, where IDLE is the lowest priority and REAL- TIME the highest. The NORMAL and HIGH priorities can be modified dynamically by NT to achieve fairness and maximize average performance. The priorities of individual threads within classes can also be prioritized by using a priority modifier. NT uses this modifier to determine how to raise and lower priority levels while doing dynamic priority adjustment. REALTIME threads do not use priority modifiers and are not affected by NT s dynamic priority adjustment scheme[1]. 3.2 Interrupt processing In NT, interrupt processing is handled using two types of routines: an Interrupt Service Routine (ISR) and a Deferred Procedure Call (DPC). When an interrupt is detected, NT interrupts normal processing and passes control to the interrupt s corresponding ISR. Because an ISR disables all processing, other than higher priority interrupts, it is recommended that only minimal processing occur within an ISR. The bulk of the processing should be performed in a DPC. DPCs are invoked via a signal from their corresponding ISR. If there are more than one DPC in the system, they are handled one at a time in FIFO order. 3.3 Design issues that limit NT s use as an RTOS The scheduling and interrupt processing design choices described above limit NT s use as a real-time operating system. The limitations imposed by these design choices are listed below: No priority inheritance for REALTIME threads - Threads at this level are the highest priority, however there is no mechanism to prevent priority inversion. Limited number of priorities - There are only seven levels of priority for real-time threads. This severely limits the amount of control the system designer has over thread priorities. When more than one thread shares a priority level they are processed in FIFO order. DPCs are processed in FIFO order - Much of the work of a device driver occurs in a DPC. Because these DPCs are processed in FIFO order, a DPC that needs to perform time critical processing may be delayed indefinitely by less critical processing. Time-critical DPCs can also be delayed by the processing of low priority ISRs. Masking interrupts - Any code, running at kernel level, can effectively gain exclusive access to the CPU by disabling interrupts or raising the interrupt request level (IRQL) to the highest level. This problem is exasperated by the previous one because if a device driver needs deterministic response its only choice is to disabled interrupts. This winner-takes-all philosophy can lead to unpredictable and ill behaved device drivers. Page Swapping - Because NT uses virtual memory, page swapping can occur at any point during the execution of a thread. This can add a significant amount of indeterminism. There is a mechanism to lock pages in memory, however NT does not guarantee that the pages will remain locked if the process is swapped out [21]. IRQL mapping - The mapping of interrupts to IRQLs is performed dynamically by the HAL at system startup as it detects the devices attached to the system. Any assumptions made about the relative order of interruptassociated IRQLs within a system will not be portable because other systems may have different hardware architectures. Interrupts and DPCs are higher priority than real-time threads - Any and all interrupts and DPCs, including non-real-time NT device drivers, can preempt any realtime thread. This means that non-real-time activities, like the movement of the mouse, will preempt time critical processing.

5 3.4 Benchmark results for NT In this section we use the benchmark tests described in Section 2.2 to evaluate NT as an RTOS. Real-time performance of NT under load. Table 5 shows the maximum and average values for the benchmarks of Table 1 under the different loads described in Table 3. For the benchmark the results are given in milliseconds, for all other benchmarks the results are given in microseconds. For all benchmarks, except the benchmarks, a good result is one that is close to zero. For the benchmarks a desirable results should be close to the base case, as defined in Section 2.2, of approximately 10 msec. TABLE 5. Benchmark results for NT under different loads Benchmark Max Avg Max Avg Jitter, T=10, R= R= R= Bintime (Add)(msec) (Copy)(msec) (Disk)(msec) ISR Latency DPC Timing As the results show the average real-time performance for NT is reasonable. The average under Heavy Load is higher than under the, but still within an acceptable region. These results are not surprising because NT is designed to optimize average performance. However, in a real-time OS, determinism is important and therefore the maximum value of the benchmarks is a more important statistic. In this case the benchmark results show that NT performs rather poorly. For example, there is a maximum jitter of close to 7 msec for a timer with a period of 10 msec without a load. The jitter is even worse (25 msec) when there is a heavy load, i.e. two and a half times the period. Percentage wise the maximum jitter is better for the 100 msec timers than the 10 msec timer because the jitter in the longer timers represents a smaller percentage of the total period. The maximum jitter, at around 25 msec, represents a jitter of about 25% of the period of the timer. For process control systems, a general rule of thumb is that scan time variations should not exceed 5% of the cycle time [7]. Many of the studies performed by process control manufacturers indicate that NT can meet the 5% scan time variation requirement. However, our tests show that under a system load this is not always the case.the general conclusion of the jitter results is that NT can support at best a timer with a period of 100 msec. The bintime results also give a similar conclusion. Under a the maximum time between calls to the time of day clock is 218 msec. Therefore the timer resolution available to a realtime application is on the order of hundreds of msec. Frequency Frequency Moderate Load Jitter (µ sec) FIGURE 1. Histogram of NT jitter for period T=10 ms and resolution R=1 ms Moderate Load Jitter (µ sec) FIGURE 2. Histogram of NT jitter for period T=100 ms and resolution R=10 ms The jitter results near the maximum value represent a very small percentage of the total number of samples. Figure 1 and Figure 2 show histograms of jitter results for

6 two different timers: (T=10,R=1) and (T=100,R=10). The results are plotted on a log/log scale and each point represents a discrete bin; the lines connecting the points are included only to improve the readability of the histogram. As Figure 1 shows, for the 10 millisecond timer, there are more than three million samples less than 100 msecs but only around 100 that are greater than 1 msecs for a Moderate Load. The samples that are greater than 1 msec only represent 0.003% of the total samples. The results for the 10 msec timer under are slightly better but very similar to the results under a Moderate Load. For a Heavy Load the maximum jitter occurs more frequently. However a jitter over 1 msec occurs in only 12% of the cases. Figure 2 shows a histogram of the results for a timer with a period of 100 msecs and a resolution of 10 msecs. Out of the more than one million samples gathered for the case with a Moderate Load, only one is on the order of 1 msec. There are also less than 0.3% that are greater than 100 µsecs. The results are worse when a is used. In this case about 12.5% are greater than 1 msec and 1.4% greater than 10 msecs. The response results shown in Table 5 show that even without a load the time to perform a fixed amount of processing can take anywhere from 59% to 210% more time than the desired base case. The average response is close to the desired time in all cases, i.e. about 10 msec. The results are the worst under a. The worst case for the Add test is more than six times that of the base case and for the Disk test the maximum time takes close to four seconds. The Disk test is especially bad because the DPCs used in its I/O processing have to contend with DPCs used by the application programs used to create the loads. The two benchmarks that test the interrupt performance of NT show that NT s interrupt response time is fairly good, but the time it takes to start processing a DPC can be more than 11 msecs. Interrupt response time is good because interrupts are higher priority than anything else in the system. Table 5 shows that the maximum interrupt latency is 54.8 µsecs. The maximum DPC service time is so great because DPCs are processed in FIFO order. When the system is loaded the DPCs in the benchmark are delayed by the DPCs generated by the application load. Testing NT using WinBench 98. Using WinBench98 to generate a load allows us to pinpoint more accurately the type of processing that causes the greatest effect on realtime performance. Table 6 shows the maximum values for the different WinBench loads. As Table 6 shows the timer jitter is most affected by a load that contains a lot of I/O processing. The CD-ROM WinBench load produces the worst case jitter followed by the Disk and Graphics tests. The worst case jitter for the CD-ROM test is close to 1.4 seconds for a timer with a period of 100 msec and resolution of 10 msec. The jitter under a CD-ROM load is worse than any of the jitter results shown in Table 5. The CPU WinBench load creates the least amount of worst case jitter, at less than 5 msec. Here again DPCs are the main cause of jitter. The I/O intensive loads create more DPCs which always run at a higher priority than the real-time thread that is waiting on the timer. This effect is substantiated by the results of the DPC timing test shown in Table 6 where a maximum delay of nearly 11 msec is seen while using the CD-ROM Win- Bench test as a load. TABLE 6. NT benchmark results under different WinBench 98 loads Maximum value CPU/ CD- Benchmark FPU ROM Disk Graphics Jitter, T=10, R= R= R= Bintime (Add)(msec) (Copy)(mses) , (Disk)(msec) ISR Latency DPC Timing The WinBench loads have an even greater effect on the real-time response than the loads of the previous section. For example the Add test has a maximum response time that is 17 times the base case under the CD- ROM WinBench load. Conversely the CPU WinBench load has the least effect on the Add test. The test and the WinBench load compete equally for the CPU. Likewise the Disk test is least affected by other disk activity. The Disk Winbench load has the least effect on the Disk benchmark. Table 6 shows that the maximum Bintime result is dramatically affected by the WinBench loads. Bintime tries to utilize 100% of the CPU and when it must compete with another CPU intensive application the delay between calls to the clock function can be as high as 870 msecs.

7 4 Dual-OS system As seen in the previous section, NT alone is not well suited for use as a real-time operating system. However, frequently real-time systems utilize a mixture of real-time and non-real-time processing. Because of the vast amounts of COTS available for NT there is an advantage, in terms of cost and usability, in using NT for the non-real-time processing. These requirements have led to the development of several products which add a real-time extension to NT, enabling a greater degree of real-time performance to be achieved while retaining the ability to run NT applications out-of-the-box. These extensions essentially create a Dual- OS system where all the real-time processing runs under the extension and the non-real-time processing runs on NT. A possible disadvantage of these products is the added software cost. The cost of these products is driven by the relatively small real-time computing market and not the mass consumer market. Therefore the extension products will have a price closer to that of an RTOS which is considerably more than the cost of Windows NT. However, these costs must be weighed with the savings of using NT COTS software for non-real-time tasks. Therefore the Dual-OS approach provides a solution that has better real-time performance than NT but is more cost effective than using a custom RTOS. To evaluate the usefulness of this Dual-OS approach we evaluated one of the products, INtime from RadiSys, using the same benchmarks used for NT in the previous section. The remainder of this section describes INtime and presents the results of the benchmark testing. 4.1 The INtime real-time extension to NT INtime encapsulates all of NT into one hardware task and creates another hardware task for the real-time extension. This encapsulation provides hardware isolation of the real-time memory space from the NT memory space. Everything that runs under NT then runs within a single task that runs at the lowest priority under INtime. INtime modifies the Hardware Abstraction Layer (HAL) of NT to prevent NT from modifying the real-time clock and disabling or remapping real-time interrupts Design of INtime Scheduling. The INtime scheduler is hooked into the NT timer interrupt, the highest priority interrupt other than the Non Maskable Interrupt (NMI). The INtime scheduler can therefore preempt any NT thread, DPC or interrupt processing. There are 256 priority levels within INtime compared to NT s 32 levels. Interrupt processing. Intime uses an interrupt mechanism similar to NT s. Interrupt processing under INtime is performed using two elements: an ISR and a corresponding interrupt thread. The ISR acknowledges the interrupt and should perform a minimal amount of processing because all interrupts are disabled while processing an ISR. The system calls that an ISR can use is also limited. The bulk of the processing should be performed in an interrupt thread. An interrupt thread is analogous to a DPC, however interrupt threads are prioritized. Communication and synchronization. INtime provides a number of means of communication and synchronization between real-time threads as well as between a real-time process and a NT process. Message passing between threads is accomplished using mailboxes. A mailbox can be used to pass up to 128 bytes of data between threads. When sending a message the sender queues the sent message and returns. The receiver of the message blocks on the receive operation until the message is delivered. Shared memory is used to share larger amounts of data between threads. All shared memory is locked into INtime s address space and cannot be paged out of memory. INtime provides two types of synchronization mechanisms: semaphores and regions. A semaphore is a autonomous shared binary counter. One thread signals the semaphore and another blocks waiting on the signal. Regions can be used to guard critical sections of code. Priority inheritance is implemented in regions thereby providing a solution to the priority inversion problem. Real-time API. To control the real-time mechanisms such as thread scheduling and communication INtime provides a real-time API which is patterned after Win32. However, since Win32 is not a real-time API, the real-time API is specific to INtime. This is one of the drawbacks of using an extension product, i.e. the code is dependent on one particular vendor. The alternative of modifying Win32 function calls to support real-time processing is not an attractive solution either. An API which is syntactically the same as Win32 but semantically different introduces inconsistencies and invites application errors. Also different vendor s modifications to Win32 will undoubtedly be inconsistent INtime s support for real-time processing Because of its design INtime does not exhibit the same limiting factors with respect to real-time performance. The differences in INtime versus the choices made by NT, as described in Section 3.3, are as follows: 256 thread priorities: INtime provides 128 user thread priorities and 128 interrupt thread priorities.

8 Support for priority inheritance: Regions support priority inheritance in INtime which provides a solution to the priority inversion problem. Interrupt thread priorities: Unlike their counterparts in NT (DPCs) interrupt threads in INtime are prioritized. This eliminates indeterminate delays within time critical interrupt processing. This allows the systems designer to prioritize interrupts and therefore a time critical device driver does not have to disable interrupts. All memory used within INtime is locked into memory: This eliminates the nondeterministic delay caused by page swapping. Because of the basic design and use of NT based systems there is a fundamental limit to the amount of non determinism that INtime or any other real-time extension can eliminated from the system. One example is the fact that code running in NT kernel mode can disable interrupts using the CLI instruction. Because the real-time scheduler is connected to the highest priority interrupt, disabling interrupts therefore also disables the real-time scheduler. Within NT itself this is rarely done, but all device drivers run in kernel mode and there is no guarantee that they will not disable interrupts for prolonged periods of time. Another source of indeterminism, that cannot be prevented, is I/O bus locking. For example if a device driver locks the PCI bus it will block any other process that needs to access it. This occurs, by default, in the PCI based Matrox video card that was used in our testing. In this particular case the problem can be eliminated by disabling the automatic PCI retry feature of the device driver. 4.2 Benchmark results for the Dual-OS system In this section we use the benchmark tests described in Section 2.2 to evaluate the effectiveness of INtime. Real-time performance of INtime with a load. Table 7 shows the maximum and average values of the real-time benchmarks described in Table 1. With a few exceptions, the benchmarks correspond closely to those run under NT. INtime allows a higher timer resolution, so we ran additional jitter tests with periods of 1 msec and 200 µsec. The Disk test could not be run because INtime does not support NT disk operations within real-time threads. The Bintime benchmark tries to use 100% of the CPU therefore it is not possible to run it along side a load. Also included are the Semaphore test, which tests signaling delay of a semaphore, and the NT/RT Round-trip test which tests the communication delay between NT and a real-time thread. TABLE 7. Benchmark results for INtime under different loads Benchmark Max Avg Max Avg Jitter, T=0.2, R= Jitter, T=1, R= Jitter, T=10, R= R= R= Bintime N/A N/A (Add) (msec) (Copy) (msec) ISR Latency Thread Timing Semaphore NT/RT Roundtrip As Table 7 shows real-time processing under INtime is much more deterministic than NT. The worst case timer jitter is less than 50 µsec even under a. For all cases the jitter is lower than the timer resolution. The average jitter is also low. Ranging in value from 0.57 µsec up to a maximum of 8.9 µsec. The response for Bintime is at Frequency Moderate Load Jitter (µ sec) FIGURE 3. Histogram of INtime Jitter for period T=10 ms and resolution R=1 ms

9 most 18.2 µsec compared to a maximum of 6.9 msec for NT. The 18.2 µsec Bintime response seen for INtime, is better than response times of 16 msec for Lynx OS and 65 msec for Solaris seen in another study[22]. Figure 3 and Figure 4 show histograms of the jitter for two timers (T=10, R=1 msec and T=100, R=10 msec). For the 10 msec timer 99.4% of the samples are less than 10 µsec without a load. With a 81% of the samples are less than 10 µsec. Frequency Moderate Load Benchmark Jitter, T=0.2, R=0.2 Jitter, T=1, R=1 Jitter, T=10, R=1 R=1 R=10 (Add) (msec) TABLE 8. INtime benchmark results under different WinBench loads CPU/ FPU Maximum value When trying to decide whether to use NT for a real-time system or whether to also include a real-time extension, one must weigh the real-time requirements of the system versus the issues of cost and portability. NT as a technology is mature and widely used so software developed under it will have better longevity than software developed using a vendor specific API. The real-time extension products are not as mature but they offer the opportunity for significantly better real-time performance. One must also consider, that when following either approach, there are always going to be limitations imposed by using an NT based system and COTS software[23]. The benchmark results in this paper show that the realtime performance of the INtime extension product is significantly better than that of NT. We show that INtime can support periodic processing on the order of microseconds, under any loading condition, whereas NT can only support millisecond timers under light or no load. INtime can per- CD- ROM Disk Graphics Jitter (µ sec) FIGURE 4. Histogram of INtime Jitter for period T=100 ms and resolution R=10 ms (Copy)(msec) ISR Latency Thread Timing Testing INtime using WinBench 98. Table 8 shows that, even though the real-time processing under INtime is still more deterministic than NT, the WinBench loads have a more significant effect on real-time performance than the application loads shown in the previous section. The worst case jitter is slightly greater, but is still less than 100 µsec. The Copy test shows the most significant difference in determinism. For all the WinBench loads the maximum is about 22% higher than the base case. This is most likely attributed to cache effects which occur because the data written in the test is flushed from the cache by the WinBench load. This worse case happens relatively infrequently. For example, for the CD-ROM load only 0.006% of the samples are greater than 20 msec. The other benchmarks also show a significant advantage for using INtime versus NT. The Add and Copy response results are also very close to the base case. The interrupt latency is slightly better for INtime and the thread timing results are significantly better than the DPC timing results of NT. The interrupt threads are higher priority than any NT processing. Communication between threads in INtime is very deterministic, as shown by the results of the Semaphore test. However, the NT/RT Round-trip benchmark shows that, communication between NT and a real-time thread can take as much as 8.75 msec. Semaphore NT/RT Round-trip , Conclusion - Comparing the real-time extensions to NT

10 form a fixed amount of processing with at most a 22% increase in processing time. NT on the other hand exhibits a worse case increase close to 10,000%. Even though our benchmark results are based on only one of the available products we expect that other products, that use the same basic underlying technology, would exhibit similar results. The difference between NT and INtime is the largest when non-real-time application loads are run alongside real-time ones. It is important to consider this non-realtime processing because the biggest advantage of using NT is the ability to leverage the vast amount of COTS software available for it. Also, our benchmark tests only consider a single thread of real-time processing; running multiple threads will also increase the overall system load. The Dual-OS approach addresses these two problems by first: isolating the non-real-time processing of NT into a low priority task and second: using a priority scheme for all objects in the system so that all real-time processing can be prioritized. Because the development of NT applications has a lower software cost, for systems with fairly soft real-time requirements using NT by itself may be more practical than the Dual-OS approach. NT is a lower cost solution because the general purpose operating system market is much larger than the real-time OS market. There are possibly other software development costs associated with using a real-time extension. Each different extension product uses its own API. This requires a time investment on the part of the developer to learn this API, and if in the future the product is no longer supported, porting the system to a new environment requires learning a different API still. Also because of limited resources the extension manufactures generally provide real-time device drivers for only a subset of the different types of peripheral devices. Using any hardware not supported requires writing a custom device driver. There are other PC based operating systems, other than NT, which may allow the implementation of real-time systems using PC hardware. Windows CE has potential as a real-time OS[24][25]. CE requires a relatively small footprint and was developed with the hand-held market in mind. It is also modular which allows a system developer to customize the OS for each system. However CE is not identical to NT and therefore has the disadvantage that it cannot run NT applications out-of-the-box. Linux is another operating system that, in the future, may allow real-time processing on PC platforms. Linux is a UNIX variant and there are current research efforts underway for the development of a real-time Linux[26][27]. References [1] H. Custer. Inside Windows NT. Microsoft Press [2] Defense Information Systems Agency (DISA). Defense Information Common Operating Environment (DII COE) [3] Defense Information Infrastructure (DII COE) Integration and Runtime Specification (I&RTS), Version 3.1, [4] Dept. of the Navy (DON CIO ITSGIPT). Information Technology Standards Guidance Version [5] M. Timmerman and J. Monfret. Windows NT as Real-time OS? Real-Time Magazine 2Q [6] K. Ramamritham, et al. Using Windows NT for Real-time Applications: Experimental Observations and Recommendations Proceedings of RTAS [7] R. Malina. Using the Windows NT OS for Soft RT Control Rockwell Automation [8] Real-Time Systems and Microsoft Windows NT Microsoft msdn/library/backgrnd/html/realtime.htm. [9] R. Simon. Windows NT WIN32 API Superbible. Waite Group Press [10] M. Timmerman et al. Windows NT Real-Time Extensions: Better or Worse Real-Time Magazine 3Q [11] INtime: [12] B. Carpenter, et al. The RTX Real-Time Subsystem for Windows NT Proceedings of the Usenix Association Windows NT System Engineering Workshop [13] VenturCom homepage: [14] Imagination Systems, Windows NT for Real-Time Control? [15] N. Frampton, J. Tsao, J. Yen. Hard Real-time Extensions of Windows NT Evaluation Report GM Powertrain [16] M. Jones and J. Regehr. Issues in Using Commodity Operating Systems for Time-Dependent Tasks: Experiences from a Study of Windows NT Proc. of NOSSDAV [17] K. Obenland, et al. Dual Real-time/Non-real-time Common Operating Environment MTR 98W [18] E. Douglas Jensen. Real-time for the Real World. [19] L. Monk, et al. Real-Time Communications Scheduling: Final Report The MITRE Corp., MTR 97B [20] Ziff-Davis Inc. WinBench98 Benchmark suite. [21] J. Richter. Advanced Windows. Microsoft Press [22] R. Freedman et al. Real-time Benchmarking of the Solaris and Lynx Operating Systems MTR [23] Radisys Corp. Determinism and the PC Architecture. [24] RT Systems and Microsoft Windows CE Microsoft [25] M. Timmerman, et al. Is Windows CE a real threat to the RTOS World Real-Time Magazine 3Q [26] Michael Barabanov. A Linux-based Real-time OS. Master s Thesis New Mexico Institute of Technology [27] B. Srinivasan, et al. A Firm Real-Time System Implementation Using COTS HW and Free SW. RTAS