Case Study: Challenges and Benefits in Integrating Real Time patch in PowerPc Based Media System

Transcription

1 Case Study: Challenges and Benefits in Integrating Real Time patch in PowerPc Based Media System Manikandan Ramachandran Infosys Technologies Limited Electronic City, Hosur Road, Bengaluru ,India mani Aviral Pandey Motorola Mobility 2450 Walsh Avenue,Santa Clara, CA 95051, USA Abstract Media systems generally have many CPU intensive as well as time critical processes. Vanilla Linux 2.6 with preemption enabled does provide solution to this kind of system. However, if the system is interrupt intensive, as ours, then vanilla Linux 2.6 performance is not expected to be good; as by definition interrupt preempts all higher priority tasks. RT patch seems to address exact issue by providing an option to handle interrupts in process context, but the solution address exact issue by providing an option to handle interrupts in process context, but the solution doesn t seem to fit customized Linux. Quite a bit of architecture changes had to be made to reap the benefits of RT patch. This paper describes about various challenges faced in integrating Ingo s RT patch on a customized PowerPc Linux package. And explains how those challenges were overcome. It describes how LTTng can be used to identify the bottlenecks and finally concludes by comparing performance of the application that was run on vanilla and RT Linux. 1

2 1 Introduction Linux operating system is evolving fast as a defacto embedded operating system. Looking back the community has made tremendous progress from being stuck with big-o-lock to a really preemptible kernel in 2.6, and to fairly predictable kernel with RT patch. However,still there are few challenges in using Linux for commercial systems that require real-time capabilities. This paper takes one of such case and walks through the challenges in integrating RT patch along with slew of other patches. After all the effort in integrating, the real performance of the system is not up to the expectation. The paper dwells deeper into a performance issue and proposes a solution to fix that issue. 2 Product Overview The system identified for this study is a head-end media device. The device is a real time video splicer and statiscal remultiplexer that can groom hundreds of video services. The device has 2 PowerPc 7448, built on a Marvell communication controller board [MV64460]. Real MPEG related operation is done by a proprietary application that is highly CPU intensive and it has the highest priority among all processes. In short, proprietary application almost expects hard real time capability from the operating system. The figure below gives intended hierarchy of CPU resource to be utilized by various processes/interrupts. 3 Problem Statement The most important goal of the product software designis that wheneverthereis aninterrupt fromvideo hardware, kernel has to schedule video processing tasks with almost zero latency. In theory assigning high real time priority should take care of this requirement; however in few instances it has been demonstrated that Linux kernel has failed to honor priority of tasks because of various other kernel dependencies. Linux Trace Tool next generation or LTTng is an open source tool that helps to find performance bottle neck on Linux based system. Using LTTng, following system bottlenecks were identified: 3.1 Multiple IDE interrupts The media product has two compact flashes each of which can be accessed through kernel s IDE driver. Any access to compact flash generates lot of interrupts that could preempt the high priority application process and cause degradation in system performance. This issue was kind of mitigated by deferring IDE request by 1 millisecond, think of it like yielding CPU to other tasks before scheduling another IDE request. This feature ensures that there is only one IDE interrupt during 1 millisecond window. This solution seems to work; however in certain condition, when there is excessive IDE access, LTTng shows that more than 1 IDE interrupt could occur in 1 millisecond window. FIGURE 2: IDE interrupt in video process context FIGURE 1: Processes Hierarchy This scenario happens when video hardware interrupts and IDE interrupts are handled one after the other. Figure 2[Refer Appendix A for legend] is a capture of LTTng viewer. Marker 1 and Marker 2

3 2. shows that there are more than 1 IDE interrupts while application process is woken up by video interrupt. Marker 3 just shows the events that occurred around Marker Softirqs preempts high priority process Linux kernel is driven by timer interrupts. Whenever there is timer interrupt, scheduler is woken up and deferred tasks are executed from.softirq. context. The occurrences of timer interrupt can be configured. In our kernel we have configured that to be 1 millisecond. So every 1 millisecond, softirq daemon is woken up and it preempts all other high priority tasks. Usually softirq daemon doesn t hog CPU [it would take microseconds], but if there are many kernel soft timers registered it could hog the CPU. In oursystemthere are2scenariosin which spinlock usage is inevitable: Custom drivers handling interrupts or to provide mutual exclusion to critical resource. Kernel usage of spinlocks. One common instance is usage of spinlocks in printk function. 4 A Case for Real-Time Patch Real-time patch [3] provides following features that make the kernel more preemptible: Makes spinlocks preemptible by default. Option to handle IRQs in kernel thread context instead of IRQ context. Softirqs are handled in kernel threads context. Our study about system performance issues with vanilla kernel made a compelling argument for using RT patch. We thought by using RT patch we could address our system performance issues. Based on that thought we made following changes to our system software architecture: FIGURE 3: softirq stealing video process context In Figure 3, marker points to context switch from application process to softirq daemon. In this case few micro seconds are lost from application process context. 3.3 Spinlocks and Preemption In multiprocessor system spinlocks are used to provide mutual exclusion for any critical resource. SpinlocksbyitsnaturedisablespreemptioninCPUofthe current process. So if a lower priority process happens to take spinlock before a higher priority tasks scheduled to run, then as long as spinlock is held by the lower priority task, higher priority is task is starved for CPU time. This is one scenario, where priority of process fails to get honored. Apply RT patchto ourkernelandmakedrivers compatible to that. All interrupts except video hardware interrupts were made to handle as kernel threads. Priority of Application process was made higher than interrupt and softirq threads. The idea of new design is that whenever there is a video interrupt, media process is woken up with zero or little latency. Another expectation from RT patch is that, the feedback mechanism used in IDE driver tothrottleiderequestcanberemovedandrtpatch would inherently take care of preempting low priority tasks including tasks that handle interrupts. 5 Challenges in Integrating RT-patch The end goal of this exercise was to have a stable real-time Linux kernel and have performance profiling capability by integrating LTTng. At the time of this exercise was the latest and stable kernel 3

4 version so we choose real-time patch version and LTTng version [version]. Patch process went about quite smoothly, but we had following run time issues: 1 Unable to boot the System in SMP mode: The cause of this issue was found to be with calling kzalloc from setup cpu. Apparently this issue is not seen in non-real-time kernel. We worked around this issue by statically allocating memory rather than using kzalloc or kmalloc. 2 Dependency on BSP code: When used realtime patch over Marvell s BSP code, we found lot of recursive locks issues. We identified those issues and fixed them. 3 Handling IRQ in a Thread: This is one of the toughest challenges that we are dealing with. In first look, after booting real-time kernel one would think that interrupts are handled in Interrupt threads, but we found it in hard way that interrupts continue to get handled in interrupt context unless a few specific changes are made[4]. This issue is the main point of this paper, and it is discussed in detail in section Performance Analysis. 6 Performance Measure After applying real-time patch we tried to get raw performance measure of our system using Cyclic testr [5] and the actual application performance in a controlled test environment. From cyclic test, we noticed that the performance of the system was similar to vanilla kernel. However, with our custom product performance test we found that the performance of the application has deteriorated a lot. Following section describes our test scenario and results. 6.1 Kernel Performance Measure Cyclic test is used to determine the internal worstcase latency of a Linux real-time system. Cyclitest measures the amount of time that passes between when a timer expires and when the thread actually runs the thread is woken[9]. We used this tool in both patched real-time kernel and vanilla kernel. The Cyclic test test was run with following parameter set: one thread with priority 80 Thread to run tight loops Interval between each loops micro seconds We conducted this test in 2 phases. In first phase, all application was stopped and made sure CPU utilization was less than 1% on both CPU. Then we ran cyclic test first with vanilla kernel then cyclic test was ran on kernel with real-time patch. Results of the test are given in Figure-4. FIGURE 4: Cyclic Test with No Load In second phase, both CPUs of the system were heavily loaded using a script that just creates a tight loop. Multiple instances of this script were run concurrently till the CPU utilization of the system reached 97%. Cyclic test was run in this loaded scenario. Figure 5 gives the result of cyclic test on both kernels. FIGURE 5: Cyclic Test with Load Inference From figure 4 we see that the minimum latency for vanilla kernel is 896 microseconds which is less than real-time patched kernel. While maximum latency for real-time patched kernel is more than 1milliseonds which is far worst than vannila kernel. From figure 5, we can infer that latency isnr t bad with loaded system. Both for vanilla and realpatched kernel we see the difference is only about 10 micro seconds. 4

5 The conclusion of this test is that we don t observe huge improvement in performance of the kernel/system with real-time patch. 6.2 Application Performance Measure The idea behind this performance test is to study how vanilla and real-time patched kernel behaves when a high priority process is made to starve for CPU time. Since real-time patch has option to handle interrupts in process context and makes normal spinlocks preemptible, the expectation and the desire was that the real-time patched kernel always honors process.s priority irrespective of other processes or kernel state. In our case, Media application is configured to have higher priority than all other processes, including processes that are supposed to handle interrupts Test Case Description The Media application was configured to run at full capacity, so that CPU resource is solely taken by this process. Executions of other trivial tasks were reduced. Thetestwasthenrunon3differentkernels: [scheduler based on RB-tree] [CFS scheduler] RT patch FIGURE 7: CPU1 load comparison Figure6and 7 comparesthe performanceof each kernel while the process was stressed for 30 minutes Inference From Figure 6 and Figure 7 it is apparent that application performance on real-time patch kernel is not better than its performance on vanilla kernel. Only difference between real-time patch kernel and vanilla kernel from a platform perspective is that a patch to throttle IDE access was not ported to real-time kernel. TomakesureIDEaccessis the causeofthe issue, excessive IDE access was made when media application process was fully loaded. Under this scenario, we found that the performance of the application deteriorated further. Ideally a process that handles IDEinterrupts should have yielded to high priority task processes like the media application process; however, from this test it looks like media application process seems to be preempted when there is an IDE interrupt. 7 Application Performance Analysis FIGURE 6: CPU0 load comparison After making the kernel real-time enabled, the expectation was that the IRQ thread will be preempted by high priority media application process. However, as demonstrated in previous section, we have seen that performance of the application was not good with real-time patched kernel. There could be two possibilities for this kind of behavior: 1. Scheduler is not honoring processes priority. 5

6 2. IDE interrupts are still handled in interrupt context like vanilla kernel. Weconfirmed Case1 isnotthecasebyrunning rt-migrate-test written by Steven Rostedt[6]. This is a simple program that creates multiple threads with various priority and then checks if scheduler honors the priority of each thread. This test passed both in normal scenario and in heavily loaded system with IDE interrupts and media application process. To check if IDE interrupts are still handled in interrupt context, we patched real-time kernel with LTTng. Then with patched kernel, the media application was stressed while generating multiple IDE interrupts. in thread, one has to use request threaded irq and split the handling of irq into two parts by using two handler functions. In first part, interrupt will be handled in interrupt context. In this context, handler should disable the source of the interrupt and wake corresponding interrupt thread. The second handler should do the actual handling of the interrupt, which obviously will be done in the thread context. Having identified the bottleneck, we are in process of converting IDE and other similar CPU intensive interrupt handlers to be run in thread context. 9 Future Work 9.1 Creating thread function for interrupt handling As discussed in section 8, just having real-time patch is not go enough for making interrupts to be handled in threads. A thread function, which handles deferred interrupt work, should be designed and implemented. In future, we intend to make low priority interrupts that are CPU intensive to be handled in threads. 9.2 High Resolution Timers FIGURE 8: Video Process being preempted by interrupts Figure 8 captures the various process states at the time of the issue. irq75-ide1 is the handler process for the IDE interrupt: in this case IDE interrupt number is 75.From the viewer, we can interpret that video process was starved for CPU as it was busy handling IDE interrupt 75 in interrupt context rather than in the context of the thread irq75-ide1. This is the cause of video process starvation. 8 Solution We started looking into request irq implementation in real-time patched kernel to understand why IDE interrupts were handled in interrupt context not in thread context. From request irq implementation it was clear that irq thread did nothing if handlers for an interrupt were installed through request irq call. In order for interrupt to be handled We didn.t enable.high Resolution Timers. in any of the current kernel configuration as we weren.t sure about the support of this feature in PowerPc based system [8]. In future, we intend to do deep profile of kernel performance with high resolution timers. 10 Conclusion Effectiveness of real-time patch on a system depends on various factors. In our case, preemption of IRQ handler is the single most important criteria in determining system performance. Although real-time patch creates interrupt threads by default, it really doesn.t mean that interrupts are handled in those threads. This behavior confuses the user. Instead, it would have been better if no interrupt threads were created. If one prefers to have an interrupt threaded then he or she may do so by using request threaded irq. This case study shows that applying real-time patch will not solve all real time requirements of a system. Instead it provides great tools like preemptive spinlocks, threaded interrupt handler etcetera. 6

7 To make a system near real time capable, one has to understand their system bottlenecks. LTTng is a great tool which clearly brings out hard to understand system behavior. As demonstrated in this case, LTTng was used extensively to get to bottom of many performance issues. It is highly recommended to use LTTng to understand system performance issues and make few system software architecture changes to take advantage of tools provided by real-time patch. To summarize, real-time patch is not a cure-all of all system real-time owes; instead it provides great tools that could be used as the first step to make a system real time capable. References [8] High resolution timers and dynamic ticks design notes [linux/documentation/timers/highres.txt] [9] rt-latency-howto.txt [ A Appendix Following is the legend for LTTV output. Source: guide/x81.html. [1] Introduction to LTTng [ [2] How to Use LTTV [ [3] Introduction to RealTime Patch[ [4] Cyclictest examples [ [5] Moving interrupts to thread by Jake Edge.[ [6] Real Time Threads [7] Investigating latency effects of the Linux realtime Preemption Patches (PREEMPT RT) on AMDs GEODE LX Platform FIGURE 9: Lengend For LTTV 7