An Implementation of the PCP, SRP, D-PCP, M-PCP, and FMLP Real-Time Synchronization Protocols in LITMUS RT

Transcription

1 An Implemenaion of he PCP, SRP, D-PCP, M-PCP, and FMLP Real-Time Synchronizaion Proocols in LITMUS RT Björn B. Brandenburg and James H. Anderson The Universiy of Norh Carolina a Chapel Hill Absrac We exend he FMLP o pariioned saic-prioriy scheduling and derive corresponding wors-case blocking bounds. Furher, we presen he firs implemenaion of he PCP, SRP, D-PCP, M-PCP, and FMLP synchronizaion proocols in a unified framework in a general-purpose OS and discuss design issues ha were beyond he scope of prior algorihmic-oriened work on real-ime synchronizaion. Inroducion Wih he coninued push owards mulicore archiecures by mos (if no all) major chip manufacurers [, 8], he compuing indusry is facing a paradigm shif: in he near fuure, muliprocessors will be he norm. While curren offhe-shelf sysems already rouinely conain processors wih wo, four, and even eigh cores (examples include he Inel Core Duo, he AMD Phenom, and SUN UlraSPARC T processors), sysems wih up o 8 cores are projeced o become available wihin a decade [8]. No surprisingly, wih mulicore plaforms so widespread, (sof) real-ime applicaions are already being deployed on hem. For example, sysems processing ime-sensiive business ransacions have been realized by Azul Sysems on op of he highly-parallel Vega plaform, which consiss of up o 768 cores [5]. Moivaed by hese developmens, research on muliprocessor real-ime sysems has inensified in recen years (see [5] for a survey), wih significan effor being focused on boh sof and hard real-ime scheduling and synchronizaion [6, ]. So far, however, few proposed approaches have acually been implemened in operaing sysems and evaluaed under real-world condiions. In an effor o help bridge he gap beween algorihmic research and real-world sysems, our group recenly developed LITMUS RT, a muliprocessor real-ime exension of Linux [9, 3]. The developmen of LITMUS RT has occurred a an auspicious ime, given he increasing ineres in real-ime varians of Linux (see, for example, []). These varians will undoubedly be pored o mulicore plaforms and hus could benefi from recen algorihmic advances in scheduling-relaed research. LITMUS RT has been used o assess he performance of various dynamic-prioriy scheduling policies wih real-world overheads considered [3]. More recenly, a sudy was conduced o compare synchronizaion alernaives under global and pariioned earliesdeadline-firs (EDF) scheduling []. The versions of LITMUS RT published so far have exclusively focused on dynamic-prioriy scheduling algorihms. In his paper, we exend his work by presening an inegraed implemenaion ha suppors five major real-ime synchronizaion algorihms under pariioned saic-prioriy (P-SP) scheduling. To our knowledge, his is he firs such implemenaion effor o be conduced on a modern generalpurpose muliprocessor operaing sysem. Moreover, including suppor for P-SP scheduling in LITMUS RT is imporan, as saic-prioriy scheduling is widely used. Prior Work. Sha e al. were he firs o propose proocols for uniprocessors o bound prioriy inversion he prioriy inheriance proocol and also avoid deadlock he prioriy ceiling proocol (PCP) [7]. As an alernaive o he PCP, Baker proposed he sack resource policy (SRP) [4]. Boh he SRP and he PCP have received considerable aenion and have been applied o boh EDF and rae monoonic (RM) scheduling. Rajkumar e al. presened wo exensions of he PCP for muliprocessor real-ime sysems under pariioned saicprioriy scheduling: he disribued prioriy ceiling proocol (D-PCP) [6], which does no require shared memory and hus can be used in disribued sysems as well as ighlycoupled muliprocessors, and he muliprocessor prioriy ceiling proocol (M-PCP) [4], which relies on globallyshared semaphores. Several muliprocessor synchronizaion proocols have been proposed for pariioned EDF scheduling. Chen and Tripahi [4] proposed a soluion ha only applies o synchronous periodic asks. Addiionally, muliprocessor exensions of he SRP for pariioned EDF were proposed by Lopez e al. [] and Gai e al. [6]. Given he experimenal focus of his paper, i is worh noing ha Gai e al. no only inroduced a new locking proocol, he muliprocessor sack resource policy (M-SRP), bu also discussed an implemenaion of i. Their sudy showed ha he M-SRP ouperforms he M-PCP. In recen work, Block e al. proposed he flexible muliprocessor locking proocol (FMLP) for boh global and pariioned EDF and showed ha i ouperforms he M- SRP [6]. Conribuions. The conribuions of our work are hreefold: (i) we exend he FMLP o P-SP scheduling and derive corresponding wors-case blocking bounds; (ii) we presen

2 and discuss in deail he firs implemenaion of he SRP, PCP, M-PCP, D-PCP, and FMLP in one unified framework (which is available publicly under an open source license [7] and, we hope, will serve as a guide for praciioners); and (iii) we discuss implemenaion and sofware design issues no fully considered in earlier algorihmicoriened work on real-ime locking proocols. The res of his paper is organized as follows: Sec. provides an overview of needed background, Sec. 3 presens he FMLP for P-SP, Sec. 4 discusses he implemenaion of he synchronizaion proocols lised above in LITMUS RT, and Sec. 5 concludes. Bounds for wors-case blocking under he FMLP are derived in an appendix in he online version of he paper []. Background In his secion, we describe background necessary for discussing he implemenaion of he aforemenioned synchronizaion proocols in LITMUS RT.. Sysem Model In his paper, we consider he problem of scheduling a sysem T of sporadic asks ha share resources upon a muliprocessor plaform consising of m idenical processors. A sporadic ask T i releases a sequence of jobs T j i and is characerized by is wors-case execuion cos, e(t i ), and is period, p(t i ). A job T j i becomes available for execuion a is release ime, r(t j i ), and should complee execuion before is absolue deadline, d(t j i ) = r(t j i ) + p(t i). A ask T i s jobs are ordered by release ime and mus be separaed by a leas p(t i ) ime unis, i.e., j < k r(t j i )+p(t i) r(ti k). On uniprocessors, boh he EDF and he RM policies are commonly used o schedule sporadic ask sysems [9]. Under EDF, jobs wih earlier deadlines have higher prioriy; under RM, asks wih smaller periods have higher prioriy. There are wo fundamenal approaches o scheduling sporadic asks on muliprocessors global and pariioned. Wih global scheduling, processors are scheduled by selecing jobs from a single, shared queue, whereas wih pariioned scheduling, each processor has a privae queue and is scheduled independenly using a uniprocessor scheduling policy (hybrid approaches exis, oo []). Tasks are saically assigned o processors under pariioning. As a consequence, under pariioned scheduling, all jobs of a ask execue on he same processor, whereas migraions may occur in globally-scheduled sysems. A discussion of he radeoffs beween global and pariioned scheduling is beyond he scope of his paper and he ineresed reader is referred o prior sudies [9, 3, 5]. In his paper, we consider only pariioned saic-prioriy (P-SP) scheduling (he use of he FMLP under global and pariioned EDF has been invesigaed previously [6, 9]). X scheduled (no resource) scheduled (wih resource ) l X job release Figure : Legend. blocked (resource unavailable) waiing for response from agen job compleion Under P-SP, each ask is saically assigned o a processor and each processor is scheduled independenly using a saic-prioriy uniprocessor algorihm such as RM. We assume ha asks are indexed from o n by decreasing prioriy, i.e., a lower index implies higher prioriy. We refer o T i s index i as is base prioriy. A job is scheduled using is effecive prioriy, which can someimes exceed is base prioriy under cerain resource-sharing policies (e.g., prioriy inheriance may raise a job s effecive prioriy). Afer is release, a job T j i is said o be pending unil i complees. While i is pending, T j i is eiher runnable or suspended. A suspended job canno be scheduled. When a job ransiions from suspended o runnable (runnable o suspended), i is said o resume (suspend). While runnable, a job is eiher preempable or non-preempable. A newlyreleased or resuming job Tk l can only preemp a scheduled lower-prioriy job T j i if T j i is preempable. Resources. When a job T j i requires a shared resource l, i issues a reques R for l. R is saisfied as soon as T j i holds l, and complees when T j i releases l. R denoes he maximum duraion ha T j i will hold l. A resource can only be held by one job a any ime. Thus, T j i may become blocked on l if R canno be saisfied immediaely. A resource l is local o a processor p if all jobs requesing l execue on p, and global oherwise. If T j i issues anoher reques R before R is complee, hen R is nesed wihin R. In such cases, R includes he cos of blocking due o requess nesed in R. Noe ha no all synchronizaion proocols allow nesed requess. If allowed, nesing is proper, i.e., R mus complee no laer han R complees. An ouermos reques is no nesed wihin any oher reques. Fig. illusraes he differen phases of a resource reques. In his and laer figures, he legend shown in Fig. is assumed. Resource sharing inroduces a number of problems ha can endanger emporal correcness. Prioriy inversion occurs when a high-prioriy job Th i canno proceed due o a lower-prioriy job T j l eiher being non-preempable or holding a resource requesed by Th i. T h i is said o be blocked by T j l. Anoher source of delay is remoe blocking, which occurs when a global resource requesed by a job is already

3 T j i issued R R saisfied, R nesed complee Figure : The differen phases of a resource reques. T j i issues R and blocks since R is no immediaely saisfied. T j i holds R.l for R ime unis. Noe ha R includes blocking incurred due o nesed requess. in use on anoher processor. If he maximum duraion of prioriy inversion and remoe blocking is no bounded, hen iming guaranees canno be given.. Local Synchronizaion Proocols Requess for local resources are arbiraed using uniprocessor synchronizaion proocols. Such proocols are preferable o global proocols (where applicable) because heir worscase blocking delays are generally shorer. In LITMUS RT, we have implemened boh he PCP and he SRP. Noe ha a mos one local proocol can be in use. The PCP and he SRP boh are based on he noion of a prioriy ceiling. The prioriy ceiling of a resource l is he highes prioriy of any job ha requess l. The sysem ceiling (on processor p) is he maximum prioriy ceiling of all (local) resources currenly in use. The sysem ceiling is if none are in use (on processor p). Under he PCP, he sysem ceiling is used o arbirae (local) resource requess direcly. When a job T j i requess a resource, T j i s prioriy is compared o he curren sysem ceiling. If T j i s prioriy exceeds he sysem ceiling (or if T j i holds he resource ha raised he sysem ceiling las), hen he reques is saisfied, oherwise T j i suspends. The PCP also uses prioriy inheriance while a lower-prioriy job Tk l blocks a higher-prioriy job T j i (direcly or indirecly), Tk l s effecive prioriy is raised o (a leas) T j i s effecive prioriy. Noe ha prioriy inheriance is ransiive. Under he SRP, resource requess are always saisfied immediaely. Blocking only occurs on release a job T j i may no execue afer is release unil i s prioriy exceeds he sysem ceiling. Thus, jobs are blocked a mos once and here is no need for prioriy inheriance. (If jobs suspend, hen hey can also block each ime hey resume.) The nesing of local resources is permied under boh he PCP and he SRP. Boh proocols avoid deadlock and bound he maximum lengh of prioriy inversions [4, 7]. This secion is inended as a brief reminder and assumes familiariy wih he discussed proocols. For a full discussion, he ineresed reader is referred o []. Example. In Fig. 3, wo schedules for hree resourcesharing jobs are shown. Inse (a) depics resource sharing under he PCP. T3 issues a reques for R a =, which is saisfied immediaely. This raises he sysem ceiling from o wo. A =, T is released and preemps T3. T requess R a = 4, bu since is prioriy does no exceed he sysem ceiling, i becomes blocked and suspends unil = 6 when R is released, which momenarily lowers he sysem ceiling o. The sysem ceiling is raised o one again when T s reques is saisfied. T arrives a ime 7 and preemps T. T requess R a = 8 and suspends, since he sysem ceiling is sill one. This gives T a chance o reques R (which is saisfied since T raised he sysem ceiling las), o finish is criical secion, and o release boh R and R a ime 9. This allows T o proceed. Finally, all jobs complee in order of prioriy. Inse (b) depics a similar schedule for he same ask sysem under he SRP. Noe ha all blocking has been moved o occur immediaely afer a job has been released. For example, when T is released a =, he curren sysem ceiling is already wo. Thus, T is blocked unil = 4, when he sysem ceiling is lowered o..3 Global Synchronizaion Proocols A global synchronizaion proocol is required if jobs execuing on differen processors may reques a resource concurrenly. In his paper (and in he LITMUS RT kernel), we focus on hree global synchronizaion proocols: he D-PCP, he M-PCP, and he FMLP. The D-PCP and he M-PCP are reviewed nex; he FMLP is discussed in greaer deail in Sec. 3. The D-PCP exends he PCP by providing local agens ha ac on behalf of requesing jobs. A local agen A q i, locaed on remoe processor q where jobs of T i reques resources, carries ou requess on behalf of T i on processor q. Insead of accessing a global remoe resource l on processor q direcly, a job T j i submis a reques R o A q i and suspends. T j i resumes when Aq i has compleed R. To expedie requess, A q i execues wih an effecive prioriy higher han ha of any normal ask (see [, 5] for deails). However, agens of lower-prioriy asks can sill be preemped by agens of higher-prioriy asks. When accessing global resources residing on T i s assigned processor, T j i serves as is own agen. Noe ha, because jobs do no access remoe global resources direcly, he D-PCP is suiable for use in disribued sysems where processors do no share memory. The M-PCP is an exension of he PCP ha relies on shared memory o suppor global resources. In conras o he D-PCP, global resources are no assigned o any paricular processor bu are accessed direcly. Local agens are no required since jobs execue requess hemselves on heir assigned processors. Compeing requess are saisfied in order 3

4 T T T, T, (a) (b) Figure 3: Two example schedules in which hree asks share wo local resources (only iniial jobs shown, deadlines omied). The prioriy ceiling of R is wo, and he prioriy ceiling of R is one. (a) PCP schedule. (b) SRP schedule. of job prioriy. When a reques is no saisfied immediaely, he requesing job suspends unil is reques is saisfied. Under he M-PCP, jobs holding global resources execue wih an effecive prioriy higher han ha of any normal ask. Boh he D-PCP and he M-PCP avoid global deadlock by prohibiing he nesing of global resource requess a global reques R canno be nesed wihin anoher reques (eiher local or global) and no oher reques (local or global) may be nesed wihin R. Example. Fig. 4 depics global schedules for four jobs (T,...,T4 ) sharing wo resources (l, l ) on wo processors. Inse (a) shows resource sharing under he D-PCP. Boh resources reside on processor. Thus, wo agens (A, A 4) are also assigned o processor in order o ac on behalf of T and T 4 on processor. A 4 becomes acive a ime when T4 requess l. However, since T3 already holds l, A 4 is blocked. Similarly, A becomes acive and blocks a ime 4. When T3 releases l, A gains access nex because i is he highes-prioriy acive agen on processor. Noe ha, even hough he highes-prioriy job T is released a =, i is no scheduled unil = 7 because agens and resource holding jobs have an effecive prioriy ha exceeds he base prioriy of T. A becomes acive a = 9 since T requess l. However, T is accessing l a he ime, and hus has an effecive prioriy ha exceeds A s prioriy. Therefore, A is no scheduled unil =. Inse (b) shows he same scenario under he M-PCP. Local agens are no longer required since T and T4 access global resources direcly. T4 suspends a = since T already holds l. Similarly, T suspends a = 4 unil i holds l one ime uni laer. Meanwhile, on processor, T is scheduled a = 5 afer T reurns o normal prioriy and also requess l a = 6. Since resource requess are saisfied in prioriy order, T s reques has precedence over T4 s reques, which was issued much earlier a =. Thus, T4 mus wai unil = 8 o access l. Noe ha T4 preemps T when i resumes a ime 8 since i is holding a global resource. 3 The FMLP under P-SP The flexible muliprocessor locking proocol (FMLP) is a global real-ime synchronizaion proocol ha was recenly proposed by Block e al. [6]. I is inended o overcome shorcomings of prior proocols such as he inabiliy o nes resources and overly pessimisic analysis. Block e al. originally proposed he FMLP for global and pariioned EDF. In his paper, we show how o adap he FMLP o P-SP scheduling. 3. Design Choices The FMLP is based on wo fundamenal design principles flexibiliy and simpliciy. We desire flexibiliy so as o no unnecessarily resric he range of opions available o applicaion designers. We favor simple mechanisms because hey allow us o bound wors-case scenarios more ighly. The laer is especially criical our abiliy o analyze a real-ime sysem is a more imporan han raw performance. Based on hese wo principles, he FMLP was originally designed and adaped for P-SP here by focusing on hree issues ha every global synchronizaion proocol mus address: how o block, how o limi remoe blocking, and how o handle nesed requess. Blocking. When a resource reques canno be saisfied immediaely, he requesing job canno proceed o execue: i is blocked. On a muliprocessor, here are wo ways o handle such a siuaion. The blocked job can eiher remain scheduled and busy-wais unil is reques is saisfied, or i can relinquish is processor and le oher jobs execue while i is suspended. Tradiionally, busy-waiing has mosly been used in scenarios where resources are held only for very shor imes, since busy-waiing clearly wases processing capaciy. (Under he D-PCP and he M-PCP, jobs block by suspending.) However, recen sudies have shown ha, for real-ime sysems, busy-waiing is ofen preferable []. In he ineres of flexibiliy, he FMLP allows boh. In he FMLP, global resources are classified as eiher 4

5 non-preempive execuion A blocked, job spins criical secion A 4 T T T Processor Processor shor long issued saisfied complee blocked, job suspends resumed, bu blocked prioriy boosed criical secion non-preempive execuion T T T 4 T T T 4 T T T 4 (a) (b) (c) (d) Figure 4: Example schedules of four asks sharing wo global resources. (a) D-PCP schedule. (b) M-PCP schedule. (c) FMLP schedule (l, l are long). (d) FMLP schedule (l, l are shor) Processor Processor Processor Processor Processor Processor Figure 5: The phases of shor and long resource requess. shor or long asks busy-wai when blocked on shor resources and suspend when blocked on long resources. Resources are classified by he applicaion designer. However, requess for long resources canno be nesed wihin requess for shor resources. Remoe blocking. When all asks are independen, processors can be analyzed individually (under pariioning). In he presence of globally-shared resources, remoe blocking may occur. As a resul, processors are no longer independen and poenially pessimisic assumpions mus be made o bound wors-case delays. To minimize he impac of remoe blocking, resource-holding jobs should complee heir requess as quickly as possible. The D-PCP and M-PCP expedie he compleion of requess by leing resource-holding jobs (or agens) execue a elevaed prioriies ha exceed normal job prioriies a resource holding job canno be preemped by a job ha does no hold a resource. However, preempions may occur among resource-holding jobs (and agens). The FMLP uses a simplified approach. To minimize he delay a job experiences when resuming, he FMLP booss he prioriy of resuming jobs equally a resource-holding job is scheduled wih effecive prioriy o preemp any non-resource holding job. Conending prioriyboosed jobs are scheduled on a FIFO basis. (Noe ha prioriy boosing was no used in prior FMLP varians.) Addiionally, o avoid delays due o preempions, all requess (boh shor and long) are execued non-preempively, i.e., a job ha execues a reques canno be preemped by any oher job. Noe ha, in he case of shor resources, spinning is carried ou non-preempively, oo. Prioriy boosing is no required for shor resources since requesing jobs do no suspend when blocked. Fig. 5 illusraes he differences beween long and shor requess. Nesing. Nesed resource requess may lead o deadlock and negaively affec wors-case delay bounds. To avoid hese problems, he D-PCP and he M-PCP disallow nes- 5

6 ing (for global resources) alogeher. However, nesing does occur in pracice (albei infrequenly) [8]. The FMLP srikes a balance beween supporing nesing and opimizing for he common case (no nesing) by organizing resources ino resource groups, which are ses of resources (eiher shor or long, bu no boh) ha may be requesed ogeher. Two resources are in he same group iff here exiss a job ha requess boh resources a he same ime. We le G(l) denoe he group ha conains l. Each group is proeced by a group lock, which is eiher a non-preempive queue lock [3] (for a group of shor resources) or a semaphore (for a group of long resources). Under he FMLP, a job always acquires a resource s group lock before accessing he resource. Noe ha, wih he inroducion of groups, he erm ouermos is inerpreed wih respec o groups. Thus, a shor resource reques ha is nesed wihin a long resource reques bu no wihin any shor resource reques is considered o be ouermos. Fig. 6 shows an example wherein seven resources (wo long, five shor) are grouped ino hree resource groups. Noe ha, even hough a reques for l l may conain a reques for l s 7, he wo resources belong o differen groups since one is shor and one is long. 3. Reques Rules Based on he discussion above, we now define he rules for how resources are requesed in he FMLP under P-SP scheduling. We assume ha resources have been grouped appropriaely beforehand, and ha non-preempive secions can be nesed, i.e., if a job eners a non-preempive secion while being non-preempive, hen i only becomes preempable afer leaving he ouermos non-preempive secion. Le T j i be a job ha issues a reques R for resource l. Firs, we only consider ouermos requess. Shor requess. If R is shor and ouermos, hen T j i becomes non-preempable and aemps o acquire he queue lock proecing G(l). In a queue lock, blocked processes busy-wai in FIFO order. R is saisfied once T j i holds l s group lock. When R complees, T j i releases he group lock and leaves is non-preempive secion. Long requess. If R is long and ouermos, hen T j i aemps o acquire he semaphore proecing G(l). Under a semaphore lock, blocked jobs are added o a FIFO queue and suspend. As soon as R is saisfied (i.e., T j i holds l s group lock), T j i resumes (if i suspended) and eners a nonpreempive secion (which becomes effecive as soon as T j i is scheduled). When R complees, T j i releases he group lock and becomes preempive. Prioriy boos. If R is long and ouermos, hen T j i s prioriy is boosed when R is saisfied (i.e., T j i is scheduled l l l l G(l l )=G(l s l) l s 7 l s 4 l s 3 G(l s 7)=G(l s 3) = G(l s 4) l s 6 l s 5 G(l s 6)=G(l s 5) Figure 6: Grouping of wo long resources (l l, l l ) and five shor resources (l s 3,...,l s 7) under he FMLP. If a reques for l may conain a reques for l, hen his is indicaed by a direced edge from l o l. wih effecive prioriy ). This allows i o preemp jobs execuing preempively a base prioriy. If wo or more prioriyboosed jobs are ready, hen hey are scheduled in he order in which heir prioriies were boosed (FIFO). Nesing. Nesing is handled in he same manner for long and shor resources: when a job T j i issues a reques R for a resource l and T j i already holds l s group lock, hen R is saisfied immediaely and no furher acion is aken when R complees. Example. Inses (c) and (d) of Fig. 4 depic FMLP schedules for he same scenario previously considered in he conex of he D-PCP and he M-PCP. In (c), l and l are classified as long resources. As before, T3 requess l firs and forces he jobs on processor o suspend (T4 a = and T a = 4). In conras o boh he D-PCP and he M- PCP, conending requess are saisfied in FIFO order. Thus, when T3 releases l a = 5, T4 s reques is saisfied before ha of T. Similarly, T s reques for l is only saisfied afer T complees is reques a = 7. Noe ha, since jobs suspend when blocked on a long resource, T3 can be scheduled for one ime uni a = 6 when T blocks on l. Inse (d) depics he schedule ha resuls when boh l and l are shor. The main difference o he schedule depiced in (c) is ha jobs busy-wai non-preempively when blocked on a shor resource. Thus, when T is released a = 3, i canno be scheduled unil = 6 since T4 execued non-preempively from = unil = 6. Similarly, T4 canno be scheduled a = 7 when T blocks on l because T does no suspend. Noe ha, due o he wase of processing ime caused by busy-waiing, he las job only finishes a ime 5. Under suspension-based synchronizaion mehods, he las job finishes a eiher ime 3 (M-PCP and FMLP for long resources) or 4 (D-PCP). Local resources. The FMLP can be inegraed wih he SRP. When a job blocks a release ime due o he SRP, i canno have requesed a global resource ye (and hus does no impac he FMLP analysis). Global shor requess can be nesed wihin local requess since jobs do no suspend when blocked on shor resources. However, global long requess canno be nesed wihin local requess since a job mus no 6

7 hold local resources when i suspends. Local requess can be nesed wihin global requess since a ask never blocks on a local reques under he SRP. However, care mus be aken o properly accoun for he ineracion beween he FMLP and he SRP every ime a job resumes, i is subjec o blocking from local resources. Properies. The FMLP avoids deadlock by consrucion, resources wihin a group canno conribue o a deadlock, and he consrain ha long requess canno be nesed wihin shor requess prohibis cyclic nesing of resource groups. Bounds for wors-case blocking under he FMLP are derived in an appendix of he online version of his paper []. 4 Implemenaion Due o space consrains, we are unable o discuss ever deail of each implemened proocol. Insead, we focus on ineresing archiecural issues ha we encounered when designing LITMUS RT. The ineresed reader is referred o [9], which conains a deailed descripion of he LITMUS RT framework and is capabiliies and limiaions, and o LITMUS RT s source code, which is publicly available online [7]. Developed by UNC s real-ime group, LITMUS RT is an exension of Linux ha suppors a variey of real-ime muliprocessor scheduling policies [3]. However, prior o his paper, LITMUS RT did no suppor saic-prioriy scheduling. The conribuion discussed in his paper is he addiion of saic-prioriy scheduling and implemenaions of he PCP, he D-PCP, he M-PCP, and he FMLP (under P-SP) in LITMUS RT. Real-ime Linux. Criics have argued ha, due o inheren non-deerminism in he kernel s archiecure, Linux is fundamenally no capable of providing (hard) real-ime guaranees. In pracice, however, varians of Linux are increasingly being adoped in (sof) real-ime seings [] he predicabiliy of Linux is sufficien for many applicaions mos of he ime. Thus, while no absolue iming guaranees can be given in Linux, i is desirable ha neiher scheduling nor resource sharing are he weakes links in erms of predicabiliy. When implemened in a general-purpose OS, real-ime algorihms face a real-world requiremen ha is ofen glanced over in algorihmic-oriened research hey mus degrade gracefully when faced wih misbehaving applicaions. In a real OS, especially during developmen and esing, jobs may unexpecedly suspend due o page fauls, perform diagnosic logging, accidenally reques wrong resources, fail o properly deallocae resources, and ge suck in non-preempive secions (among many oher possible failures). While real-ime guaranees canno be given for misbehaving jobs, in pracice, (parial) resilience o failure is a very desirable propery for a well-designed OS. We revisi his issue in more deail in he following paragraphs. Real-ime asks. A fundamenal design decision is how he sporadic ask model is mapped ono he Linux process model. In Linux, one or more sequenial hreads of execuion ha share an address space are called a process. There are hree obvious ways o implemen sporadic asks: (i) a sporadic ask is a process, and each job is a hread; (ii) a sporadic ask is a hread, and each job is he ieraion of a loop; and (iii) a sporadic ask is jus a concep, and jobs are he invocaion of inerrup service rouines. Approach (iii), while popular in embedded sysems, suffers from a general lack of robusness and he limiaions ha are imposed on code execuing in kernel space (e.g., absence of floaing poin arihmeic, ec.). Approach (i) suffers from high job release overheads due o forking. This may be alleviaed by recycling hreads by means of a hread pool, bu deermining he maximum number of hreads required in he face of deadline overruns is non-rivial. Approach (ii) limis how deadline overruns can be handled lae jobs canno be easily abored and jobs of he same ask canno be scheduled concurrenly. Noneheless, in LITMUS RT, we chose his approach because i mos closely resembles he familiar UNIX programming model. When sporadic asks are hreads, he quesion arises as o wheher all real-ime asks should reside in he same process. From an efficiency poin of view, a single-process soluion may be beneficial, whereas from a robusness poin of view, address space separaion is clearly favorable. In LITMUS RT, we suppor boh. Resource references. Blocking-by-suspending requires kernel suppor, as does mainaining and enforcing prioriy ceilings and enacing prioriy inheriance. Thus, each resource is modeled as an objec in kernel space, which conains sae informaion such as he associaed prioriy ceiling, unsaisfied requess, ec. (The excepion are shor FMLP resources, which are unknown o he kernel, since hey are realized almos enirely in user space. See [9] for deails.) All asks ha share a given resource mus obain a reference o he same in-kernel objec. Since LITMUS RT is commied o no unnecessarily resricing he applicaion design space, references mus be (ransparenly) obainable across process boundaries. For performance reasons, resource references mus be resolved by he kernel wih as lile overhead as possible. Furher, in a general-purpose OS such as Linux, securiy concerns such as visibiliy of resources and access conrol mus also be addressed he resource namespace mus be managed by he kernel. Prior versions of LITMUS RT simply allocaed a predefined number of resources saically and le real-ime programs refer o objecs by heir offse. While his inerim mehod had low overheads, i was also compleely insecure and brile. Furher, he lack of flexibiliy inheren in saic 7

8 allocaion also quickly proved o be roublesome. As par of he FMLP under P-SP implemenaion effor, we inroduced a new soluion o manage resources in a secure, reliable, and efficien maer. Insead of inroducing a new namespace (which would require appropriae access policies and semanics o be defined), we oped o reuse he filesysem o provide access conrol by aaching LITMUS RT resources a run-ime o inodes (an inode is he in-kernel represenaion of a file). When a ask aemps o obain a reference o a resource, i specifies a file descripor o be used as he naming conex. By specifying he same file, synchronizaion across process boundaries is possible (bu only if allowed by he appropriae permissions). If permied, he kernel locaes he requesed resource and sores is address in a lookup able in he hread conrol block (TCB). Similar o he concep of he file descripor able, he resource lookup able enables fas reference-o-address ranslaion in he performance criical pah of synchronizaion-relaed sysem calls. Wih he new mehod, LITMUS RT resources are creaed dynamically on demand. Prioriy ceilings. I is commonly claimed ha proocols such as he PCP are hard o use in pracice because prioriy ceilings mus be deermined offline and specified manually a runime. However, ha is no he case, as ceilings can be compued auomaically when hreads obain references o resources. The prioriy ceiling of a resource is iniially (INT MAX in pracice) and raised (if necessary) when a real-ime ask obains a reference o i. To ensure correcness, no hread may reques a resource before all asks ha share he resource have obained a reference (for ha resource). Oherwise, he compued ceiling may be incorrec. In pracice, his problem does no occur since i is ensured ha he iniializaion of all real-ime asks is complee by he ime he firs job of any ask is released. A processor s sysem ceiling is mainained as a sack of he local resources ha are currenly in use. Under he SRP, when a ask releases a new job or a job resumes, he kernel checks wheher he ask s prioriy exceeds he prioriy ceiling of he op-mos resource on he sysem ceiling sack (unless he sack is empy). If he job s prioriy does no exceed he ceiling, hen i is added o a per-processor wai queue (a wai queue is a sandard Linux componen used o suspend jobs; see below). When an SRP resource is popped off he sysem ceiling sack, jobs wih prioriies exceeding he new sysem ceiling are resumed. Under he PCP, he op-mos resource s prioriy ceiling is checked every ime a resource is requesed. In our experience, auomaic deerminaion of prioriy ceilings faciliaes ask sysem seup grealy and eliminaes he possibiliy for human error. Prioriy inheriance. Transiive prioriy inheriance, as mandaed by he PCP, requires he kernel o be able o raverse he wai-for dependency graph o arbirary dephs. The necessary sae informaion is kep parially in he TCBs and parially in he resource objecs. When a hread is blocked, he address of he resource is sored in is TCB. Similarly, he address of he holding hread is sored in he resource objec. When a job T j i blocks on a PCP resource l held by T k l (as deermined by he address sored in he resource objec) and T j i has a higher effecive prioriy han Tk l, hen T k l s TCB is updaed o reflec ha i inheris T j i s effecive prioriy. If Tk l is already blocked on anoher PCP resource (as indicaed by is TCB), hen ransiive prioriy inheriance is riggered. Tk l s effecive prioriy is recompued when i releases l by examining all PCP resources ha Tk l holds a he ime of release. Wai queues. In Linux, hreads suspend by enqueuing hemselves in a wai queue, which is a reusable componen used hroughou he kernel. However, he sandard Linux API does no enforce any ordering of blocked hreads. Modificaions were required o ensure sric ordering under he FMLP (FIFO order), and he M-PCP and D-PCP (prioriy order). Under he PCP, each resource has is own wai queue o conrol prioriy inheriance. (The SRP only requires a single wai queue per processor). When a PCP resource is released, all jobs in is wai queue are resumed saicprioriy scheduling ensures ha he highes-prioriy blocked job will proceed nex. This has he grea benefi ha he PCP does no acually require sored prioriy queues. Sha e al. [7] and Rajkumar [5] noe ha an implemenaion of he PCP does no necessarily require per-resource wai queues. Insead, hey propose o keep blocked jobs in he ready queue since he prioriy order will ensure ha hey do no execue premaurely. This may be a valid approach for an OS in a closely conrolled seing (e.g., in embedded sysems), bu for a general purpose OS such as Linux, i is no a sufficienly robus approach. This is because i relies on correc behavior on he par of resource-holding jobs. Wha happens if resource-holding jobs suspend unexpecedly? If blocked asks are kep on he run queue, such an even would allow wo or more jobs o execue in a criical secion a behavior ha is clearly no correc. One migh argue ha in a correc real-ime sysem he resource holding job does no block. However, in real-world sysems such behavior canno be ruled ou. Even a simple prinf saemen, maybe insered for debugging purposes, can lead o (very shor) suspensions. Similarly, an unexpeced page faul due o he omission of disabling demand paging migh also cause a lock-holding ask o suspend. Again, such an even will no occur in a correc real-ime sysem, bu canno be ruled ou compleely (especially during developmen). In he ineres of robusness, a kernel-based muual-exclusion primiive should no rely on he correcness of user-space 8

9 programs. Insead, i should reac as gracefully as possible when facing incorrec applicaions. Aomiciy of resource requess. Since he FMLP requires jobs holding a long resource o be non-preempable (under pariioned scheduling), care mus be aken o ensure ha group lock acquisiion and non-preempiviy are enaced aomically, i.e., if a job were o ener is criical secion in a second sep, hen i could be preemped in he ime beween hese wo evens. The LITMUS RT kernel avoids his race condiion by marking he resource-holding hread as nonpreempable before reurning o user space. D-PCP. Due o he use of local agens, he D-PCP implemenaion differs significanly from he M-PCP and FMLP implemenaions. There are wo approaches for realizing he concep of a local agen: (i) since LITMUS RT suppors exclusively shared-memory archiecures, he requesing hread could be migraed o he processor where he resource resides; or (ii) an addiional hread is provided o serve as he local agen. Since we conjecure ha losing cache affiniy due o a migraion is more expensive han sending a reques, we chose o implemen approach (ii) in LITMUS RT. Noe ha, since only one local agen can execue a any ime, providing a local agen hread for each remoe ask is unnecessary i suffices o provide one local agen hread per address space ha conains global resources. In pracice, we provide a local agen for each resource anyway assuming ha every resource resides in is own address space is always correc and simplifies he implemenaion significanly. Performance comparison. Due o space consrains, we are unable o horoughly compare he implemened synchronizaion approaches. A deailed sudy incorporaing realword overheads is currenly in preparaion [7]. However, o give a rough esimae of relaive performance, Table shows average and maximum observed sysem call overheads, which were recorded on a sysem consising of four Inel Xeon processors clocked a.7 GHz. For each proocol, we measured he reques and release overhead based on over 3, pairs of imesamps ha were recorded jus before and afer he sysem calls of ineres. The wors-case and average overheads were deermined afer discarding he op one percen of daa poins o filer for inerrups and oher noise (similar o he mehodology used in []). Noe ha he sysem was mosly idle during hese measuremens. The obained values hus are only meaningful relaive o each oher, bu do no necessarily reflec a wors-case scenario. We are currenly engaged in experimens o obain more realisic wors-case overheads [7]. Based on hese resuls, we conclude ha local synchronizaion proocols are slighly more efficien o implemen han suspension-based global shared-memory synchronizaion proocols. Of grea ineres are he coss associaed wih Proocol Reques Release SRP.36 (.56).43 (.5) PCP.38 (.49).46 (.5) M-PCP.58 (.66).5 (.59) D-PCP 6.9 (8.8) 5.9 (6.57) FMLP (long).5 (.56).59 (.6) FMLP (shor).9 (.).9 (.9) Table : Average (maximum) overheads encounered for invoking kernel-based synchronizaion proocols. All imes are in µs. he D-PCP. Due o is disribued naure (which requires IPC), is overhead is an order of magniude larger han ha of shared-memory global synchronizaion proocols. This discrepancy makes i unlikely ha he D-PCP is a favorable choice for synchronizaion on shared-memory muliprocessors. However, more deailed sudies are required o obain a definiive answer. 5 Conclusion In his paper, we have exended he FMLP o P-SP scheduling and bounded is wors-case blocking behavior (in he online version of he paper). Furher, we have presened he firs implemenaion ha inegraes he SRP, he PCP, he M-PCP, he D-PCP, and he FMLP in a single framework in a general-purpose OS. We also discussed some of he archiecural design issues ha arise when implemening real-ime synchronizaion proocols in such an OS. We are currenly preparing an exensive performance comparison of he aforemenioned synchronizaion proocols, which will be presened in a companion paper o his work [7]. Lessons learned. In our ongoing work wih Linux and LITMUS RT in paricular, we have come o recognize hree principles ha were no readily apparen o us prior o our implemenaion effors.. Robusness is essenial. Algorihms ha produce mosly correc resuls when faced wih small gliches are always preferable o algorihms ha have superior heoreical performance bu fail caasrophically when assumpions are violaed. In pracice, i is impossible o foresee all possible ineracions in a complex generalpurpose OS such as Linux.. Algorihmic performance dominaes. On our plaform, he impac of non-deerminism inheren in Linux (such as inerrup handlers) is small compared o he impac ha real-ime algorihms have on deerminism inerrups rarely execue for longer han µs. In conras, even a single-quanum prioriy-inversion will delay a hread by (a leas) ms (which is he quanum size in many varians of Linux). Thus, for he vas majoriy of ime-sensiive applicaions ha do no require submillisecond response imes, a lack of proper real-ime 9

10 scheduling and synchronizaion suppor has far greaer consequences han oher sources of OS laency. 3. Design for change. Linux is a fas-moving arge. The rae of change can be overwhelming for an academic research group. When implemening prooypes in Linux, always choose he leas-inrusive implemenaion possible. In our experience, archiecures ha are srucured as a layer of paches work bes. Several ineresing avenues for he fuure presen hemselves. While he FMLP now suppors several major muliprocessor scheduling algorihms, i would beneficial o exend he FMLP o PD [], Earlies-Deadline-unil-Zero- Laxiy [3], and uiliy-based [8] scheduling. Finally, we would like o analyze he impac of mulicore archiecures on he performance of real-ime resoure sharing algorihms. References [] IBM and Red Ha announce new developmen innovaions in Linux kernel. Press release. hp://www-3.ibm.com/ press/us/en/pressrelease/3.wss, 7. [] J. Anderson and A. Srinivasan. Mixed Pfair/ERfair scheduling of asynchronous periodic asks. Journal of Compuer and Sysem Sciences, 68():57 4, 4. [3] T. Anderson. The performance of spin lock alernaives for shared-memory muliprocessors. IEEE Transacions on Parallel and Disribued Sysems, ():6 6, 99. [4] T. Baker. Sack-based scheduling of real-ime processes. Journal of Real-Time sysems, 3():67 99, 99. [5] S. Bisson. Azul announces 9 core Java appliance. hp:// 9-core-java-app liance.hml, 6. [6] A. Block, H. Leonyev, B. Brandenburg, and J. Anderson. A flexible real-ime locking proocol for muliprocessors. In Proceedings of he 3h IEEE Inernaional Conference on Embedded and Real-Time Compuing Sysems and Applicaions, pages 7 8, 7. [7] B. Brandenburg and J. Anderson. A comparison of he M- PCP, D-PCP and he FMLP on LITMUS RT. In preparaion. [8] B. Brandenburg and J. Anderson. Feaher race: A lighweigh even racing oolki. In Proceedings of he Third Inernaional Workshop on Operaing Sysems Plaforms for Embedded Real-Time Applicaions, pages 9 8, 7. [9] B. Brandenburg, A. Block, J. Calandrino, U. Devi, H. Leonyev, and J. Anderson. LITMUS RT : A saus repor. In Proceedings of he 9h Real-Time Workshop, pages 7 3. Real-Time Linux Foundaion, 7. [] B. Brandenburg, J. Calandrino, A. Block, H. Leonyev, and J. Anderson. Synchronizaion on real-ime muliprocessors: To block or no o block, o suspend or spin? In Proceedings of he 4h IEEE Real-Time and Embedded Technology and Applicaions Symposium, 8 (o appear). [] B. Brandenburg and J.Anderson. An implemenaion of he PCP, SRP, D-PCP, M-PCP, and FMLP real-ime synchronizaion proocols in LITMUS RT (exended version). hp:// /anderson/papers.hml. [] J. Calandrino, J. Anderson, and D. Baumberger. A hybrid real-ime scheduling approach for large-scale mulicore plaforms. In Proceedings of he 9h Euromicro Conference on Real-Time Sysems, pages 47 56, 7. [3] J. Calandrino, H., A. Block, U. Devi, and J. Anderson. LITMUS RT : A esbed for empirically comparing real-ime muliprocessor schedulers. In Proceedings of he 7h IEEE Real-Time Sysems Symposium, pages 3, 6. [4] Chia-Mei Chen and Saish K. Tripahi. Muliprocessor prioriy ceiling based proocols. Technical Repor CR-TR-35, Universiy of Maryland, 994. [5] U. Devi. Sof Real-Time Scheduling on Muliprocessors. PhD hesis, Universiy of Norh Carolina, Chapel Hill, NC, 6. [6] P. Gai, M. di Naale, G. Lipari, A. Ferrari, C.Gabellini, and P. Marceca. A comparison of MPCP and MSRP when sharing resources in he Janus muliple processor on a chip plaform. In Proceedings of he 9h IEEE Real-Time And Embedded Technology Applicaion Symposium, pages 89 98, 3. [7] UNC Real-Time Group. LITMUS RT homepage. hp:// anderson/limus-r. [8] E. Jensen, C. Locke, and H. Tokuda. A ime-driven scheduling model for real-ime sysems. In 6h IEEE Real-Time Sysems Symposium, pages, 985. [9] C. Liu and J. Layland. Scheduling algorihms for muliprogramming in a hard real-ime environmen. Journal of he ACM, 3:46 6, 973. [] J.W.S. Liu. Real-Time Sysems. Prenice Hall,. [] J. M. López, J. L. Díaz, and D. F. García. Uilizaion bounds for edf scheduling on real-ime muliprocessor sysems. Real- Time Sysems, 8():39 68, 4. [] SUN Microsysems. SUN UlraSPARC T. Markeing maerial. hp:// 8. [3] X. Piao, S. Han, H. Kim, M. Park, Y. Cho, and S. Cho. Predicabiliy of earlies deadline zero laxiy algorihm for muliprocessor real-ime sysems. 9h Inernaional Symposium on Objec and Componen-Oriened Real-Time Disribued Compuing, 6. [4] R. Rajkumar. Real-ime synchronizaion proocols for shared memory muliprocessors. h Inernaional Conference on Disribued Compuing Sysems, pages 6 3, 99. [5] R. Rajkumar. Synchronizaion In Real-Time Sysems A Prioriy Inheriance Approach. Kluwer Academic Publishers, 99. [6] R. Rajkumar, L. Sha, and J.P. Lehoczky. Real-ime synchronizaion proocols for muliprocessors. Real-Time Sysems Symposium, 988., Proceedings., pages 59 69, 988. [7] L. Sha, R. Rajkumar, and J. P. Lehoczky. Prioriy inheriance proocols: An approach o real-ime synchronizaion. IEEE Transacions on Compuers, 39(9):75 85, 99. [8] S. Shankland and M. Kanellos. Inel o elaborae on new mulicore processor. hp://news.zdne.co.uk/hardware/chips/,39354,39643,.hm, 3.