CS Lab 2: Scheduling

Transcription

1 CS Palmer Dabbelt March 1, 2013 Contents 1 Real-time Scheduling Linux Scheduler Classes Kernel Snapshotting Testing a Real-Time Process with Cucumber Resource Allocation x264 Case Study Dynamic Resource Fairness DRF Work Queues Distributed Code realtime/cbs.[ch] realtime/dfs,q.[ch] realtime/rt,ctl.c webvideo Tasks Kernel Interface (0 pts) CBS Scheduler Class (30 pts) Kernel Snapshots (20 pts) Web Server Modifications (5 pts) Dynamic Resource Fairness (25 pts) Schedule 10 1

2 For this assignment you will be implementing improvements to the Linux scheduler. First you ll be implementing a simple real-time scheduler, and second you ll be implementing a fair resource allocation scheme. It s important to note that we will be using your HTTP server to launch these real-time processes. In order to attempt to break the dependency chain as much as possible, none of the performance features implemented for the previous assignment will be necessary here. To successfully test your scheduler you ll just need to be able to launch CGI scripts in parallel. If any group feels their implementation from the previous lab isn t up to stuff then please come and talk with me early and we ll try to work something out. 1 Real-time Scheduling The first section of this lab consists of adding a real-time scheduler to Linux. You ll be taking advantage of Linux s support for scheduler classes in order to cleanly implement your new scheduler. For this assignment you will be implementing an extension to the EDF algorithm that allows you to mix real-time tasks with what the paper refers to as constant bandwidth servers, which are simply processes that get a fixed amount of CPU bandwidth.. The CBS paper availiable on the course website, you should read it before proceeding further We will be taking advantage of the fact that the CBS algorithm has a natural method of determining the amount of slack present in the system. The paper uses this to criteria to determine if a given set of tasks is schedulable, but the math works out such that we can also determine the largest allocation that we can give CFS in order to mantain the guarantees of CBS. 1.1 Linux Scheduler Classes The simplest way to add CBS support to Linux is to implement a new scheduler classes. It s important to note that Linux scheduler classes aren t loadable at runtime (there s no technical reasons this can t happen, it s just that nobody has bothered to implement it). This means you ll need to decide upon your scheduler hierarchy at compile time, which isn t actually a problem because CBS has an clean way to assign the slack to other schedulers. Linux defines scheduler classes as a simple integer value that defines the scheduling algorithm that decides when to run a given process. Schedulers in Linux have a fixed priority: the highest priority scheduler gets to run all its jobs to completion before the next highest priority scheduler gets to run any jobs. Your new CBS scheduler should be the highest priority scheduler in the system, producing the ordering of schedulers shown in Figure 1. CBS Realtime CFS Idle Figure 1: Scheduler Priority Order In order to make this happen you ll first need to do a bit of build work such that you can install your scheduler beside CFS reference Lab 0 for how to do this. You ll then need to add a scheduler identifier along side the other identifiers inside linux/include/uapi/linux/sched.h. You will need to modify struct task struct in order to keep track of the necessary accounting data you ll be using to implement CBS. Additionally, you ll need to define a CBS-specific run queue look at struct cfs rq inside linux/kernel/sched/. In order to actually get your code to be run, you ll need to add some code sched init that initializes your run queue. At this point you can create a new scheduler class that contains a set of function pointers 2

3 that actually consist of your implementation of CFS (look at the other struct sched class entries in linux/kernel/sched/sched.h). Be sure to set your scheduler as the highest priority scheduler inside linux/kernel/sched/sched.h with the sched class highest macro. At this point you re effectively done with the overhead involved in teaching Linux how to recognize your scheduler. 1.2 Kernel Snapshotting As you probably discovered during lab 0, testing realtime processes is inherently difficult because any code you use for instrumentation will inherently break your timings. In order minimize this breakage, we ve attempted to design a general but efficient API that allows you to interogate the scheduler without too much overhead. We ve chosen to use snapshots in order to achieve this. The snapshoting algorithm is defined in realtime/snapshot.h. The idea is that you pass an array of (event, trigger) pairs into the kernel. The data is passed into the kernel using the snapshot() system call. When the user calls snapshot() the kernel will first empty all of its snapshot buffers. Every time the kernel sees an event that causes the trigger to fire, it fills up the next snapshot buffer with the relevant information. This will hopefully be a very fast operation, as all the buffers will be staticly allocated. The user can then go ahead and poll these buffers at their leisure without breaking the hard timing path in the kernel. This is probably best worked through as an example. The idea is that a scheduling policy looks at the current state of runnable processes and determines which one should be running. A correct scheduler will always make the correct decision as to who to schedule next, so if we sample the decisions a scheduler makes and they re all correct then we can have a reasonable certainty that the scheduler is correct. An example of what the snapshot() system call produces can be seen in Figure 2. In this case this would be an invalid run of the CBS scheduler because the wrong process was chosen to run next it should be B, as B has the earliest deadline. History Cur Next Queue /proc/snapshot/0 A 5ms B 10ms C 8bs A 5ms C 3ms B A C D +10ms +20ms +30ms BLK F BLK History Cur Next Queue /proc/snapshot/1 B 10ms C 8bs A 5ms C 3ms B A C 3ms +17ms +27ms D BLK F BLK B +10ms Figure 2: The result of calling snapshot({ CBS, CBS}, { BEFORE, AFTER}, 2) on a broken In order to get the data out of the kernel, you will be implementing a /proc interface. You should create the directory /proc/snapshot and populate it with SNAP MAX TRIGGERS files named from 0 to SNAP MAX TRIGGERS 1. Each one of these files will represent a snapshot buffer that the user has access to. Reads from these virtual files will return formatted data to the user so they can parse it. 3

4 Note that you must implement the functions in realtime/cbs proc.h, which are what we ll be using to implement our /proc interface to get data out. Additionally, you must setup you build such that realtime/cbs proc impl.c gets linked into your kernel image (again, that s how we ll be implementing the /proc interface). There are examples of how to cleanly do this in lab Testing a Real-Time Process with Cucumber The end goal of the snapshotting interface is to allow you to test CBS using Cucumber. There are two important properties of the CBS algorithm that you will need to test: that real-time tasks are being scheduled EDF, and that CBS tasks end up with the correct compute bandwidth in aggregate. The snapshotting interface we ve specified allows you to test both of those constraints. It s probably best to take a look at exactly how data is going to move into and out of the VM. Figure 3 shows how the test system is laid out and the interface between each module QEMU Cucumber httpd CGI Kernel CBS Model Tasks Steps realtimectl read() /proc Valid cbs_proc.h snapshot() Snapshot fork()/exec() Triggers realtime cbs_create() CBS Figure 3: Moving data from CBS to Cucumber You should already have the HTTP server working, and we have provided you with realtime and realtimectl. You ll want to begin by writing some Cucumber tests that describe the behavior of the system. An example of a behavior of CBS is shown in Figure 4. Most of these lines are fairly straight-forward: process creation and snapshoting translate directly into some calls to realtimectl. The statement A should always be scheduled before B, however, is interesting. Effectively this means that your Cucumber code must have a model of CBS that understands when processes should be executed. In this specific case it s simple because all the threads have the same execution period, but in the arbitrary case it ll be significantly more complicated. Effectively you need a model of CBS that, given a snapshot, is capable of determining if the correct process is being executed. This will involve looking at the entire list of queued processes and determining that your kernel implementation of CBS has picked the correct process to schedule. 4

5 When the following processes are started ID Type CPU Peroid A RT B BW C BW And the system executes for 10 seconds When any snapshot is taken Then C should recieve 20% of the CPU time And B should recieve 9% of the CPU time And A should always be scheduled before B And A should always be scheduled before C Figure 4: A sample behavior of CBS 2 Resource Allocation The fundamental concept that underlies UNIX systems is the idea of resource virtualization. UNIX systems provide the programmer with an infinate number of threads of execution, each of which has access to an infinate amount of memory. The operating system uses time multiplexing in order to provide the appearance of having more resources than it actually does. Providing the abstraction of infinite resources makes programming UNIX systems easy because the programmer doesn t have to explicitly manage resources. These abstractions made sense on traditional UNIX systems because they had only a single processor. The key here is that it s impossible for the user to distinguish between getting all of the time on a slow machine and getting some of the time on a faster machine. For the past few years, modern hardware has been increasing in performance by increasing the amount of parallelism exposed to the user. This has the unfortunate consequence that we no longer get perfect scaling: there is additional overhead involved in synchronizing work between threads that the programmer has to assume can be run asynchronusly but are actually just multiplexed onto a few CPUs. 2.1 x264 Case Study An example application that demonstrates the problems behind the UNIX programming model is the x264 video encoder. Like most video encoders, x264 is a very finely tuned program on amd64 x264 is actually written in hand-tuned assembly, which provides about a 5x speedup over C! Depending on exactly which options you use, x264 can be very computationally expensive. From just using x264 s default set of options, I can get encode speeds ranging from 5 FPS to 500 FPS. As is standard for video encoding codes, there is a tradeoff between encoding speed and output quality. The x264 encoder has been parallelized to run on an arbitrary number of CPUs. As the H.264 standard (x264 is an implementation of H.264) was designed for parallel implementations, it turns out that it s very easy to parallelize x264. Figure 5a shows the result of the same x264 computation being spread over multiple threads. The machine used was running Linux and was pretty much idle. You can see how the encode speeds up nicely as more CPUs are used. As is standard for parallel implementations, x264 doesn t get a perfect parallel scaling: if n CPUs produces f(n) FPS, then 2n CPUs produces less than 2f(n) FPS. There are many reasons for non-linear scaling on multiprocessor machines. At least one reason is that extra work is required to synchronize threads that aren t actually running in parallel. Figure 5b 5

6 shows what happens when we attempt to run an x264 encoding instance that s bound to a single CPU but is running in multi-threaded mode. There is a fairly significant performance hit that results from running more threads than CPU instances Frames per Second Frames per Second Threads Threads (a) Multiple CPUs (b) Single CPU Figure 5: Parallel x264 video encoding 2.2 Dynamic Resource Fairness In order to resolve the issues that stem from attempting to provide a UNIX enviornment highly tuned parallel applications we ll have to look at the entire streaming system. Your web server will look something like Figure 6 when it is processing streaming video: clients connect to the web server which then launches CGI scripts to encode the video. The output from those CGI scripts is routed back to the clients through the web server. httpd Client kbps Encode 1 Client kbps Encode 2 Client kbps Encode 3 Stats Best Effort Stats Figure 6: Anatomy of a HTTP video encoding server The problem is that each CGI instance has no way to inform the other CGI instances of how many resources they re using. Effectively this boils down to the fact that UNIX has a very simple resource allocation scheme: all processes are allowed to consume an infinate amount of resources and the operating system is required to transparently virtualize these resources as best as it possibly can. This virtuilization can have a serious effect on performance, as shown in the case study about x264. 6

7 One better method of resource allocation in specified in the DRF paper. DRF stands for Dominant Resource Fairness. The general idea is that all processes specify a resource vector. Resource vectors specify the amount of resources that are required to perform a single unit of computation. The operating system is then able to ensure that a fair and efficient resource allocation across the machine. More information is availiable in the DRF paper DRF Work Queues The most interesting part of DRF is the ability to dynamically change the number of threads working on a system in order to ensure that fairness is mantained. We need two mechanisms to achieve this: the ability to increase the number of processing threads and the ability to decrease the number of processing threads. In order to increase the number of threads availiable for processing we can simply call the provided work func function another time to create another worker thread. In order to decrease the number of threads availiable for processing a thread can simply be killed asynchronously. Allowing the kernel to asynchronusly create and kill threads adds a lot of complexity to userspace programs. In the standard UNIX resource model programmers only have to worry about arbitrary execution orderings of threads, but using this DRF API the programmer also has to ensure that the program executes correctly even when threads of execution can terminate at any point. A side effect of this new constraint is that it is no longer safe to use locks anywhere in code that s managed by DRF. An example parallel application in shown in Figure 7. This application uses locks to partition out work to multiple threads, ensuring that each thread computes a different value in the table. Unfortunately, this program breakes horribly when we allow threads to arbitrarily terminate. If execution is terminated within a critical section then the program will deadlock. If execution is terminated while the computation is running, then a unit of work can be missed (one of the entries in blocks might not get written). #define MAX_BLOCKS 10 int blocks[max_blocks]; int get_block(void) { static int cur = 0; static lock_t lock = UNLOCKED; int out; lock(&lock); out = cur++; unlock(&lock); } return (out < MAX_BLOCKS)? out : -1; void thread_main(void) { int block; while ((block = get_block())!= -1) blocks[block] = pow(10, block); } Figure 7: A sample UNIX parallel application In order to solve these two problems we re going to have you implement a new synchronization 7

8 construct that I m calling DRF work queues that directly solves the problem of handing out work to multiple threads that may or may not actually finish their transactions. Figure 8 shows the original application after being ported to use this new synchranization construct. #define MAX_BLOCKS 10 int blocks[max_blocks]; drfq_t queue; int main(void) { drfq_create(&queue, MAX_BLOCKS); } void thread_main(void) { int block; while ((block = drfq_request(&queue))!= -1) { blocks[block] = pow(10, block); drfq_commit(&queue, block); } } Figure 8: A sample DRF Queue parallel application 3 Distributed Code We ve distributed some code along with this assignment that s designed to get you started. All this code should be availiable inside the class s public git repo, in the master branch. The vast majority of the code provided to you lives inside the realtime folder in your git repo. 3.1 realtime/cbs.[ch] These files contain the CBS API that you will have to implement for this assignment along with a userspace implementation of those APIs. Note that the userspace implementation doesn t actually do CBS, it s just the minimal amount of code required in order to actually make code that uses the CBS APIs execute. The userspace CBS implementation provides no guarantees about execution time at all. You can easily see this: start up a few CPU intensive processes and your real-time tasks will start missing their deadlines. This would be a good comparison for you to make against your proper CBS application: how much background CPU throughput can you get while still hitting your deadlines? 3.2 realtime/dfs,q.[ch] This provides an implementation of both DFS and DFS queues entirely in userspace. This implementation is layered on top of both CBS and pthreads. Much like CBS, this is just the minimal amount of code required to make things work. This code doesn t actually do any dynamic resource allecation, for example, it just always runs the specified maximum number of threads. Again, this could make for some decent comparisons with your in-kernel implementation. 3.3 realtime/rt,ctl.c This contains a pair of applications that are designed to allow you to verify that your real-time deadlines are met. They work as a pair: realtimectl is a CGI application that forks to execute realtime. 8

9 Additionally, realtimectl is able to use the snapshot() interface to get data out of the running kernel and into Cucumber. This allows you to leverge your HTTP server to easily test your scheduler, and as such should probbaly be the first application you take a look at. 3.4 webvideo webvideo contains a program that is designed to implement some of the same computational patterns as a video codec. This application already uses our DFS and CBS APIs, so it should require no modification to run. That said, it will probably be useful to extend webvideo to allow you to cover more interesting edge cases. 4 Tasks 4.1 Kernel Interface (0 pts) A major part of this assignment will be deciding what functionality you want to implement in the kernel and what functionality you want to implement in userspace. The goal here is to make you do a similar design decision as to what went into Linux s futexes: do the stuff that s impossible to do in userspace in the kernel, but keep everything else in userspace. As a hint, you probably don t want to add a cbs create() system call but should instead write that as a library routine that calls fork() and sched setscheduler(). If you pick a clean implementation boundary then you ll end up with a significantly easier assignment. This isn t worth any points, I just wanted to list it first so you sit down with your team and hash out the kernel interface you re going to provide. This should definately be a major component of your initial design document. 4.2 CBS Scheduler Class (30 pts) Implementing CBS in the kernel will probably be the largest piece of code you re implementing for this assignment. Luckily it should be pretty straight-forward: just follow the tutorial to learn where to modify the kernel, and then follow the CBS algorithm for scheduling. There are two interesting edge cases for CBS that aren t mentioned anywhere else CBS kills real-time threads by passing SIGXCPU to them when they ve exceeded their processing time limet. CBS must yield the remainder of its time to CFS. 4.3 Kernel Snapshots (20 pts) In order to test your CBS implementation you will have to implement the kernel snapshotting mechanism that was described earlier in the document. The snapshot API is contained in realtime/snapshot.h. Be sure to also implement the compatability layer for our /proc filesystem, which is located at realtime/cbs proc.[ch]. 4.4 Web Server Modifications (5 pts) You will need to modify how you handle CGI applications. Your previous CGI implementation needed to buffer the entire CGI response in order to handle error codes returned by the CGI application. Buffering the entire response breaks the live aspect of live video streaming, so it s not going to work for this assignment. The provided CGI video streaming application will provide the header X-Buffering: Streaming 9

10 to signify that the server should not buffer the response and should instead transmit data to the client directly as in comes out of the CGI script instead of waiting for the CGI script to finish. You can assume that all CGI scripts that run in streaming mode return successfully. This means you don t have to worry about checking the error code of streaming CGI scripts. Additionally, you can assume that all CGI scripts that run in streaming mode won t specify any cache-related headers. This means you can bypass your cache for streaming responses. 4.5 Dynamic Resource Fairness (25 pts) Assuming you pick your userspace / kernel barrier correctly, implementing DRF should be pretty straight-forward. Just be sure to create CBS-class CBS threads for DRF and you should be OK. Implementing DRF work queues will be a bit tricky: you ve got to do this without any userspace locks so you can avoid the mentioned race conditions. Note that you also can t use something like pthread cancel() because that s akin to allowing the user to turn off interrupts. 5 Schedule The schedule for this assignment is the same as for the previous assignment: the project is three weeks long, and at the half way point we will be requiring a design document. 10