Scalable Linux Scheduling (Paper # 9)

Transcription

1 Scalable Linux Scheduling (Paper # 9) Scalability: the ability to support large number of threads. 30% of the total CPU time is spent in the scheduler when the number of running threads is high. Linux scheduler use expensive and redundant algorithm for task selection. How to improve?

2 Linux Thread Model Kernel thread and user-level thread Many established operating systems support many-to-one or manyto-many thread models. In such model, a kernel-level thread has many user-level threads mapped to it. A secondary scheduler chooses which of the mapped user level threads to run. The multi-tie scheduling approach of these models assures a manageable number of kernel threads. Linux use a one-to-one model: every user-level thread has its own kernel thread. One-to-one mapping usually generate too many kernel level thread. Linux scheduler evaluate each runnable threads to pick the bestone

3 Many of the calculation is simple repeating of its last evaluation. The computation proportionally increases at the number of kernel-level threads.

4 Current Default Linux Scheduler Task Structure: maintaining a task s address space information, memory management, file descriptors, and other information. volatile long state blocking, sleeping... unsigned long policy FIFO, round robin, or other long counter remaining time of current quantum, 10ms tick long priority 1 to 40, long rt priority 1 to 99 struct list head run list int has cpu 1 when is executing int processor on which processor it is executing Table 1: Task structure Run Queue: a circular linked list containing all tasks in the running state. The scheduler traverses this list when it looks for a task to run. The list is not sorted.

5 when the scheduler finds a tie in priority, the one closer to the front of the list is chosen. New task or awakened tasks are placed at the head. schedule(): THE kernel function do the actual scheduling. called from over 500 places within the kernel. is called when a task yields the processor, blocks for I/O, expires its quantum, or is preempted by another (higher priority) task. Goodness calculation: For FIFO or RT tasks, 1000+rt priority. For others, =0, if counter=0 =counter+priority, else 15 points bonus if last run was on the current processor (processor affinity bonus).

6 schedule() in kernel * * Note! there may appear new tasks on the run-queue during this, as * interrupts are enabled. However, they will be put on front of the * list, so our list starting at "p" is essentially fixed. */ /* this is the scheduler proper: */ { int c = -1000; next = idle_task; while (p!= &init_task) { if (can_schedule(p)) { int weight = goodness(p, prev, this_cpu); if (weight > c) c = weight, next = p; } p = p->next_run; } } } /* Do we need to re-calculate counters? */ if (!c) { struct task_struct *p; read_lock(&tasklist_lock); for_each_task(p) p->counter = (p->counter >> 1) + p->priority; read_unlock(&tasklist_lock); }

7 Performance Evaluation with Java Thread (paper # 10) Threads are an essential part of programming in Java: Java language lacks an interface for non-blocking I/O. Multi-thread are especially necessary in communication intensive applications. Typically at least one or more thread are created for each communication stream. Java thread implementation: Java thread is defined in the context of Java Virtual Machine, it can be mapped to kernel thread in different ways. Some Java VM use many-to-one mapping, such as Blackdown JDK. Java threads are not visible to kernel scheduler Blocking calls have to be trapped by the Java run environment, and go through a user-level scheduler in the Java VM. Otherwise, it will block other runnable java threads.

8 This is also called user-level thread support, it provides a multithread programming even the OS does not support thread. Not able to take full advantage of multiple processor. IBM Java VM uses a one-to-one mapping. Benchmark: VolanoMark Simulation of chat service. Simulating parameters: chat room number and number of messages per user. Each simulated user establishes a socket connection to the server, and exchange messages with the server. The number of Java thread vary from 400 to Key statistic: message throughput. Benchmark results: Up to 24% reduction in message throughput. Amount of kernel time spent in schedule is from 30% to 50%. Highest run queue length is 414. Proposed improvement by IBM:

9 Reduce the calculation cost for goodness() function reduce the cache misses during the goodness calculation. average cycle for goodness() reduced from 167 CPU cycles to 108 cycles. message throughput increased by 7%.

10 Improvement Proposed by Paper #9: ELSC Scheduler Goals: Keep changes local to the scheduler. Keep the concept and implementation simple. Behave like the current scheduler as much as possible. Maintain existing performance for light loads, and scale gracefully under heavy loads. ELSC keeps the run queue in a sorted order to make a quick scheduling decision. To reduce cost associated with sorting, ELSC uses a table based structure instead of linked list. Observation of the goodness calculation: Static goodness: a task s counter and priority does not change if a task is on the run queue but not running on a processor.

11 Dynamic goodness: related to memory map and processor affinity depends on which task and processor are calling schedule().

12 Implementation of ELSC Use an array of 30 double linked list as runnable queues. Each list in the array is used to hold tasks in a certain static goodness range. List head (a) Run queue for default linux scheduler List head List head List head (a) Run queue for ELSC scheduler queue array manipulation: Figure 1: Run queue table

13 add to runqueue(): put a task into a proper list according to its priority value. Queue array searching algorithm Try to limit its search to one list in its table Worst case: all threads are put in the same queue. Limit the number of tasks examined in each list to half the number of processors in the system plus 5. The rest of the list is not considered, as all tasks in the list have about the same static goodness. For real-time tasks, the search is much simpler, and the ELSC simply run the task with the highest rt priority value. Might schedule a task that is not highest in priority. Performance results

14 Scheduler in Kernel Multiple runqueue, each per processor. Load balancing consideration. Each runqueue contain a table of list. The size of table is maximum priority value. Priority value only calculated whenever necessary, thus there are no recalculation of priority value in schedule(). Select the first task of highest priority Fast searching algorithm: how many instruction needed to find the location of a given priority?

15 static inline int effective_prio(task_t *p) { int bonus, prio; /* * Here we scale the actual sleep average [0... MAX_SLEEP_AVG] * into the bonus/penalty range. * * We use 25% of the full priority range so that: * * 1) nice +19 interactive tasks do not preempt nice 0 CPU hogs. * 2) nice -20 CPU hogs do not get preempted by nice 0 tasks. * * Both properties are important to certain workloads. */ bonus = MAX_USER_PRIO*PRIO_BONUS_RATIO*p->sleep_avg/MAX_SLEEP_AVG/100 - MAX_USER_PRIO*PRIO_BONUS_RATIO/100/2; } prio = p->static_prio - bonus; if (prio < MAX_RT_PRIO) prio = MAX_RT_PRIO; if (prio > MAX_PRIO-1) prio = MAX_PRIO-1; return prio; asmlinkage void schedule(void) {... array = rq->active;... idx = sched_find_first_bit(array->bitmap); queue = array->queue + idx; next = list_entry(queue->next, task_t, run_list);...

16 } static inline int _sched_find_first_bit(unsigned long *b) { if (unlikely(b[0])) return ffs(b[0]); if (unlikely(b[1])) return ffs(b[1]) + 32; if (unlikely(b[2])) return ffs(b[2]) + 64; if (b[3]) return ffs(b[3]) + 96; return ffs(b[4]) + 128; }