Take charge of processor affinity

Transcription

1 English Sign in (or register) Technical topics Evaluation software Community Events Take charge of processor affinity Why (three reasons) and how to use hard (versus soft) CPU affinity Eli Dow, Software Engineer, IBM Linux Test and Integration Center Summary: Knowing a little bit about how the Linux 2.6 scheduler treats CPU affinity can help you design better userspace applications. Soft affinity means that processes do not frequently migrate between processors, whereas hard affinity means that processes run on processors you specify. This article describes current affinity mechanisms, explains why and how to use hard affinity, and provides sample code showing you how to use the available functionality. Date: 29 Sep 2005 Level: Intermediate Comments: 2 (View Add comment - Sign in) Rate this article Average rating (67 votes) Simply stated, CPU affinity is the tendency for a process to run on a given CPU as long as possible without being moved to some other processor. The Linux kernel process scheduler inherently enforces what is commonly referred to as soft CPU affinity, which means that processes generally do not frequently migrate between processors. This state is desirable because processes that seldom migrate often incur less overhead. The 2.6 Linux kernel also contains a mechanism that allows developers to programmatically enforce hard CPU affinity. This means your applications can explicitly specify which processor (or set of processors) a given process may run on. What is Linux kernel hard affinity? In the Linux kernel, all processes have a data structure associated with them called the task_struct. This structure is important for a number of reasons, most pertinent being the cpus_allowed bitmask. This bitmask consists of a series of n bits, one for each of n logical processors in the system. A system with four physical CPUs would have four bits. If those CPUs were hyperthread-enabled, they would have an eight-bit bitmask. If a given bit is set for a given process, that process may run on the associated CPU. Therefore, if a process is allowed to run on any CPU and allowed to migrate across processors as needed, the bitmask would be entirely 1s. This is, in fact, the default state for processes under Linux. The Linux kernel API includes some methods to allow users to alter the bitmask or view the current bitmask: sched_set_affinity() (for altering the bitmask) sched_get_affinity() (for viewing the current bitmask) Note that cpu_affinity is passed on to child threads,so you should place calls to the sched_set_affinity appropriately. Why should you use hard affinity? Normally the Linux kernel does a good job of scheduling processes to run where they should (that is, running on available processors and obtaining good overall performance). The kernel includes algorithms for detecting skewed workloads across CPUs, enabling process migration to less busy processors. As a rule of thumb, you should simply use the default scheduler behaviors in your applications. However, you might want to alter these 1/8

2 default behaviors to optimize performance. Let's look at three reasons for using hard affinity. Reason 1. You have a hunch Hunch-based scenarios come up so often in scientific and academic computing that they are no doubt applicable to public-sector computing as well. A common indicator is when you know intuitively that your application will need to consume a lot of computational time on multiprocessor machines. Reason 2. You are testing complex applications Testing complex software is another reason to be interested in the kernel affinity technology. Consider an application that requires linear scalability testing. Some products claim to perform better with a throw-more-hardware-at-it mantra. Rather than just purchasing multiple machines (a machine for each processor configuration), you can: Purchase a single multiprocessor machine Incrementally allocate processors Measure your transactions per second Plot the resulting scalability If your application truly does scale linearly with CPU additions, a plot of transactions-per-second versus number of CPUs should yield a linear relationship (such as a straight diagonal graph -- see the next paragraph). Modeling behavior this way can indicate whether your application can use the underlying hardware efficiently. Amdahl's Law Amdahl's Law governs the speedup of using parallel processors on a problem versus using only one serial processor. Speedup is the time it takes a program to execute in serial (with one processor) divided by the time it takes to execute in parallel (with many processors): T(1) S = T(j) Where T(j) is the time it takes to execute the program when using j processors. Amdahl's Law states this probably won't happen in reality, but the closer the better. For the general case, we can deduce that every program will have some sequential component. As problems sets get larger, the sequential component eventually places an upper limit on the optimal solution time. Amdahl's Law is especially important when you want to keep the CPU cache hit rate high. If a given process gets migrated, it loses the benefits of the CPU cache. In fact, if the CPU you are using needs to cache some specific piece of data for itself, all other CPUs invalidate any entry (for that data) from their own cache. So, if multiple threads need the same data, it might make sense to bind them to a particular CPU to ensure they all have access to the cached data (or at least improve the odds of a cache hit). Otherwise, the threads might execute on different CPUs and constantly invalidate each other's cache entries. Reason 3. You are running time-sensitive, deterministic processes A final reason to be interested in CPU affinity is for real-time (time-sensitive) processes. For example, you might wish to use hard affinity to specify one processor on an eight-way machine, while allowing the other seven processors to handle all the normal scheduling needs of the system. This action ensures that your long-running, time-sensitive application gets run, and also allows other/another application(s) to monopolize the remaining computing resources. The following sample application shows how this all works. 2/8

3 How to code hard affinity Let's devise a program to make a Linux system very busy. You can construct this program using the system calls mentioned previously along with some other APIs that indicate how many processors are on the system. In essence, the goal is to write a program that can make each processor in a system busy for a few seconds. Download the sample application from the "Download" section below. Listing 1. Keeping the processors busy /* This method will create threads, then bind each to its own cpu. */ bool do_cpu_stress(int numthreads) int ret = TRUE; int created_thread = 0; /* We need a thread for each cpu we have... */ while ( created_thread < numthreads - 1 ) int mypid = fork(); if (mypid == 0) /* Child process */ printf("\tcreating Child Thread: #%i\n", created_thread); break; else /* Only parent executes this */ /* Continue looping until we spawned enough threads! */ ; created_thread++; /* NOTE: All threads execute code from here down! */ As you can see, the code simply creates a bunch of threads by forking. Each executes the remaining code in the method. Now let's have each thread set affinity to its own CPU. Listing 2. Setting CPU affinity for each thread 3/8

4 cpu_set_t mask; /* CPU_ZERO initializes all the bits in the mask to zero. */ CPU_ZERO( &mask ); /* CPU_SET sets only the bit corresponding to cpu. */ CPU_SET( created_thread, &mask ); /* sched_setaffinity returns 0 in success */ if( sched_setaffinity( 0, sizeof(mask), &mask ) == -1 ) printf("warning: Could not set CPU Affinity, continuing...\n"); If the program executed thus far, our threads would be set with their individual affinity. The call to sched_setaffinity sets the CPU affinity mask of the process denoted by pid. If pid is zero, then the current process is used. The affinity mask is represented by the bitmask stored in mask. The least significant bit corresponds to the first logical processor number on the system, while the most significant bit corresponds to the last logical processor number on the system. Each set bit corresponds to a legally schedulable CPU, while an unset bit corresponds to an illegally schedulable CPU. In other words, a process is bound to and will run only on processors whose corresponding bit is set. Usually, all bits in the mask are set. The CPU affinity of each of these threads is passed on to any children forked from them. Note that you should not alter the bitmask directly. You should use the following macros instead. Though not all were used in our example, they are listed here in case you need them in your own program. Listing 3. Macros to indirectly alter the bitmask void CPU_ZERO (cpu_set_t *set) This macro initializes the CPU set set to be the empty set. void CPU_SET (int cpu, cpu_set_t *set) This macro adds cpu to the CPU set set. void CPU_CLR (int cpu, cpu_set_t *set) This macro removes cpu from the CPU set set. int CPU_ISSET (int cpu, const cpu_set_t *set) 4/8

5 This macro returns a nonzero value (true) if cpu is a member of the CPU set set, and zero (false) otherwise. For our purposes, the sample code will go on to have each thread execute some computationally expensive operation. Listing 4. Each thread executes a compute-intensive operation /* Now we have a single thread bound to each cpu on the system */ int computation_res = do_cpu_expensive_op(41); cpu_set_t mycpuid; sched_getaffinity(0, sizeof(mycpuid), &mycpuid); if ( check_cpu_expensive_op(computation_res) ) printf("success: Thread completed, and PASSED integrity check!\n", mycpuid); ret = TRUE; else printf("failure: Thread failed integrity check!\n", mycpuid); ret = FALSE; return ret; There you have the basics of setting CPU affinity for 2.6 Linux kernels. Let's wrap this method call with a fancy main program that takes a user-specified parameter for how many CPUs to make busy. We can even use another method to determine the number of processors in the system: int NUM_PROCS = sysconf(_sc_nprocessors_conf); This method lets the program make wise decisions about how many processors to make busy, such as spinning all by default and allowing users to specify something only in the range of actual processors available on the system. Running the sample application 5/8

6 When you run the sample application described above, you can use a variety of tools to see that the CPUs are busy. For simple testing, use the Linux command top. Press the "1" key while running top to see a per-cpu breakdown of executing processes. Conclusion The sample application, although trivial, shows you the basics of hard affinity as implemented in the Linux kernel. (Any application using this code sample will no doubt do something much more interesting.) At any rate, with a basic understanding of the CPU affinity kernel API, you are in position to squeeze every last drop of performance out of complicated applications. Download Description Name Size Download method Sample app using CPU affinity kernel API thrasher.zip 3 KB HTTP Information about download methods Resources Learn "Hyper-Threading speeds Linux" (developerworks, January 2003) gives the results of an investigation into the effects of hyperthreading on the Linux SMP kernel. "Identify performance bottlenecks with OProfile for Linux on POWER" (developerworks, May 2005) demonstrates the Linux module tool that uses CPU affinity to identify performance problems. "With Scheduler improvements in Linux 2.6, is it ready for the data center?" (CCR2, September 2004) explains what makes a good scheduler (including CPU affinity). Find more resources for Linux developers in the developerworks Linux zone. Get products and technologies Get the CPU affinity patch for nice. Order the SEK for Linux, a two-dvd set containing the latest IBM trial software for Linux from DB2, Lotus, Rational, Tivoli, and WebSphere. Build your next development project on Linux with IBM trial software, available for download directly from developerworks. Discuss How do I do cpu affinity in Linux? is a good, hands-on forum discussion. Get involved in the developerworks community by participating in developerworks blogs. About the author Eli Dow is a Software Engineer in the IBM Linux Test and Integration Center in Poughkeepsie, NY. He holds a B.S. degree in Computer Science and Psychology and a Masters of Computer Science from Clarkson University. His interests include the GNOME desktop, human computer interaction, and Linux systems programming. You can contact Eli at emdow@us.ibm.com. 6/8

7 Close [x] developerworks: Sign in If you don't have an IBM ID and password, register here. IBM ID: Forgot your IBM ID? Password: Forgot your password? Change your password After sign in: Stay on the current page Keep me signed in. By clicking Submit, you agree to the developerworks terms of use. Submit Cancel The first time you sign into developerworks, a profile is created for you. This profile includes the first name, last name, and display name you identified when you registered with developerworks. Select information in your developerworks profile is displayed to the public, but you may edit the information at any time. Your first name, last name (unless you choose to hide them), and display name will accompany the content that you post. All information submitted is secure. Close [x] Choose your display name The first time you sign in to developerworks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerworks. Please choose a display name between 3-31 characters. Your display name must be unique in the developerworks community and should not be your address for privacy reasons. Display name: (Must be between 3 31 characters.) By clicking Submit, you agree to the developerworks terms of use. Submit Cancel All information submitted is secure. Average rating (67 votes) 1 star 1 star 2 stars 2 stars 3 stars 3 stars 4 stars 4 stars 5 stars 5 stars 7/8

8 Submit Add comment: Sign in or register to leave a comment. Note: HTML elements are not supported within comments. Notify me when a comment is added1000 characters left Post Total comments (2) Thanks for the post. I am getting linking errors wen i am compiling the program, gcc -g -D_GNU_SOURCE ThreadAndCoreAssign.c then i am getting link errors like, undefined reference to `do_cpu_expensive_op' undefined reference to `check_cpu_expensive_op' ld returned 1 exit status Someone help me to solve the problem. Posted by ChAmarnath on 10 September 2011 Report abuse This is a really good article that I hope is updated for the new CFS. Posted by mdturnerinoz on 28 December 2009 Report abuse Print this page Share this page Follow developerworks Technical topics Evaluation software Community About developerworks IBM AIX and UNIX Java technology By IBM product Forums Site help and feedback Solutions IBM i Linux By evaluation method Groups Contacts Software Information Management Lotus Rational Tivoli WebSphere Open source SOA and web services Web development XML More... By industry Events Briefings Webcasts Find events Blogs Wikis Terms of use Report abuse IBM Champion program Article submissions Related resources Students and faculty Business Partners Software services Support Product information Redbooks Privacy Accessibility Cloud computing More... Industries Integrated Service Management 8/8