Setting Thread Affinity in Ultra Messaging Applications 2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.
Abstract You can force program processes or threads to run on specific processors or cores. This is typically referred to with a variety of names: thread affinity (the term used in, and the focus of, this article), process affinity, processor affinity, CPU affinity, or CPU pinning. Basically, we are binding a process or thread to a CPU or core. Supported Versions Ultra Messaging 6.7 and earlier Table of Contents Thread Management and Message Latency.... 2 Operational Mode.... 2 Context Thread Embedded Mode.... 3 Context Thread Sequential Mode.... 3 LBT-IPC Transport Operational Modes.... 3 Thread Affinity.... 3 C vs. Java/.NET.... 3 Platforms.... 3 Getting the Thread ID.... 3 Setting Affinity.... 4 Java Notes.... 4.NET/C# Notes.... 4 Variations and Conclusion.... 5 Thread Management and Message Latency For machines that run messaging applications, binding a messaging thread to a core reduces variations in that application's performance latencies to a smaller range. This increases the application's performance predictability and manageability, or, in a sense, reduces jitter. However, note that using thread affinity does not guarantee performance improvements, and in some cases may even degrade the application's performance. You can set thread affinity by setting application configuration options or issuing command-line commands, but for Ultra Messaging and this article, we explore a programmatic approach using system or library calls. The thread affinity call assigns an affinity mask value to a thread ID. An affinity mask is a thread or process bit mask that instructs an operating system scheduler which CPU or core to run on. Using a system call from C to set thread affinity is fairly straightforward; however, the challenges in Ultra Messaging lie in a) determining the thread ID, and b) using Java or.net languages to access this functionality. Operational Mode Ultra Messaging offers a choice of how event-processing threads are created with the context operational_mode configuration option, with either embedded mode or sequential mode. You can set thread affinity in either mode: 2
Context Thread Embedded Mode When you create a context, Ultra Messaging creates an independent context thread, which handles timer and socket events, and which does protocol-level processing, like retransmission of dropped packets. Ultra Messaging also creates, per context, one IPC, SMX, or RDMA thread for a receiver of the corresponding transport type. To set thread affinity in embedded mode, you must wait for Ultra Messaging to create the thread and deliver an event. Context Thread Sequential Mode Upon context creation, Ultra Messaging does not create a context thread. Instead, the application must donate a thread by calling lbm_context_process_events() when needed. Sequential mode lets the application control when and for how long the context thread runs. With sequential mode, you can call lbm_context_process_events() on any thread to do things such as Set a thread's timeout, so that the application regains control by the end of a desired period. Set thread affinity on a thread before calling any Ultra Messaging functions. LBT-IPC Transport Operational Modes You can also set operational mode for message-processing threads using the LBT-IPC transport. When you set option transport_lbtipc_receiver_operational_mode to sequential, the application calls lbm_context_process_lbtipc_messages() to process received LBT-IPC messages. However, this works only with the Ultra Messaging C API, because equivalent message processing calls do not exist in the Ultra Messaging Java or.net APIs. Thread Affinity To set affinity, first determine which is the context thread, and also if applicable, which thread is handling IPC, SMX, or RDMA reception. Also determine how to get a callback on that thread. With the thread ID you can then call the function to pin the thread to a core. For most UNIX platforms, the POSIX Threads API pthread_setaffinity_np() function sets the CPU affinity mask of the thread to the CPU set pointed to by cpuset. If the call is successful, and the thread is not currently running on one of the CPUs in cpuset, then it is migrated to one of those CPUs. The pthread_getaffinity_np() function returns the CPU affinity mask of the thread in the buffer pointed to by cpuset. C vs. Java/.NET C code can make system calls and therefore, has direct access to the thread. Java does not offer this access, however, you can use the JNI (Java Native Interface) or JNA (Java Native Access) libraries to call native (platformspecific) code. For.NET programming, the Platform Invocation Services (P/Invoke) feature enables.net code to call native code. Platforms Since we perform thread management at the platform or OS level, there are variations to system calls or their arguments. Thus, this article provides general programmatic approaches to setting thread affinity. Depending on your platform and thread type, you must use the system call that is appropriate for your platform. Getting the Thread ID You can call a system call from within a thread that returns a thread ID, for example pthread_self(). 3
Setting Affinity When setting affinity, first determine the right time to effectively pin the thread. For example, a good place to set thread affinity is on receipt of a BOS message, as in the following example: int rcv_handle_msg(lbm_rcv_t *rcv, lbm_msg_t *msg, void *clientd) { if (msg->type == LBM_MSG_BOS) { thread_set_affinity( ); where thread_set_affinity( ) is a generic for the actual system call you use based on platform. The following table shows platform-based affinity system calls: Platform To Get Thread ID To Set Affinity AIX pthread_self() bindprocessor() Darwin/OS-X pthread_self() thread_policy_set() HP-UX pthread_self() pthread_processor_bind_np( ) Linux pthread_self(), pthread_getthreadid_np() pthread_setaffinity_np() Solaris/SunOS thr_self() processor_bind() Win32 GetCurrentThreadId() SetThreadAffinityMask() Note that the preceding UNIX call examples use POSIX Threads (pthreads). Java Notes Peter Lawrey's Java-Thread-Affinity library uses JNI and JNA and lets you assign critical threads to isolated CPUs/ cores. See the following URL: https://github.com/peter-lawrey/java-thread-affinity For example, you can call the API setaffinity() to bind the current thread to the desired CPU, as in the following: // In receiver callback if (msg.type() == LBM.MSG_BOS) { int cpuid = 3; AffinitySupport.setAffinity(1L << cpuid); Likewise, AffinitySupport.getAffinity() returns the current thread's affinity..net/c# Notes.NET programming deals with managed threads, which are essentially OS threads with the Thread object attached. Use the P/Invoke feature to call the desired OS thread API. For example: [DllImport("kernel32.dll")] public static extern int GetCurrentThreadId(); /// <summary> /// Set the current thread affinity to cpumask /// </summary> public static void setthreadaffinity(int cpumask) { 4
Thread.BeginThreadAffinity(); int currthreadid = GetCurrentThreadId(); foreach (ProcessThread currthread in process.threads) { if (currthread.id == currthreadid) { currthread.processoraffinity = new IntPtr(cpuMask); currthread.prioritylevel = ThreadPriorityLevel.Highest; break; // In receiver callback if (msg.type() == LBM.MSG_BOS) { int cpuid = 3; setthreadaffinity(1 << cpuid); Variations and Conclusion This article addresses an approach for setting thread affinity in Ultra Messaging applications, but can also serve as a model for other thread management operations, such as setting thread priority. Also note that you can approach thread management by setting attributes to use when creating threads. Author Nate Stelton Technical Writer Acknowledgements Thanks to Eric Bowden and Brian Horst. 5