A GPU Heterogeneous Cluster Scheduling Model for Preventing Temperature Heat Island

Transcription

1 A GPU Heterogeneous Cluster Scheduling Model for Preventing Temerature Heat Island Yun-Peng CAO 1,2,a and Hai-Feng WANG 1,2 1 School of Information Science and Engineering, Linyi University, Linyi Shandong, China Institute of Linyi University of Shandong Provincial Key Laboratory of Network based Intelligent Comuting, Linyi Shandong, China Abstract. With the develoment of GPU general-urose comuting, GPU heterogeneous cluster has become a widely used arallel data rocessing solution in modern data center. Temerature management and controlling has become a new research hotsot in big data continuous comuting. Temerature heat island in cluster has imortant influence on comuting reliability and energy efficiency. In order to revent the occurrence of GPU cluster temerature heat island, a big data task scheduling model for reventing temerature heat island was roosed. In this model, temerature, reliability and comuting erformance are taken into account to reduce node erformance difference and imrove throughut er unit time in cluster. Temerature heat islands caused by slow nodes are revented by otimizing scheduling. The exerimental results show that the roosed scheme can control node temerature and revent the occurrence of temerature heat island under the remise of guaranteeing comuting erformance and reliability. 1 Introduction After GPU (Grahic Processing Unit) was roosed by NVIDIA comany and its birth, it has been develoing raidly beyond the seed of Moore's Law, its comuting caability has been rising continuously. At SIGGRAPH conference in 2003, GPGPU(General-urose comuting on grahics rocessing units) was introduced. GPUs gradually shifted from dedicated arallel rocessors consisting of fixed functional units to architectures with rimary general-urose comuting resources and secondary fixed functional units. GPU is comosed of a large number of arallel rocessing units and memory control units, its rocessing ower and memory bandwidth has obvious advantages comared with CPU. However, GPU cannot comletely relace CPU, a lot of oerating systems, softwares and codes cannot run on GPU. GPU general-urose comuting usually uses CPU/GPU heterogeneous mode, CPU executes comlex logic and transactions and other tasks unsuitable for arallel rocessing, GPU imlements comute-intensive large-scale data arallel comuting tasks. With its high erformance, low energy consumtion and other advantages, CPU/GPU hybrid architecture has been widely used in grahics and image rocessing, video encoding and decoding, matrix comuting and simulation, medical industry alication, life science research, high-erformance comuting, signal rocessing, database and data mining and many other fields. With technology advances and breakthroughs, GPU is laying an imortant role currently in a Corresonding author: lyucy@163.com The Authors, ublished by EDP Sciences. This is an oen access article distributed under the terms of the Creative Commons Attribution License 4.0 (htt://creativecommons.org/licenses/by/4.0/).

2 large-scale arallel comuting. With the raid increase of roblem scales of various alication fields, single GPU s comuting caability has become insufficient, so multi-gpu and GPU cluster general-urose comuting has become a new research hotsot. As an imortant aroach of high-erformance comuting, GPU clusters have such advantages as low cost, high erformance and low energy consumtion for comute-intensive alications. In constructing GPU clusters, CPU and GPU cooerate with each other, articiate in data rocessing, and form GPU heterogeneous cluster. GPU heterogeneous cluster can make full use of hardware resources, imrove rocessing seed and throughut. It has become an imortant means of big data rocessing. Processing big data, esecially real-time big data stream needs cluster s continuous comuting and rocessing, and it will inevitably require comuter s high-load and continuous work, so the temerature of CPU, GPU and other comonents will continue to rise. On one hand, comuting energy consumtion increases, on the other hand, fans and air conditioners are needed for reducing temerature, thereby increasing cooling energy consumtion. When temerature rises to a certain extent, the temerature of one or some nodes will be too high. The node with too high temerature is known as temerature heat island. The occurrence of temerature heat island will reduce comuting reliability, ranging from result error to system s aralysis and halt. Once errors occur in comuting results, recomuting is needed, resulting in time and resource waste, increasing rocessing costs. In this case, we must reasonably design cluster task scheduling scheme to minimize cluster overall run time, control temerature to aroriate range, revent individual node from running so long that leading to over high temerature and forming temerature heat island, to ensure reliable comuting results, reduce energy consumtion as much as ossible and achieve green comuting. This aer studied the task scheduling on GPU heterogeneous cluster, and roosed a task scheduling scheme of reventing temerature heat island. The scheme s main features and advantages are: (1) strong robustness. The structure of GPU heterogeneous cluster is comlex, each node s configuration is different, and the node is often changed and adjusted. This task scheduling scheme can sense and adat to this comlicated and changeable situation. (2) high rocessing erformance. A task is divided into some sub-tasks, and then they are scheduled to multile nodes for arallel rocessing. The main roblem is determining the mode of division and treatment. The concet of comuting scale threshold and asymmetric artitioning method are roosed in order to adat to the diversity and heterogeneity of node configuration, imrove the arallelism and shorten the whole running time of cluster. This not only revents temerature heat island from occurring because of individual node s overlong running time, but also imroves rocessing erformance. 2 Related researches With the wide alication of GPU heterogeneous cluster, its task scheduling, temerature and heat management and energy consumtion otimization has become a research hotsot. Many scholars have ut forward various scheduling schemes and methods to solve the roblem of energy consumtion and reliability. This has layed a ositive role in reducing cluster energy consumtion and ensuring the reliability of comuting results. In [1] a dynamic task artition method was roosed. It divides arallel comuting tasks according to execution seed to achieve best overall system erformance. In [2] a multi-gpu self-adative load balancing method was roosed. GPU can self-adatively select tasks to execute according to local free-busy state by establishing task queue model between CPU and GPU. In [3] a load balancing strategy that combines task artitioning and stealing was roosed. It takes into account task affinity and rocessor diversity to direct task scheduling between CPU and GPU. In [4] feedback controlling was combined with mixed integer rogramming, and the energy consumtion controlling model of Web server cluster was constructed. In [5] model redictive controlling strategy was introduced from global ersective. The energy consumtion state is changed by adjusting comuting frequency and changing active stream multirocessor. The feedback controlling and rolling otimization mechanism 2

3 are used to redict future controlling to reduce redundant energy consumtion. In [6] the energy loss at idle state is reduced by a secific node selection strategy. CPU resource utilization is imroved by task tye division, combination distribution and DVFS. The above researches mainly focus on cluster task scheduling, changing CPU/GPU core voltage, frequency, hardware-based statistics, and so on to design cluster energy consumtion model, study task scheduling algorithm and achieve energy-saving urose, but do not consider temerature much. In GPU cluster comuting, esecially continuous comuting, temerature has obvious relationshi with energy consumtion and reliability. When temerature is too high, energy consumtion increases, reliability declines, and the robability of result error increases. Therefore, temerature should be controlled in a reasonable range to minimize energy consumtion under the remise of ensuring reliability. The task scheduling scheme roosed in this aer distributes tasks reasonably among comuting nodes to revent the occurrence of temerature heat island and ensure the correctness of comuting results. 3 Task scheduling model In GPU heterogeneous cluster, CPU and GPU all articiate in data rocessing. They are regarded as comuting units uniformly when distributing tasks. The comuters in cluster are controlling nodes and comuting nodes. The controlling node can be simultaneously used as a comuting node. All tasks form a queue. Each task is decomosed into several sub-tasks to form sub-task queue. The controlling node runs the main scheduling rocess, Scheduler. Each comuting node has a scheduling agent rocess, Agent. Scheduler and Agent cooerate to finish task scheduling. The architecture is shown in Figure 1. Task queue Sub-task queue Controlling node Scheduler Comuting node Agent Comuting node Agent Sub-task queue Sub-task queue Comuting node Agent Sub-task queue Figure 1. Task scheduling architecture 4 Task scheduling algorithm and strategy Scheduler algorithm is as follows: Algorithm 1 Controlling node Scheduler scheduling algorithm 1. Obtain a task from task queue 2. Obtain the hardware configuration and running status information of each comuting node 3. Determine the number of comuting nodes articiating in arallel rocessing 4. Divide task into sub-task queue and assign sub-tasks to corresonding comuting node 5. Wait for the results of each sub-task 6. Modify the status of corresonding sub-tasks and the associated tasks in queue. 7. Reschedule sub-tasks that timed out or requested to transfer, modify corresonding status 8. Go to 1 For each comuting node, the sub-tasks that controlling node disatches to it form a queue. The scheduling algorithm of Agent on comuting node is as follows: 3

4 Algorithm 2 Comuting node Agent scheduling algorithm 1. Obtain a sub-task from the sub-task queue of local node 2. Assign the sub-task to local node for rocessing 3. Wait for the result to be returned from local node 4. Reort results to controlling node (comletion, timeout, or requesting transfer) 5. Go to Acquiring hardware configuration information Scheduler first obtains the hardware configuration information of each node in cluster. The information can be manually created in advance and saved in file. When cluster is started, Scheduler loads cluster hardware configuration information file. It olls each comuting node, Agent resonds to the oll and reorts hardware change information to Scheduler. Or, Agent reorts hardware change information to Scheduler actively. Then Scheduler modifies cluster's hardware configuration information. In this way, controlling node can gras the latest changes in cluster hardware configuration, avoiding unnecessary acquisition and reorting of hardware configuration information, thus adating to actual hardware configuration changes and reducing network communication overhead. 4.2 Scheduling strategy Comuting scale is used to measure task size. Comuting scale is the number of instructions to be executed or the amount of data to be rocessed to comlete the task. A task contains arallelizable and non-arallelizable art. Suose the comuting scale of a task is T, T T s +T, T s is the comuting scale of non-arallelizable art, and T is the comuting scale of arallelizable art. Let T t be the critical value of the comuting scale of arallelizable art, then task scheduling strategy is as follows: (1) 0 T <T t, the task does not have arallelizable art or has relatively smaller arallelizable art, it cannot or has no need to be divided into small sub-tasks, the comuting unit with strongest comuting caability is selected directly from idle comuting units to rocess the task. (2) T T t, the task has arallelizable art and it reaches a certain scale. The arallelizable art of task is divided into smaller sub-tasks, many comuting units with strongest comuting caability are selected from idle rocessing units to rocess them Determining T t and the number of comuting units The rocessing caability is assumed to be C s when task is rocessed searately by a single comuting unit. Without loss of generality, assuming that when arallel rocessing, the number of comuting units articiating in rocessing is n, their rocessing caability is all C. In order to obtain better erformance, then: Tt C s T nc t + Q (1) Where Q is the additional time overhead required for arallel rocessing, including arallel comuting rearation, result merging, synchronization, network transmission, and so on. At the same time in order to ensure high rocessing efficiency, then: Tt Q nc (2) Solving the inequality grou consisting of above two inequalities will get: 4

5 T nc CsQ t max{ nc Q, } nc C s (3) The value of Q can be determined exerimentally or by accumulating historical emirical data. C can be taken as the average of the current comuting caability of all comuting units, and C s is the average of the current comuting caability of all CPUs in cluster. Let T mc CsQ m = max{ mc Q, } mc C s, m=1,2,,n idle, N idle is the number of all idle comuting units in current cluster. In order to increase the arallelization degree of task rocessing, change from N idle in descending manner until the first number k which lets T T k is found, then k is the number of units involved in arallel rocessing, the algorithm to determine it is as follows: Algorithm 3 Determining the number of arallel rocessing units 1. get N idle 2. i N idle k 1 3. if i 1 goto 7 ic CsQ 4. T i max{ ic Q, } ic C s 5. if T T i then k i goto 7 6. i i-1 goto 3 7. end If k<2 or qualified k value cannot be found, the task is handled by one CPU and not scheduled in arallel manner Partitioning arallel art Assuming that the current comuting caability of k comuting units involved in arallel comuting is C 1,C 2,.,C k, the scale of sub-tasks assigned to each rocessing unit is T 1,T 2,,T k, then the time to comlete the task is: t = t = max{,,..., } (4) T1 T2 T kk C1 C2 C i Where T 1 +T 2 + +T k =T. It can be roven that when (i=1,2,,k), t is minimum and T C,where C=C 1 +C 2 + +C k. Therefore, the roortion of allocated task to total task scale being equal to the ratio of the current comuting caability of the comuting unit to the sum of the current comuting caabilities of all comuting units articiating in arallel rocessing, can effectively reduce overall rocessing time, balance load, and avoid the case that some units are idle and some units run for long time and cause temerature heat islands to occur. T = i CT C 4.3 Estimating current comuting caability Current comuting caability is related to its own hardware configuration and hardware's current state of utilization. For comuting units with same configuration, the busier ones have stronger current comuting caability than the idle ones. By referencing [7] and imroving, the current comuting caability is estimated. For any comuting node N i, consider its five hardware configuration arameters: CPU frequency rate_cu i, memory size mem i, cache size cache i, GPU frequency rate_gu i, GPU memory size mem_gu i and five state arameters: CPU utilization utlz_cu i, memory 5

6 utilization utlz_mem i, cache utilization utlz_cache i, GPU utilization utlz_gu i, GPU memory utilization utlz_gumem i. The current comuting caability of node N i is: C = kq + k Q + k Q + k Q + k Q (5) i k 1, k 2, k 3, k 4 and k 5 reresents the level roortion weight of influence on node current comuting caability of CPU, memory, Cache, GPU and GPU memory resectively. Their sum is 1. Q 1 -Q 5 resectively denotes CPU current caability, memory current caability, cache current caability, GPU current caability and GPU memory current caability after normalization of node N i. Q 1 is calculated as: rate _ cui (1 utlz _ cui) Q1 = N ( rate _ cu (1 utlz _ cu )) j= 1 j j (6) The formulas for Q 2 -Q 5 are similar. For a certain node, C i, Q 1, Q 2, Q 3, Q 4 and Q 5 can be determined exerimentally, and then the aroximate value of k 1, k 2, k 3, k 4 and k 5 can be determined by regression method. 5 Exeriment and analysis The scheme roosed in this aer was verified exerimentally. Two exeriments were conducted on same cluster. The exeriment rogram is: Some relative softwares (such as CPU-Z, HWMonitor, CoreTem, etc.) were used to measure temeratures of CPU and GPU of each comuting unit at different time during cluster s running, and the temerature curve of each comuting unit was drawn according to them. Seven comuters were used to constitute GPU heterogeneous cluster. Five of them have the configuration: model is Lenovo Erazer X700, memory is 16G, CPU is Intel i7-3930k, GPU is NVIDIA GTX660i, oerating system is Ubuntu12.04 LTS, cluster environment is hadoo2.2.0, Java version is JDK1.7. The other two have lower configuration: CPU is Intel Pentium (R) Dual-Core E GHz, memory is 4G, GPU is NVIDIA GeForce 9400GT, oerating system is Windows 7 64-bit Ultimate. The exerimental data is taxi GPS data and data generated continuously by loadrunner. 5.1 Conventional scheduling method Firstly, conventional scheduling method was used. Only task balanced scheduling was considered, regardless of temerature changes. Every 1 minute temerature was samled once. The result is shown in Figure 2, where C1, C2,..., C7 is each comuting node. Temrature( o C) Samling Interval C 1 C 2 C 3 C 4 C 5 C 6 C 7 Figure 2. Temerature change in conventional scheduling method 6

7 The cluster rocesses taxi GPS data firstly. The data amount is larger, but because it is historical data, it does not take long time to rocess it. Temerature and ower are measured with measuring instruments, temeratures of CPU, GPU and so on are monitored with softwares. It is found that the temerature and ower of CPU and GPU are increasing during rocessing, but the task has been finished before temerature rises to the set threshold, and the roblem of temerature heat island and reliability does not occur. Then simulation data that loadrunner software continues to generate is rocessed. At this time, CPU and GPU temerature continues to rise, energy consumtion continues to increase. After a certain time, temerature exceeds the threshold and temerature heat island is formed, comuting result error occurs. The difference between the lowest and highest temeratures of various comuting nodes is about 12 C. 5.2 Scheduling scheme roosed in this aer In the second exeriment, the same exerimental environment and data were used, but the scheduling scheme reventing temerature heat island roosed in this aer was used. During rocessing task, temerature is collected. The result is shown in Figure 3. Temrature( o C) Samling Interval C 1 C 2 C 3 C 4 C 5 C 6 C 7 Figure 3. Temerature change in scheduling method reventing temerature heat island The result of rocessing taxi GPS data is similar to the revious, but the result shows that the difference of temerature and energy consumtion of each node tends to decrease. This shows that this scheme is more time-balanced in task scheduling to revent temerature heat island from occurring and guarantee overall stability. Data streams generated continuously by rogram are rocessed by cluster. It was found that, although the temerature of CPU and GPU increased, the temerature and ower did not increase continuously when temerature rised nearly to threshold value, and no temerature heat island and comuting error occurred. When data suly amount was increased, the henomenon that temerature and ower increase did not occur. This shows that the scheduling scheme trys to balance running time, inhibit the increasing of temerature and energy consumtion to revent temerature heat island from occurring. The difference between the lowest and highest temeratures among various comuting nodes is about 9 C. Analyzing above exerimental results, it is shown that, if conventional method is adoted, the temerature of each comuting node increases continuously with the rocessing of task, the temerature of some nodes exceeds threshold, and the temerature fluctuates greatly. However, when the scheduling method roosed in this aer is used, temerature is also rising, but because task division makes node running time be consistent as far as ossible, the range of temerature fluctuation is small, the overall temerature change is relatively calm, thus it is avoided that the temerature heat island occurs. 7

8 Conclusion The GPU heterogeneous cluster task scheduling scheme roosed in this aer avoids long running time of individual nodes as far as ossible, revents temerature heat island from occurring, guarantees comuting reliability, controls energy consumtion in a certain range, and also considers the constraints among temerature, reliability, erformance and energy consumtion, minimizes energy consumtion or imroves rocessing seed as far as ossible under the remise of ensuring reliability. The main work of next ste is to study how GPU heterogeneous cluster erceives and redicts cluster temerature and its variation, and aly it to cluster task scheduling. Acknowledgments This research roject is suorted by the joint secial roject of Shandong Provincial Natural Science Foundation (Project No.: ZR2015FL014) and the secial roject of Shandong Provincial Indeendent Innovation and Achievement Transformation (Project No.: 2014ZZCX02702). References 1. C. Q. Yang, F. Wang, Y. F. Du, et al. Adative otimization for etascale heterogeneous CPU/GPU comuting. The 2010 IEEE Int'l Conf. on Cluster Comuting. (2010) 2. L. Chen, O. Villa, S. Krishnamoorthy, G. R. Gao. Dynamic load balancing on single- and multi-gpu systems. The 2010 IEEE Int'l Sym. on Parallel & Distributed Processing (IPDPS). (2010) 3. E. Hermann, B. Raffin, F. Faure, T. Gautier, J. Allard. Multi-GPU and multi-cpu arallelization for interactive hysics simulations. The 16th Int'l Euro-Par Conf. on Parallel Processing: Part II (Euro-Par 2010). Berlin, Heidelberg: Sringer-Verlag. (2010) 4. L. Bertini, C. B. Julius, D. Mosse. Power otimization for dynamic configuration in heterogeneous web server clusters. Journal of Systems and Software, 83(4): (2010) 5. H. F. Wang, Y. P. Cao. GPU Power Consumtion Otimization Control Model of GPU Clusters. Acta Electronica Sinica, 43(10): (2015) 6. H. P. Huo, X. M. Hu, C. C. Sheng, B. F. Wu. An energy efficient task scheduling scheme for node-layer heterogeneous GPU clusters. Comuter Alications and Software, 30(3): (2013) 7. H. Liu, J. G. Wang, Z. Z. Ge, et al. Self-learning Load Balancing Scheduling Algorithm for GPU Heterogeneous Cluster. Journal of Xi'an Shiyou University, 30(3): (2015) 8