Parallel Simulation of Equation-Based Models on CUDA-Enabled GPUs

Transcription

1 Parallel Simulation of Equation-Based Models on CUDA-Enabled GPUs Per Ostlund Department of Computer and Information Science Linkoping University SE Linkoping, Sweden Kristian Stavaker Department of Computer and Information Science Linkoping University SE Linkoping, Sweden Peter Fritzson Department of Computer and Information Science Linkoping University SE Linkoping, Sweden ABSTRACT Our contributions wit tis work are metods and a prototype implementation for compiling and executing a limited set of equation-based matematical models (written in te object-oriented equation-based modeling language Modelica) on CUDA-enabled GPUs. We look at metods of finding parallelism in Modelica models, tat can be used on te massively parallel CUDA arcitecture. Te metods ave been implemented in a new back-end module of te OpenModelica compiler (an open-source Modelica compiler). Tis paper sows tat it is possible to automatically generate simulation code for pure continuous-time models tat can be reduced to an ordinary differential equation system witout algebraic loops and were te initial values of all variables and parameters are known at compile time. It is possible to get some speedup compared wit simulation on a single CPU core, a (approximated) relative speedup of 4.6 was for instance obtained for one model. Categories and Subject Descriptors D.3.4 [Programming Languages]: Processors Code generation General Terms Algoritms Keywords CUDA, GPU, GPGPU, Modelica, Matematical Simulation 1. INTRODUCTION Grapics Processing Units (GPUs) ave in recent years become increasingly programmable, giving rise to an emerging field of General-Purpose computation on Grapics Processing Units (GPGPU) [7]. Te teoretical processing power of GPUs ave far surpassed tat of CPUs due to te igly Permission to make digital or ard copies of all or part of tis work for personal or classroom use is granted witout fee provided tat copies are not made or distributed for profit or commercial advantage and tat copies bear tis notice and te full citation on te first page. To copy oterwise, to republis, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. POOSC OCT-2010, Reno, USA Copyrigt 2010 ACM /10/10...$ parallel structure of GPUs. [14] Harnessing te performance of GPUs, owever, is problematic. GPUs are in general good at solving certain massive parallel problems, for instance problems tat exists in te field of grapic computation were computations are applied pixel by pixel. In tis work, owever, we are dealing wit a more irregular problem, as we sall see. Compute Unified Device Arcitecture (CUDA) is a software platform for Nvidia GPUs tat simplifies GPGPU. Modelica is a language for equation-based object-oriented matematical modeling tat is being developed troug an international effort [6]. Compilation of Modelica models is a complex process tat involves many steps. Te end result of te compilation process is primarily a system of equations tat is linked wit run-time simulation code. Generation of parallel executable code from Modelica models as been a researc topic for several years at our researc group, see for instance [1], [5]. For oter work on parallel differential equation solver implementations, see for instance [10], [11], [4]. Our contributions wit tis work are metods and a prototype implementation for compiling and executing a limited set of equation-based matematical models, written in Modelica, on CUDA-enabled GPUs. Since Modelica is a large and complex language we restrict our models in tis work to pure continuous-time models (discrete time and ybrid models are also possible in Modelica) tat can be reduced to an explicit Ordinary Differential Equation (ODE) system witout algebraic loops and were te initial values of all variables and parameters are known at compile time. Te metods ave been implemented in a new back-end module of te OpenModelica compiler, an open-source Modelica compiler developed by te Open Source Modelica Consortium (OC) [9]. We obtained, for instance, a relative speedup of 4.6 for te WaveEquationSample model (presented in section 3) wit 3840 sections (about 7680 state variables), comparing te Nvidia GeForce 8800 GTS to te Intel E6600 CPU using single precision calculations. See section 5 for more details about tis measurement. A different and less general approac to parallelism tat as potential for better speedup for some models, for instance te WaveEquation- Sample model introduced in section 3, is to compile data parallel Modelica array computations into CUDA code. A first feasibility study of tis approac can be found in [12]. Tis paper is mainly based on [13] wic is an application of te work in [1] and [5].

2 Te rest of te paper is organized as follows. Section 2 contains background information on te languages and tools involved in tis work. Section 3 introduces one of te models we wis to generate code for and simulate. In section 4 we describe our implementation and in section 5 we provide measurements of generating and executing code wit our new implementation. Finally in section 6 we discuss our results and in section 7 we give some general conclusions and discuss future work. 2. BACKGROUND TECHNOLOGIES Tis section contains background information on te languages and systems involved in tis work. 2.1 Te Modelica Modeling Language Modelica is a language for equation-based object-oriented matematical modeling and it is being developed troug an international effort [6]. Modelica allows te user to write is/er equations directly in te model code witout aving to manually transform te equations into computational low-level form in a standard imperative programming language. Instead tis is done by te Modelica compiler in question. Te user can also make use of ig-level concepts suc as object-oriented programming and component composition. Te multi-domain capability of Modelica gives te user te possibility of combining electrical, mecanical, ydraulic, termodynamic, etc., model components witin te same application model. 2.2 OpenModelica OpenModelica is an open source implementation of a Modelica compiler, simulator and development environment for researc, education and industrial purposes. OpenModelica is developed and supported by an international effort, te Open Source Modelica Consortium (OC) [9]. It consists of a Modelica compiler as well as oter tools tat form an environment for creating and simulating Modelica models. 2.3 Compute Unified Device Arcitecture Streaming Multiprocessor Instruction Unit Data cace SFU SFU Sared memory Grapics Processing Unit Off-cip DRAM Figure 1: Simplified scematic of a GPU wit 96 s. For a more detailed description te reader is referred to [7]. A CUDA-enabled GPU consists of several Streaming Multiprocessors (s) wom eac contain several Scalar Processors (s). See Figure 1. In eac clock cycle, eac can produce a result. All te s in one execute te same instruction. Te s execute asyncronously witout communication and no formal consistency model exists between tem. One could say tat we ave SIMD execution (or SIMT execution, see [7]) inside one and MIMD execution across several s. Some syncronization between te s is possible via te global GPU memory. Te programmer launces blocks of treads from te ost (normally a CPU) to be executed on te GPU. Te blocks are automatically sceduled onto te different s. Eac block is divided into several warps (typically 32 treads). One warp is run at a time on eac and if one warp is stalled (due to memory latency, etc.) anoter warp may execute. If any treads in a warp take divergent execution pats, ten eac of tese pats will be executed separately, and te treads will ten converge again wen all pats ave been executed. Tere are several different types of memories on a CUDAenabled GPU. Eac as a set of registers. Access typically requires no extra clock cycles per instruction, except for some special cases. Eac as a sared memory (16 Kbytes on te Tesla C1060) wic is sared by all te s. Accessing te sared memory is typically as fast as a accessing a register and te sared memory can be used for cooperation between te different treads in a block. Eac also as a constant and a texture cace. We don t use te constant and texture caces in tis work. Finally we ave an off-cip Dynamic Random Access Memory (DRAM) (4 Gbytes on te Tesla C1060). Tis memory is accessible from all s as well as externally e.g. from te CPU wit DMA transfers. Te DRAM is muc slower for te treads to access tan te sared memory, typically it takes 400 to 600 clock cycles for a memory access. 3. CASE STUDY Here we introduce one of te models we wis to simulate in order to introduce te Modelica language and give te user an idea of te problem at and. Tis model is taken from [3](page 584). Te one-dimensional wave equation is given by a partial differential equation of te following form: 2 p t = 2 p 2 c2 x. (1) 2 were p = p(x, t) is a function of bot space and time. We consider a duct of lengt 10 were we let 5 x 5. We discretize te problem in te spatial dimension and approximate te spatial derivatives using difference approximations using te approximation: 2 p t = (p i 1 + p i+1 2p i) 2 c2 (2) x 2 were p i = p(x 1 + (i 1) x) on an equidistant grid and x is a small cange in distance. We get te following Modelica model, were te initial pressure is 0.0 and were n can be altered to increase/decrease te number of sections and tus te number of state variables:

3 model WaveEquationSample parameter Real L = 10 "Lengt of duct"; parameter Integer n = 30 "Number of sections"; parameter Real dl = L/n "Section lengt"; parameter Real c = 1; Real[n] p(start = fill (0,n)); Real[n] dp(start = fill (0,n)); equation p[1] = exp(-(-l/2)^2); p[n] = exp(-(l/2)^2); dp = der(p); for i in 2:n-1 loop der(dp[i])=c^2*(p[i+1]-2*p[i]+p[i-1])/dl^2; end for; end WaveEquationSample; Tis model consists of two sections, a section wit parameters and variables and a section wit equations. Parameters are constants tat can only cange teir values between simulation runs. Te vectors p and dp are vectors wit state variables since tese variables occur wit te derivative operator. Te for loop is a for-equation tat will be expanded into scalar equations. All of te equations in te model are merged togeter in one system and andled as described in section IMPLEMENTATION In tis section we start wit a sort overview of compilation and simulation of Modelica code and ten we discuss our new back-end module for te OpenModelica Compiler tat generates CUDA code. 4.1 Compilation and Simulation of Modelica Models Due to te special nature of Modelica, te compilation process of Modelica code differs quite a bit from imperative programming languages suc as C, C++, and Java. For a more detailed description te reader is referred to for instance [2], [3]. Te output from te OpenModelica front-end is an internal data structure wit separate lists for variables, equations, functions and algoritm sections. Te mapping of time-invariant parts of a model into executable code is done in a relatively straigtforward manner. Te andling of te equations is muc more complex toug and involves among oter tings symbolic index reduction, topological sorting (according to te causal dependencies between te equations), conversion into assignment form, etc. Simulation corresponds to solving te resulting equation system wit respect to time using a numerical solver metod. Te compiled Modelica model (te final equation system and te compiled Modelica functions) is linked wit a simulation runtime system tat contains among oter tings an integration metod. Fourt-order Runge-Kutta is te numerical integration metod used in tis paper. Initial values are given in a data file (tese ave been extracted from te original model) and a final plot file is produced (containing values for te state and algebraic variables at different time intervals). 4.2 CUDA Code Generation In [1] it was investigated ow parallel executable code from equation-based models could be generated, wic resulted in a OpenModelica back-end module. Tis work was ten Modelica Model Front-End, Equation Handling,... DAELow CUDA Codegen Task Grap Creation Task Grap Task Merging Sceduling Equations Code Generation CUDA Code Merged Task Grap Sceduled Tasks Figure 2: Te process of compiling a Modelica model to CUDA code. continued in [5] by inlining te numerical solver and by introducing so-called software pipelining. In tis work we ave tried to adopt some of tese tecniques, especially from [1], for generating CUDA code. Te reader is referred to [13] for a more in-dept description. See Figure 2 for te different compilation pases Task Grap Creation and Task Merging Te CUDA code generation module is invoked in te backend of te compiler wit a sorted equation system as input. As a first step a task grap is generated from te equation system. A task grap is a directed acyclic grap were eac vertex represents a task (a scalar operation) and te edges of te grap represents precedence constraints on te tasks. Tere is a cost associated wit eac edge and task. Te cost of an edge is te cost of sending data between two tasks and te cost of a task is te cost of executing te task. So far we ave only used crude costs for te tasks: te cost of unary and binary operations are set to 1 and te cost of special functions are set to 4 (tis sould reflect te fact tat a as eigt s but only two special function units). Te cost of communication is set to 100, wic sould reflect te latencies introduced by te global memory. A task grap is a convenient structure to work wit for detecting parts of an ODE system tat can be computed in parallel. Because te initial task grap is extremely fine grained, a merging algoritm is applied to it. Scalar tasks are merged togeter into larger nodes Sceduling Wen te task grap as been merged it is necessary to determine in wic order te tasks sould be executed, and weter tey sould be executed in parallel on different s or not. Te sceduling is done in two main steps. In te first step te nodes in te merged task grap is sceduled wit te so-called critical pat algoritm and ten te tasks

4 inside eac node are sceduled. Te critical pat algoritm selects te critical pat in a grap, wic is te pat wit te longest execution time. Te critical pat algoritm is run over and over again until te wole grap as been covered. Te tasks inside one node are sceduled using a first in, first out queue wit te tasks to be sceduled. An example can be seen in figure 3. a c d e f b g g e c f d b a Time Figure 3: An example of te task sceduling algoritm. In addition to tese two steps te sceduler also tries to find nodes tat are operationally equivalent to oter nodes, and scedules tose nodes to be executed in parallel on te same (SIMD style execution). As of now we use a relatively crude test for operational equality, see [13] for more details. Execution Pat Execution Pat List Processor Scedule Figure 4: An example of a scedule for two streaming multiprocessors. Te result of te sceduling is a processor scedule. See figure 4. Te processor scedule contains execution pats and execution pat lists. An execution pat is a list of task executed in order. One executes one execution pat list, tat opefully contains several execution pats, at a time. Inside one we can, if tere are several execution pats, execute in SIMD mode. Running several s in parallel and syncronizing via te global memory, corresponds to MIMD execution. Te blocks of treads are sceduled automatically on te different s and we do not know in wic order te different blocks are going to execute. However, we never execute more blocks tan tere are s (to avoid dead-locks) and we syncronize via te global memory, tus we can run several blocks in parallel. Communication between processors is necassary in many cases. Eac node as to be inspected to see if any of te task as a dependency tat is sceduled on anoter processor. If tis is te case te sceduler inserts signals and locks into te scedule and determines wat data sould be sent were. Special execution pats for communication are inserted into te scedule Code Generation Code generation is done by iterating troug te scedule one processor at a time and one execution pat list at a time. If te execution pat list is a special execution pat for communication, communication code is generated. If te execution pat list contains several execution pats code for SIMD is generated. Oterwise if it is a normal execution pat list, code for eac task is generated one by one. A tecnique to reduce long off-cip DRAM latencies is memory coalescing. Te DRAM memory can access wole cunks of memory at a time. We use coalesced memory reads tat read 16 variables at a time from te device memory, te size of a coalesced read of 32-bit variables, and ten we move tose variables to were tey sould be in te sared memory on an. It does not matter if not all 16 variables are used since it costs as muc to read one variable as it does to read 16 variables. By coalescing memory accesses and aving several warps active at te same time it is tus possible to mask te latencies associated wit te off-cip memory somewat Generated Code In te code listning below te main simulation function is given. A fourt order Runge-kutta integration sceme is used. Tis integration sceme was used bot for te GPU implementation and te normal sequential CPU implementation, tat we used for comparison. Tis integration metod is relatively easy to implement yet gives decent enoug results. k 1 = f (x (t), u (t), t) k 2 = f ( x ( t + ) ( ) ) 2 k1, u t + 2, t + 2 k 3 = f ( x ( t + ) ( ) ) 2 k2, u t + 2, t + 2 k 4 = f (x (t + k 3), u (t + ), t + ) x(t + ) x(t) + 1 (k1 + 2 k2 + 2 k3 + k4) 6 Te first four steps in te metod evaluates f at different points in time, and in te last step a weigted sum of tese values is calculated. Te next value of x is ten calculated using te weigted sum. Te execution of tasks and incrementation of steps are done in parallel by launcing kernels. //Determine te size of te sared memory needed. int smem_size = 100 * sizeof(real); for(int step = 0; step < steps; ++step) { //Move te pointers of te result arrays forward. r_dx += DERIVATIVES; r_x += STATES; r_y += ALGEBRAICS; //Execute te tasks, call integration kernel. execute_tasks<<<7, 20, smem_size>>>(d_dx, d_x, step_and_increment1<<<2, 32>>>(d_x, d_old_x, d_dx, d_k, alf_); //Increment te time by alf a time step.

5 GeForce Tesla 8800 C1060 GTS Streaming Multiprocessors Scalar Processors Scalar Processor Clock (MHz) Single Precision GFLOPS Double Precision GFLOPS N/A 78 Memory Amount (MB) Memory Interface 320-bit 512-bit Memory Clock (MHz) Memory Bandwidt (GB/s) PCIe Version (1.0 used) PCIe Bandwidt (GB/s) 4 8 (4 used) CUDA Compute Capability Table 1: GPU Specifications Execution Time (seconds) E GTS C1060 single C1060 double N t += alf_; //Do two more steps of te RK4 metod. execute_tasks<<<7, 20, smem_size>>>(d_dx, d_x, step_and_increment2<<<2, 32>>>(d_x, d_old_x, d_dx, d_k, alf_); execute_tasks<<<7, 20, smem_size>>>(d_dx, d_x, step_and_increment3<<<2, 32>>>(d_x, d_old_x, d_dx, d_k, ); //Increment te time again wit alf a time step. t += alf_; //Do te final integration. execute_tasks<<<7, 20, smem_size>>>(d_dx, d_x, step_and_integrate<<<2, 32>>>(d_x, d_old_x, d_dx, d_k, _div_6); //Save te new values. cudamemcpy(r_x, d_x, STATES * sizeof(real), cudamemcpydevicetohost); cudamemcpy(r_dx, d_dx, DERIVATIVES * sizeof(real), cudamemcpydevicetohost); cudamemcpy(r_y, d_y, ALGEBRAICS * sizeof(real), cudamemcpydevicetohost); } 5. EXPERIMENTAL RESULTS In tis section we present measurements of executing CUDA code from one model on two GPU arcitectures and on one CPU arcitecture. For more measurement data see [13]. Table 1 sows te specifications of te two GPU arcitectures used. Figure 5 sows te time-measurements obtained. We alter te parameter n for eac simulation run. Te CPU used was an Intel Core 2 Duo E6600 (2.4 GHz clock frequency) but only one core was used (since tere were no code generator tat generates parallel code from OpenModelica available). Te model used, WaveEquationSample, was introduced in section 3 and section 6 contains a discussion of te results. An approximation was done tat gives an upper limit of te performance, see [13]. Figure 5: Execution Time for te WaveEquation- Sample Model as a Function of te Number of Sections GTS C1060 C1060 single double Task Execution Sared Mem Allocation Integration Memory Transfers Table 2: Seconds spent in te different parts of te simulation of te WaveEquationSample model. 6. DISCUSSION OF RESULTS We obtained a relative speedup of 4.6 wit n=3840 for WaveEquationsSample (comparing te GeForce 8800 GTS to te Intel E6600 CPU using single precision calculations). As noted before, tis was an approximation. For small models (few state variables) te GPU performs badly but as te number increases te GPU retain a relatively flat curve, wile te simulation times on te CPU rises muc faster. Te reason for tis is likely tat te GPUs need many tread blocks wit many treads to fully utilize teir power, and smaller models terefore do not use all s available. Te simulation times on te CPUs instead beaves as could be expected by approximately doubling wen te model size is doubled. It was sown tat memory transactions take most of te time, and te actual computations take a smaller amount of time. See table 2. Wile te models used in tis work are parallel enoug to easily spawn many treads, te computations per variable ratio is quite low. Models wit more computations per variable sould see an even larger performance increase wen using a GPU.

6 7. CONCLUSIONS Te goal of tis work was to evaluate te feasibility of simulating Modelica models on CUDA-enabled GPUs. We ave demonstrated tat it is possible to simulate at least a subset of typical models. Nvidia will soon release te new Fermi arcitecture [8]. Tis arcitecture as several improvements. It as a cace ierarcy, improved support for double precision operations, more sared memory on te s, support for running several kernels at a time (better utilization of idle s), etc. A potential future work could terefore be to compile CUDA code for te Fermi arcitecture using te metods outlined in tis paper. A different approac of using GPUs and CUDA for simulating Modelica models (or a subset of Modelica models) is to compile Modelica array-equations into CUDA code. Modelica models containing for-equations operating over large arrays are for instance common in models derived from discretizations of partial differential equations. Tis is recently initialized work at our researc group, see [12]. 8. ACKNOWLEDGMENTS Funded by te European ITEA2 OPENPROD Project (Open Model-Driven Wole-Product Development and Simulation Environment) and CUGS Graduate Scool. 9. REFERENCES [1] P. Aronsson. Automatic Parallelization of Equation-Based Simulation Programs. PD tesis, Linköping University, Sweden, Dissertation No [2] E. K. Francois Cellier. Continuous System Simulation. Springer, [3] P. Fritzson. Principles of Object-Oriented Modeling and Simulation wit Modelica 2.1. IEEE Press, [4] M. Korc and T. Rauber. Scalable parallel rk solvers for odes derived by te metod of lines. In H. Kosc, L. Böszörményi, and H. Hellwagner, editors, Euro-Par, volume 2790 of Lecture Notes in Computer Science, pages Springer, [5] H. Lundvall. Automatic Parallelization using Pipelining for Equation-Based Simulation Languages, Licentiate tesis Linköping University, [6] Modelica and te Modelica Association, accessed January ttp:// [7] CUDA Zone, accessed January ttp:// ome new.tml. [8] Nvidia Fermi, accessed January ttp:// arcitecture.tml. [9] OpenModelica, accessed January ttp:// [10] T. Rauber and G. Rünger. Iterated runge-kutta metods on distributed memory multiprocessors. In PDP, pages IEEE Computer Society, [11] T. Rauber and G. Rünger. Parallel execution of embedded and iterated runge-kutta metods. Concurrency - Practice and Experience, 11(7): , [12] K. Stavåker, D. Rolls, J. Guo, P. Fritzson, S. Scolz. Compilation of Modelica Array Computations into Single Assignment C for Efficient Execution on CUDA-enabled GPUs. EOOLT 2010, Oslo, Norway, October 3, [13] P. Östlund. Simulation of Modelica Models on te CUDA Arcitecture. Master Tesis. LIU-IDA/LITH-EX-A 09/062 SE. Linköping University, [14] M. Wolfe, Te Portland Group, Inc: Compilers and More: GPU Arcitecture and Applications, September