Real-Time High Performance Computing Platform for Smart Grid Applications

Real-Time High Performance Computing Platform for Smart Grid Applications Bipul Luitel, Member, IEEE Real-Time Power and Intelligent Systems Laboratory Holcombe Dept. of Electrical and Comp Engineering Clemson University Clemson, SC 29634 USA iambipul@ieee.org Ganesh Kumar Venayagamoorthy, Senior Member, IEEE Real-Time Power and Intelligent Systems Laboratory Holcombe Dept. of Electrical and Comp. Engineering Clemson University Clemson, SC 29634 USA gkumar@ieee.org Abstract Real-time control of smart grid will require fast and accurate implementation of intelligent monitoring and decision making techniques. Intelligent methods with faster speed and better accuracy will be possible by implementing in parallel on high performance computing platforms. Such methods need to be rigorously tested in simulations before implementing in actual systems. Therefore, integration of simulation environments with high performance computing (HPC) platforms is necessary to develop and test real-time monitoring and control methods for smart grids. Such integration is a challenge because of hardware and software limitations of real-time digital simulator (RTDS), HPC platforms as well as physical inaccessibility of HPC systems. In this paper, an integrated platform consisting of RTDS, HPC and data acquisition system is developed for real-time streaming of simulation data for computation. A wide area monitoring system for a multi-machine power system is illustrated on such integrated platform. Index Terms HPC, Real-time simulation, RTDS, Smart grid, WAMS I. INTRODUCTION Addition of renewable resources, distributed generation and bidirectional power flows in the smart grid will make it more complex than ever. Smart grid will have high uncertainty and variability owing to high penetration of intermittent energy sources. Therefore, it is necessary to have real-time simulation of power system phenomena, and related systems of communication and control under these changed scenarios in order to ensure stability, reliability, integrity and security of the electric power grid. Real-time simulation is a computer model of a physical system that can be executed at the same rate as the actual system. Real-time simulator is a combination of computer hardware and software that enables real-time simulation. Because of the deterministic times for simulation, real-time simulators are used in various engineering fields, and more so in the simulation of mission and time critical applications. Real-time digital simulators (RTDS) are a set of hardware platforms and accompanying software suite for simulation of power systems []. These allow for the simulation of power system and control components to evaluate system performance under various operating conditions, and to test devices and control systems. Real-time simulation of power system has been of interest for quite some time now [2]. Today, RTDS are widely used by industries and universities across the world [3], [4]. Power system equipment manufacturers, electrical utilities and research institutes benefit from the realtime simulation technology which allows them to test and implement new ideas, including closed loop testing of physical hardware, before integrating with the actual system [5]. Many researchers over the past decade have used RTDS as a power system test bed for real-time power system simulation studies [6], [7]. RTDS R is a real-time hardware platform used in power system simulation. It is widely used for simulation of various power system components, phenomena and control. As smart grid research has taken a leap in the past few years, RTDS R is also being used in simulation of distributed power grids [8], for the study of HVDC grid [9], and several other smart grid applications such as simulation studies on integration and testing of phasor measurement units [0] [2]. Thus, RTDS R has proven to be an important tool for successful deployment of smart grid technologies. Real-time digital simulators also enable hardware-in-the-loop based simulations. These are useful for implementing parts of the physical system in simulation and still operate as a complete physical system. Such advanced features allow fast integration of new technologies into the smart grid. Real-time simulation also allows testing of various operating conditions and phenomena that cannot be realistically implemented on the physical system. But as the complexity of physical system increases, so does the complexity of monitoring and control methods. In the future, intelligent monitoring and control of smart grid will be carried out on high performance computing (HPC) platforms using computational intelligence methods implemented using parallel programming [3]. Therefore, it is important to have a closed loop communication between the HPC and the RTDS for simulation and study of intelligent monitoring and control techniques implemented on HPC to be used in smart grid applications implemented on RTDS. Until now, such implementation has been challenged by the following problems: ) RTDS and HPC infrastructures are not physically collocated (and may be several miles apart) to share appropriate signals between them.

2) HPC cluster consists of several computing nodes and which nodes will be used for a particular computation is non-deterministic. 3) RTDS does not natively support sharing raw data over the Ethernet for the purpose of wide area monitoring and control. In this paper, a method for using HPC in closed loop communication with the RTDS R is presented. The proposed real-time high performance computing (RT-HPC) platform is used to implement a wide area monitoring system (WAMS) for smart grids. The remaining sections of the paper are arranged as follows: different components of the RT-HPC platform are described in Section II. The wide area predictive state estimation is described in Section III. The results of the study are shown in Section IV and conclusions in Section V. II. THE RT-HPC PLATFORM In an actual physical system, the health of the system is assessed by monitoring different parameters of the system. Based on the values of these parameters, decisions are made and control signals dispatched to drive the system from one operating condition to the other or keeping it stable. Therefore, decision making and control systems need to know the status of the system. In a power system, certain events take place very fast and therefore fast decision making and control systems are required for real-time control. With increase in system complexity, more complex decision making and control systems will be required. These systems need to be implemented in parallel using high performance computing platforms in order to meet the timing requirements of real-time monitoring and control. Therefore, in order to simulate and study intelligent real-time monitoring and control techniques utilizing HPC, a platform for integrating simulation environment with computing resources is required. The RT-HPC platform provides the ability to share data between the RTDS R and the HPC. Power systems are modeled on RSCAD and simulated on RTDS R. Intelligent monitoring and control algorithms that are implemented on HPC require the data generated from the simulation in real-time. However, HPC clusters are generally physically inaccessible. HPC clusters may consist of thousands of individual nodes connected to each other and it is not known which node is going to be utilized when a program is executed. Therefore, it is impractical to share data between the RTDS and the HPC via direct physical connection. Therefore, in the platform developed in this study, data is shared between the two via TCP sockets using the Labview programming environment and PXI controller hardware as the intermediate component between the RTDS R and the HPC cluster. The different signals of interest in the simulation are connected between the RTDS R and the PXI controller using physical wires. A program written in Labview software on the PXI controller then communicates the signals over the Ethernet. The output of monitoring and control algorithms from the HPC cluster is then transmitted back to the RTDS R using the reverse path. The RT-HPC platform with all the components is shown in Fig.. A. Real-Time Digital Simulator Real-time digital simulator used in this study is the RTDS R from RTDS Technologies. A single rack of the RTDS is shown in Fig. 2. The RSCAD graphical user interface (GUI) module provides design tools and component libraries for creating a model of the power system to be simulated on RTDS R. Fig. 3 shows the Draft case of the test system modeled on RSCAD. The draft is then compiled and loaded and executed on the RTDS R at 50 µs time step (i.e. 20 KHz). The Runtime GUI module provides visualization and control interface for the simulation running on the RTDS. This module consists of GUI tools for the components included in the Draft case. Fig. 4 shows the Runtime window with the controls and graphs for the simulation. Each rack of the RTDS R consists of processing, communication and interface cards. In this study, two racks consisting of three Gigaprocessor cards each are used for running the test system. The racks also consist of analog and digital input/output (I/O) cards. RSCAD library provides with components that can be used to receive/send analog and digital signals to/from the simulation. The analog input cards of the RTDS, called GTAI, are used to receive signals into the simulation, and the analog output cards, called GTAO, are used to to send them from the simulation. This is done by analog to digital (A/D) and digital to analog (D/A) converters on the I/O cards. The GTAI/O cards provide analog interface to the external devices for hardware-in-the-loop studies. Fig. 5 shows the GTAI and GTAO cards consisting of 2 channels each. In this study, voltages of 2 buses is sent from the RTDS R to the HPC cluster. Therefore, 2 analog channels from a GTAO cards are used. Similarly, the predicted bus voltages are sent back to the RTDS R from the HPC cluster for visualization purpose, hence using 2 channels of two GTAI cards. B. PXI Controller The PXI controller from National Instruments (NI) is a computer with an i5-250e 2.5 GHz processor running Windows 7 32-bit operating system. The PXI controller hosts NI s Labview development suite. Apart from the controller, the PXI chassis consists of several slots that can be used to add analog and digital I/O cards. In Fig. 6, the configuration of PXI controller used in this study is shown. It consists of two analog input (AI) and two analog output (AO) cards. Each card consists of 32 analog channels for external interface. The 8 channels of the AI card on the PXI are connected to the 8 channels of the two GTAO cards of the RTDS R by physical wires. Similarly, the 5 channels of the AO card on the PXI are connected to the 5 channels of the two GTAI cards. A Labview program is written to read/write the signals from/to the I/O cards connected to the PXI controller, using which a two-way communication between the RTDS and the PXI controller is established. Another task of the PXI controller is to communicate the signals received from the RTDS R to the HPC cluster, and vice versa. Since both the cluster and the PXI controller are connected to the Ethernet, the communication is performed

Fig.. Real-time high performance computing platform showing the different components of the platform. Fig. 2. A single RTDS R rack showing processing, communication and interface cards. Fig. 4. The Runtime module of the RSCAD showing control and visualization components. Fig. 5. Fig. 3. The Draft module of the RSCAD window showing the power system model and component library. The GTAI and GTAO cards of the RTDS R. in real-time using TCP sockets. The Labview program implements a TCP server and listens to a specific communication port. Data is then shared between the server and any client that connects to the port. The TCP read/write program works like a file read/write to a communication port, and hence the shared data is in string format. Therefore, the data from the

Fig. 6. cards. The PXI chassis showing the controller, and analog input and output! " #$ $%% & % #$ $ (* +%, (#) ' % (#) ' % $- (* +%, $- Fig. 7. The block diagram shows different operations performed using Labview on the PXI controller. different signals is first converted to string and concatenated before sending it via TCP ports. Similarly, a concatenated string received from the cluster is first split and converted to numerical value before sending it back to the RTDS via AO cards. The Labview program is shown in Fig. 7. C. High Performance Computing Platform The high performance computing platform used in this study is the Clemson University Palmetto cluster located at Pendleton, SC approximately 0 miles from where the RTDS and the PXI controller are located. It consists of,68 compute nodes with a total of 5,20 cores, and benchmarked operation at 96 TF/S. The cluster operates on Scientific Linux operating system and uses PBS job queuing system [4]. The different compute nodes use Infiniband and Myrianet network fabrics to communicate with each other. Each user program that needs to run on the cluster is considered as a job and is handled by the PBS. The number of processing units requested by the user in each job may be allocated from any combination of available compute nodes in the pool. A program written in C using message passing interface (MPI), a set of libraries that enable communication between the processors, is used to implement the WAMS in this study. The program runs in parallel on multiple processors and data among the processors is shared using send and receive functions available in MPI libraries. The program running on the master processor acts a a TCP client and connects to a specific port on the TCP server, which is the PXI controller. Thus, the master processor is able to establish a two-way communication with the PXI controller. Once the concatenated string of data is received, it is shared with the other processors using MPI after splitting and converting it to a numerical value. The master processor also collects the data from different processors, converts it to string and concatenates before sending it the PXI controller using TCP. Thus, the algorithm for predicting the voltages at different buses running concurrently on different processors is able to utilize the data from the RTDS received via PXI controller. III. WIDE AREA PREDICTIVE STATE ESTIMATION The test system used is a 2-bus system shown in Fig. 8 consisting of three generators [5]. Prediction of power system variables allows a wide area monitoring system to estimate the system states ahead of time. In this study, a WAMS for the test system simulated on RTDS R is implemented on the HPC platform. The WAMS is developed using computational elements arranged as cells that connect to each other in a certain fashion. The WAMS consists of 2 cells, each cell representing one bus of the system. The computational units predict the voltage of the bus they represent based on the past values, and predicted time-delayed inputs from the connected buses. The computational units of the WAMS are implemented using neural networks in this study [6]. The WAMS is thus developed based on the connectivity matrix of the buses and the generators of a power system. A WAMS developed for bus voltage prediction of 2-bus system is shown in Fig. 9. The figure shows that each bus is represented by a cell which is connected to the other cells representing the connected buses. Thus it is able to capture the topology of the system. The WAMS is implemented on HPC where each cell is implemented on a different processor and executed in parallel. The output of the WAMS is sent to the RTDS R and visualized on the Runtime. The predicted output of WAMS can be further utilized for situational awareness, decision making and predictive control in smart grids. IV. RESULTS In this study, the test system is perturbed by applying small disturbances to the excitation system of the generators. Since the operation takes place in real-time, the prediction algorithm utilizes past values of the arriving signals and predict the current values (as opposed to using current values to predict the future values) for visualization and comparison with the actual values. In order to see the ability of the WAMS to predict the behavior of the system during faults, a 0-cycle three phase short circuit is applied to bus five of the system, and the response of the WAMS is captured along with the response of the system. Figs. 0 and show the comparison

Fig. 8. IEEE standard 2-bus system used in the simulation. of actual against the predicted bus voltages when a small disturbance is applied and during a three-phase short circuit fault, respectively. These results are captured in the Runtime of RSCAD when the WAMS is running on the Palmetto cluster. Since the captured signal has a time step of 0 ms, the total time for computation and communication needs to be less than 0 ms in order to visualize without a delay. The platform developed in this study is able to satisfy this requirement. V. CONCLUSIONS An integrated platform consisting of RTDS R, the HPC cluster, and a data acquisition system is developed for streaming real-time simulation data for computation is developed. Real-time implementation of intelligent monitoring and control techniques in high performance computing platform is demonstrated. The data required by the algorithm running on HPC cluster is provided in real-time through the power system being simulated on RTDS R via Ethernet using Labview implemented on a PXI controller. As the size and complexity of modern power system increases, use of real-time high performance computing platforms becomes more significant. In the future, detailed study on timing based on distance between the simulation and computation platforms need to be performed in order to identify the limitations due to communication. The study presented in the paper also opens future area of research on applications of real-time high performance computing platforms in smart grids. ACKNOWLEDGMENT 92 2./ 93 3 9/4 9/ /.3 /4 97 7 95 9/3 5 /3.0 98 9 8. 96 90 6 9// 0 // Fig. 9. A wide area predictive state estimator overlaid on top of the 2-bus system for bus voltage prediction. The funding provided by National Science Foundation, USA under the CAREER grant ECCS #23820, #232070 and EFRI #238097 is gratefully acknowledged. REFERENCES [] R. Kuffel, J. Giesbrecht, T. Maguire, R. Wierckx, and P. McLaren, RTDS - a fully digital power system simulator operating in real time, in IEEE Communications, Power, and Computing Conference, vol. 2, 995, pp. 300 305. [2] M. Rafian, M. Sterling, and M. Irving, Real-time power system simulation, IEE Proceedings of Generation, Transmission and Distribution, vol. 34, no. 3, pp. 206 223, May 987. [3] RTDS, Clients of RTDS technologies, http://www.rtds.com/ clients-rep/clients.html,last accessed: Nov 2, 202. [4] OPAL-RT, Clients of Opal-RT technologies, http://www.opal-rt.com/ Customers,Last accessed: Nov 2, 202. [5] P. Forsyth, T. Maguire, and R. Kuffel, Real time digital simulation for control and protection system testing, in IEEE Power Electronics Specialists Conference, vol., June 2004, pp. 329 335 Vol.. [6] I.-Y. Chung, W. Liu, K. Schoder, and D. A. Cartes, Integration of a bidirectional DC-DC converter model into a real-time system simulation of a shipboard medium voltage DC system, Electric Power Systems Research, vol. 8, no. 4, pp. 05 059, 20. [7] P. Tatcho, Y. Zhou, H. Li, and L. Liu, A real time digital test bed for a smart grid using RTDS, in Symposium on Power Electronics for Distributed Generation Systems, June 200, pp. 658 66. [8] K. Ravikumar, N. Schulz, and A. Srivastava, Distributed simulation of power systems using real-time digital simulator, in IEEE/PES Power Systems Conference and Exposition, March 2009, pp. 6. [9] P. Mitra, K. Vinothkumar, and L. Zhang, Dynamic performance study of a HVDC grid using real-time digital simulator, in Complexity in Engineering (COMPENG), 202, June 202, pp. 6.

V.0396.0394.0392 V 2.02 V 3 7 V 4 5 75 7 5 V 5.03.02.0 V 6 7 V 7 02.0398 V 8 7 V 9 V 0.02 V.0 :;<=>? @ABCD;<BC V 2.0 0 5 9 Fig. 0. Bus voltage prediction from WAMS compared against the actual signals when a small disturbance is applied to the excitation system of the generators. V.039.038.037 V 2 5 V 3 4 2 0.88 V 4 5 0.85 0.8 V 5 0.8 0.6 0.4 0.2 V 6 4 V 7.039.038.037 V 8 4 2 0.88 V 9 0 V 0 5 EFGHIJ V 4 2 KLMNOFGMN V 2 0 5 9 Fig.. Bus voltage prediction from WAMS compared against the actual signals when a 0-cycles three phase short circuit is applied at bus five. [0] C. Liu, Z. H. Rather, N. Stearn, Z. Chen, C. L. Bak, and P. Thogersen, Practical testing and performance analysis of phasor measurement unit using real time digital simulator (RTDS), in IEEE International Energy Conference and Exhibition (ENERGYCON), Sept. 202, pp. 408 44. [] H. Chen, B. Bhargava, F. Habibi-Ashrafi, J. Park, and J. Castaneda, Integration of RTDS with EPG synchrophasor applications for visualization and analysis of simulation scenarios at Southern California Edison, in North American Power Symposium, Sept. 202, pp. 5. [2] D. S. Ouellette, M. D. Desjardine, and R. Kuffel, Using a real time digital simulator with phasor measurement unit technology, in th International Conference ondevelopments in Power Systems Protection, April 202, pp. 6. [3] B. Luitel and G. Venayagamoorthy, Parallelization of cellular neural networks for smart grid applications, in Clemson University Power Systems Conference (PSC2), Clemson, SC, Mar. 202. [4] CU CITI, The palmetto cluster, http://citi.clemson.edu/palmetto/, last Accessed: Jan 4, 202. [5] S. Jiang, U. Annakkage, and A. Gole, A platform for validation of FACTS models, IEEE Transactions on Power Delivery, vol. 2, no., pp. 484 49, Jan. 2006. [6] B. Luitel and G. Venayagamoorthy, Decentralized asynchronous learning in cellular neural networks, IEEE Transactions on Neural Networks and Learning Systems, vol. 23, no., pp. 755 766, Nov. 202.