Realization of Big Data Platform for PMU - Real-Time Data Processing & Historical Data Analysis -

Transcription

1 Realization of Big Data Platform for - Real-Time Data Processing & Historical Data Analysis - March 27, 2014 Yutaka Kokai Hitachi America, Ltd. Shinji Fujiwara Hitachi, Ltd.

2 Contents 1.Technical requirements for Control Center 2.High-Speed Data Access Platform for Control Center 3.Live Demo ~ Application for Control Center ~ 1

3 1. Technical requirements for Control Center 2

4 Number 1-1 Installation in North America The number of installed s are growing in North America Source: NASPI Oct 22, installation in North America 1,126 About 500 March 2012 March 2013 Source: DOE Synchrophasor Technologies and their Deployment in the Recovery Act Smart Grid Programs, August 2013 In the case of WECC, 481 s are installed. Source: DOE Synchrophasor Technologies and their Deployment in the Recovery Act Smart Grid Programs, August 2013 The number of installed s are increased. Now the usage of data needs to be considered. 3

5 1-2 Database Requirement for Data In the case of 500 s are installed data size Number of s Message rate Time 94[bytes] (8 phasors, 4 analogs, 2 digitals) [messages/sec] [seconds/year] 40 TB data will be accumulated about 40 TB / year! And the number of s are expected to increase more and more in the future... PDC Information Storage & Retrieval System for Big Data Analysis ( data) Operator PDC Platform for Big Data Analysis is required 4

6 1-3 Applications for Control Center To realize applications using data, both Real-Time Data Processing and Historical Data Analysis are required. Real-Time Data Processing Wide-area monitoring Voltage stability analysis Oscillation analysis Islanding Mode meter State estimation Historical Data Analysis Power plant model validation Frequency response analysis Post event analysis Baselining Fault location 5

7 2. Data Analysis Platform for Control Center 6

8 2-1 Data Analysis Platform for Control Center Real-Time Data Processing: Online power system analysis using the data in real time. Historical Data Analysis: Quick Data Loading and High-Speed data access to historical data, comfortable data viewing and quick power system analysis. PDC Send Sensor Data (every a few sec) Super PDC Real-Time Data Processing Stream Data Processing Analysis Applications Model Online Power System Analysis Model updates PDC In-memory Data Cache DBMS Load Sensor Data (every 5 to10 min) Archive Files Access Historical Data (in 2 to 3 sec) Off-line Analysis Historical Data Analysis 7

9 2-2 History of Hitachi s Database Management System 8

10 2-3 High-speed Data Access Platform for Control Center Hitachi Advanced Data Binder Platform *1 (HADB) - x100 fast Database Engine *2 - Configured with Hitachi server and storage system High-speed Data Loading for Huge Data Multi-Table JOIN over Multiple Data Sources BI MicrosoftR Excel Customer Value Ad-hoc Data Analysis Business Improvement *1 The outcome of Development of the fastest database engine for the era of very large database, and Experiment and evaluation of strategic social services enabled by the database engine project (Principle Investigator: Prof. Masaru Kitsuregawa, University of Tokyo and also Director-General, National Institute of Informatics), supported by the Japanese Cabinet Office s FIRST Program (Funding Program for World- Leading Innovative R&D on Science and Technology). *2 Compared with Hitachi conventional technology. The performance was measured based on standard analytical benchmarks. The speeding-up effect on various queries are different. We compared the performance of searching a fixed amount of data which met a certain condition in the database. High-speed Data access Platform Hitachi Advanced Data Binder Platform High-speed Database Engine Hitachi Server Hitachi Storage What s the New Technology? 9

11 2-4 Innovative Technology of Database Engine Out-of-Order Database Engine technology *3 - SQL is automatically divided into thousands of task and each task issues asynchronous I/O to get data from disk - These tasks are executed by Out-of-Order control method Thousands of Asynchronous I/O improves the SQL processing *3 A principle proposed by Prof. Kitsuregawa (Univ. of Tokyo and also Director-General, National Institute of Informatics) and Project Associate Prof. Goda (Univ. of Tokyo) Conv. DB Engine (In-Order) Out-of-Order DB Engine Sparse I/O Conv. DB Engine 従来型データベースエンジン In-order 決定的な processing 処理順序 Out-of-Order DB Engine 超高速データベースエンジン Out-of-Order 非決定的な processing 処理順序 Dense I/O 時間 time 時間 time 同期入出力 Small number of Sync. I/O 発行 超大量 Large number of Async. I/O 非同期入出力 発行 Most of CPU time is I/O wait SQL is executed by thousands of task without waiting I/O completion 10

12 2-5 Technology in Detail Key point Server DB engine automatically divides search process into thousands of tasks and executed them asynchronously. [Conventional Technology] In-order exec. Search processing (μs) Task assignment Search processing Wait for I/O completion Disk I/O E B Storage Synchronous I/O processing (ms) [New Technology] Out-of-Order exec. *4 I/O wait time causes long proc. time D A C Server Storage E A B C D B E Search process is divided into thousands of tasks, and they are carried out in parallel. Using asynchronous I/O, DB server issues as many I/O as possible. DB server continues execution of each task in I/O completion sequence. C D A *4 A principle proposed by Prof. Kitsuregawa (Univ. of Tokyo and also Director-General, National Institute of Informatics) and Project Associate Prof. Goda (Univ. of Tokyo) 11

13 2-6 HADB broke TPC-H Benchmark Record

14 2-7 Demonstration Please watch a Live Demonstration 13

15 2-7 Demonstration (Cont d) 14

16 2-7 Demonstration (Cont d) 15

17 3.Live Demo ~ Application for Control Center ~ 16

18 3-1 Application for Control Center Technological Merits Imports measurement data generated at a speed of tens of thousands per second without delay. Retrieves requested data within a few seconds from 100 TB scale historical data. Thermal power plant Hydro power plant 1. Fault STATCOM Power Grid Wind farm PDC Sensor data are sent every a few seconds High-speed data import Transmission Network Operator Information Storage & Retrieval System Real-time Check HADB Historical Data (45 MBytes/sec; every 5 minutes) Real-time problem analysis 2. Fault event 3. Analysis Operator Visualize historical data Retrieves data in a few seconds from a 100TB-scale historical data HADB: Hitachi Advanced Data Binder PDC: Phasor Data Concentrator : Phasor Measurement Unit STATCOM: Static Synchronous Compensator IS & R: Information Storage and Retrieval 17

19 3-2 Demonstration System Configuration Data source : Time series data from Phasor Measurement Units (dummy) - # of s: Measurement interval: 33.3 milliseconds Data import operation: Imports data in every five minutes (400 MBytes) Size of historical Data: 1.2 Tbytes (14.5 Billion records, 11 days) DB server : HA8000/RS220 (12 cores/24 threads) + AMS2500 (64 HDDs) Client PC DBMS : Notebook PC : Hitachi Advanced Data Binder Platform : Open Source DBMS Application for Control Center Voltage Trend of s Client PC NW HA8000/ RS220 DB Server Application Server HADB Open Source DBMS Sensor data are imported periodically Sensor Data 1 st Demo : Phasor Measurement Unit 2 nd Demo AMS2500 Historical Data HADB based IS & R Historical Data Traditional DBMS based IS & R 18

20 3-3 Demonstration 1 st Data Import Demonstration 2 nd Data Retrieve Demonstration 19

21 3-4 Summary - Data Import Performance HADB based IS & R 45.0 MBytes/sec Traditional DBMS based IS & R 4.5 MBytes/sec x 10.0 Faster 20

22 3-5 Summary - Data Retrieve Performance Traditional DBMS based IS & R HADB based IS & R 14.5 sec 1.1 sec x 13.2 Faster 21

23 END Realization of Big Data Platform for - Real-Time Data Processing & Historical Data Analysis - March 27, 2014 Hitachi America, Ltd. Hitachi, Ltd. Infrastructure Systems Company, Power Information & Control Systems Division Information & Telecommunication systems Company, IT Platform Division Group, IT Platform R & D Management Division 22

24 Trademarks Microsoft and Excel are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. TPC Benchmark and TPC-H are trademarks of the Transaction Processing Performance Council. Other company and product names mentioned in this document may be the trademarks of their respective owners. Throughout this document Hitachi has attempted to distinguish trademarks from descriptive terms by writing the name with the capitalization style used by the manufacturer, or by writing the name with initial capital letters. Hitachi cannot attest to the accuracy of this information. Use of a trademark in this document should not be regarded as affecting the validity of the trademark. 23

25