A High-Performance Platform for Real-Time Data Processing and Extreme-Scale Heterogeneous Data Management

Transcription

1 i PCGRID Workshop 2016 A High-Performance Platform for Real-Time Data Processing and Extreme-Scale Heterogeneous Data Management March 31, 2016 Hitachi, Ltd., Hitachi America, Ltd.,

2 Development of High Performance Data Base Hitachi s DB technology since 1970s. Application example ArcGIS x SAP Hana x HADB(Hitachi Advanced Data Base) GIS( Geographical Information System) In-memory database High-speed database Source: ESRI Japan 1

3 HADB broke TPC- H Benchmark Record Hitachi Advanced Data Base as the outcome with a collaborative research with the University of Tokyo achieves the first registration to TPC-H 100TB (Class) 100TB Area of circle represent the number of enrollment Hita chi First product to be listed in the performance result list for TPC- H 100TB class ( Maximum DB Size) Hitachi BladeSymphony BS2000 x 4 30TB Company A 8 TB Memory 60 HDD FC cables 10TB 3TB ~1TB Hitachi Unified Storage 150 x GB Cache 1600 HDD Source:TPC-H - Top Ten Performance Results Version 2 Results ( / tpch/ results/ tpch_perf_results.asp) The number of enrollment of TPC-H (Year) Hitachi, Ltd All rights s reserved. 2

4 HADB vs. conventional database Fully utilizes the hardware (server, storage) resources SQL processing for DB search is automatically divided and executed with a high degree of parallelism Storage Access Trace in Conventional Approach [Conventional Approach] Sequential Execution Server Search Performance(μ s) Address (sector) Storage Periodic I/ O Processing(ms) Processing time (sec) [New Approach] Out-of-Order Execution Principle *1 Server Storage more than thousands of tasks Address (sector) Storage Access Trace in New Approach Fully utilize storage s performance Reduce the processing time dramatically Processing time (sec) Task Allocation Search Processing I/ O Wait Dis k I/ O Application of the outcome of Development of the fastest database engine for the era of very large database, and Experiment and evaluation of strategic social services enabled by the database engine project (Principle Investigator: Prof. Masaru Kitsuregawa, University of Tokyo and also Director-General, National Institute of Informatics), supported by the Japanese Cabinet Office s FIRST Program (Funding Program for World-Leading Innovative R&D on Science and Technology). * 1 A new principle invented by Professor Kitsuregawa and Project Associate Professor Goda (The University of Tokyo). 3

5 Applying HADB to eruption damage prediction Retrieve the most similar simulation result in two hours. Enable to predict damage after an eruption occurs. Building damage estimation caused by lava flow around Mt. Fuji 3. Retrieve the most similar simulation result after 100 hours of an eruption in two hours. data NIED researcher 4. Damage prediction based on visualized information query Hitachi Advanced Data Binder Platform 2.Store simulation results into HADB 1.Simulate 240 billion patterns of lava flow behavior Simulated lava flow behavior data Building data around Mt. Fuji NIED: National Research Institute for Earth Science and Disaster Prevention 4

6 HADB for Power System Application 5

7 Monitoring of power system stability Multiple data such as PMU, EMS/SCADA, etc. are used to monitor power system Stability and Alarm operators. PMU-based applications are crucial for situational awareness. Power system PMU PMU PDC Multiple data Alarm! EMS/SCADA Operator PMU: Phasor Measurement Unit, SCADA: Supervisory Control and Data Acquisition, EMS: Energy Management System 6

8 Future solutions for decision support Goal: Prevent critical wide-area blackouts and economical damage. Our Solution: Decision Support System (DSS) Decision support with actionable information in addition to the current operation: monitoring and alarming. Heterogeneous big data management (PMU, EMS/SCADA, ) and integration scheme for online/offline analysis. Now PMU EMS Log Near-future Monitoring (Alarm) Decision Support (Actionable Information) Online Analysis Offline Analysis 7

9 Heterogeneous big data management platform concept Ad-hoc big data analysis expands the possibilities of decision support. PDC Hitachi Stream Data Platform (HSDP) I/F Real-time data Real-time data processing Past~current Future (prediction) Current time Hitachi Real-time application SCADA I/F I/F I/F Hitachi Advanced Database (HADB)* Historical data *) Uses results from Development of the Fastest Database Engine for the Era of Very Large Database and Experiment and Evaluation of Strategic Social Services Enabled by the Database Engine (Principal Investigator: Prof. Masaru Kitsuregawa, The University of Tokyo/Director General, National Institute of Informatics), sponsored by the Japanese Cabinet Office s FIRST Program (Funding Program for World-Leading Innovative R&D on Science and Technology). Archived files Ad-hoc historical data analysis Time PMU_0001 PMU_0000 Trend search PMU_xxxx Snap-shot search ~100TB/year SQL Partners Products Hitachi Offline-analytical application 8

10 Applying HSDP and HADB to DSS - overview DSS leverages PMU data and a robust platform to support operators in making decisions.! 3. Display Real-time Event analytics results!! Hitachi Streaming Data Platform 1. Detect events by HSDP. 2. Store PMU data 5. Store Operational history 4.Store real-time Event analytics results 6. Find out similar incidents from stored data *) Uses results from Development of the Fastest Database Engine for the Era of Very Large Database and Experiment and Evaluation of Strategic Social Services Enabled by the Database Engine (Principal Investigator: Prof. Masaru Kitsuregawa, The University of Tokyo/Director General, National Institute of Informatics), sponsored by the Japanese Cabinet Office s FIRST Program (Funding Program for World-Leading Innovative R&D on Science and Technology). 9

11 Event detection and similarity search Events are detected out of streaming PMU data using power system-tuned statistical methods. Similar events are extracted by utilizing the power of a highspeed database. PMU Data Stream [Case] Line fault Voltage Event detection Current event [Case] Generation drop Frequency Event detection Current event Similar Past Event Similarity search Similar Past Event Time Similarity search Time High-speed DB 10

12 DEMO PMU PMU PDC Real-time monitoring and event detection 1System model (*) Power system 2 Real-time monitoring 5 Detected Event key Operational support 4 Event list 3 Event detection 7 The most Similar event 18 PMU, 4 months & 1 year events Data (**) (*) Modified Kundur 4-machine model (**)DB size: About 18TB on HADB 6 Similar Event lists 8 Operational Log 11

13 Demo movie - event detection 12

14 Demo movie similarity search 13

15 Potential impacts of the HADB on PMU data search Scenario 2: The analyst finds similar snapshot patterns in the past from other PMUs. Scenario 1: The analyst finds trend patterns in the past at the same PMU. PMU:A(t1) PMU:A(t1) PMU:A(t2) PMU:C(t1) PMU:A(t2) PMU:A(t3) PMU:A(t4) PMU:C(t2) PMU:A(t5) PMU:B(t1) PMU:B(t2) High performance makes decision support real. Query f or scenario 1 10 minutes PMU Conform Load Sensor Data for 4 substations A ccess data min minxx60 60x x3030xx44sensor data xx44substation data substation= =288K 288K data Query f or scenario All PMU Sensor Data for substations at a specific time snapshot A ccess data snapshot 4536sensors data x 5xsnapshot = =2323KK data data Ad-hoc data analyst can not wait for the response in this area. x Query for scenario 1 Conv. DB x Query for scenario 2 HADB 14

16 DSS similarity search evaluation second 1.4 Response time Search key waveform Similarity The most similar waveform (similarity = (0 is equal)) * Response time is proportional to # of events in the database 6TB 18TB MAX MIN AVERAGE ** * 21PMU, 32 days PMU data ** 18PMU, 4 month PMU data DSS powered by HADB achieves both short response time and high similarity. 15

17 Future discussion Retrieves similar events 0.6 sec from 6TB database,1.2 sec from 18 TB database. Approximately 29.8 sec to retrieve similar events from WECC 1 year database (approximately 700TB, 1000PMU). Retrieves similar past event, get a similar snapshot, then analyze power system using a snapshot for evaluation of new power grid control method PMU data <1 year> 29.8 sec Similar events Select a snapshot of area where similar event was occurred. Snap shot (PMU/SCADA) * Events occur 6 times per month Facility DB Generator Line Switch SC ShR SV/TM SCADA data <1 year> Load Flow, OPF, Voltage Stability Transient Stability Power system analysis New control method New control method New control method 16

18 17