Evaluation of Latent Similarity Search in DSS4LA by Using Big PMU Data

Transcription

1 Evaluation of Latent Similarity Search in DSS4LA by Using Big PMU Data i-pcgrid Workshop /31/2017 Hitachi America, Ltd. Big Data Lab Mika Takata Panitarn (Joseph) Chongfuangprinya

2 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Consideration Related works Summary & DEMO 1

3 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Consideration Related works Summary & DEMO 2

4 Requirement: Urgent operation Disturbance status & operational support High Required Urgent Operation Blackout! damage low Initial Event Steady-state progression Triggering Event High-speed cascade time minutes~ second~minutes Reference by IEEE PES GM 2015 Tutorial 3

5 DSS Application - Overview DSS leverages PMU data and a robust platform to support operators in making decisions.! 3. Display Real-time Event Analytics Results Hitachi Streaming Data Platform!! 1. Detect Events by HSDP. 2. Store PMU data 4. Store Operational history 5. Retrieve appropriate countermeasures by finding out Similar Incidents from Stored Data 4

6 Time Heterogeneous Big Data Management Platform - Overview 4 PDC Ad-hoc big data analysis expands the possibilities of decision support. Hitachi Stream Data Platform (HSDP) *1) SCADA I/F I/F I/F *2) I/F 3 Real-time data Hitachi Advanced Database (HADB)* 3 Historical data *3) Uses results from Development of the Fastest Database Engine for the Era of Very Large Database and Experiment and Evaluation of Strategic Social Services Enabled by the Database Engine (Principal Investigator: Prof. Masaru Kitsuregawa, The University of Tokyo/Director General, National Institute of Informatics), sponsored by the Japanese Cabinet Office s FIRST Program (Funding Program for World-Leading Innovative R&D on Science and Technology). Archived files 2 Real-time data processing Past~current Ad-hoc historical data analysis PMU_0001 PMU_0000 Trend search PMU_xxxx Future (prediction) Current time Snap-shot search ~100TB/year SQL *1) Phasor Data Concentrator *2) Interface Today s focus 1 Similarity Search 5 Hitachi Real-time application Partners Products Hitachi Offline-analytical Application (e.g DSS) AI 5

7 Challenges to realize DSS4LA in data management 1. Fast similarity pattern search from historical big data Once a event is detected in real-time streaming data, operators need to make right decision for the countermeasure Storage cost needs to be saved =>Latent Similarity Search 2. Storage cost to store historical data Raw data needs storing by NERC s requirements =>SQL-Interfaced Federated Data Access Today s focus 3. Flexible data access within a few seconds =>Fast database access 4. Real-time operational intelligence for large numbers of PMU =>Geo-distributed Streaming Data Processing 5. Data preparation time for AI analysis =>Standard Data Model Based Data Preparation 6

8 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Consideration Related works Summary & DEMO 7

9 Oscillation frequency Conventional approach and issue-1 Conventional approach PMU data feature is extracted and stored on database Put tag on each past event on feature data-mart, and post-event analysis are performed by using the data-mart PMU data Offline Feature extraction (batch execution) PMU data Feature data-mart Tag Past event Event record time Event /01/05 00:13 Event /03/24 14:22 Database Oscillation damping 8

10 Conventional approach and issue-2 Conventional approach s issue Database management operation become complicated by Feature data-mart Analytical flexibility is reduced by pre-created feature data-mart (Trial-and-error is not easy and new knowledge/methods are hard to be integrated into analysis) Periodic batch execution Offline Feature extraction (batch execution) PMU data PMU data Feature data-mart Database Pre-defined features More data table management 9

11 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Related works Summary & DEMO 10

12 PMU data characteristics # Measured data by PMU Data feature 1 Voltage(Magnitude) Nominal value in steady state. Outliers may happen in events. 2 Current(Magnitude) Fluctuates up to season, time. (e.g. May be high value in evening, but low value in midnight.) 3 Voltage(Phase) Fluctuate in +180~ Current(Phase) Same as above 5 Frequency Nominal value in steady state. Outliers may happen in events. E.g.Voltage-Magnitude kv Steady state Event Outlier Voltage(magnitude) and frequency are mostly steady state, but outlier values may be caused when events happen. There are 1 or 2 events per a month. sec 11

13 Overview: Our proposed approach Latent similarity search Step1: Event data retrieval Step2: Single event recognition Step3: Similarity calculation th1 th2 th3 th N Step3.1: Data retrieve Step3.2: Feature creation/ Data conversion Step3.3: Similarity calculation thread Step 1 Retrieve event data Step 2 Recognize single events Step 3 Similarity calculation Step 3.1 Data retrieval, allocation Step 3.2 On-the-fly feature extraction Step 3.3 Similarity calculation There are multiple events in history, Then step3 needs multithreading processing 12

14 Step 1: Event data retrieval Define B-tree index on value data, which is mostly nominal voltage or frequency Retrieve value which is over certain upper threshold or lower than lower threshold As the number of events is small, this retrieval can be fast kv sec Outlier value B-tree index Outlier value SELECT * FROM PMUdata WHERE (Data_Type=Voltage and (value>upper_th1 or value<lower_th1) OR (Data_Type=Frequency and (value>upper_th2 or value<lower_th2) 13

15 Step 2: Single event recognition Step 1 outputs multiple values from one single event Step 2 recognizes points as single event if there are in certain time span The time span is calculated by event detection statistics kv Event #1 Event #2 Event #3 Certain term sec 14

16 Step 3: Similarity calculation 3-1 Retrieve Data event data based on step2 timestamp by using PMU database 3-2 Calculate features by using retrieved data on step Calculate similarity between current events(key) and candidates retrieved from historical data on step 3-2 and compare the similarities e.g. similarity equation like Euclidian distance w V, w F, w P ΔV, ΔF, ΔP w f, w d, Δf, Δd n weight for voltage, frequency, and power difference between key and candidate in respect with voltage, frequency and power weight for oscillation frequency and damping difference between key and candidate in respect with oscillation frequency and damping # of modes for similarity search 15

17 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Related works Summary & DEMO 16

18 Machine configuration Server CPU: Intel Xeon E (8 cores) 2.90GHz 2 Memory: 126GB Application Program Hitachi Advanced Data Binder * DBAREA00 (Table) Page size 16KiB DBAREA10 (Index) Page size 4KiB work (work for query) db (DB directory) server (server directory) load work (work for load) data (for import data) LV 16 PV PV OS: RHEL 6.2 LV 16 LV LV LV LV LVM PV PV PV PV... PV LV 16 PV FC: 8Gbps 4 500GB RAID 5(5D+1P)x16 LV : Logical Volume LU LU LU LU... LU LVM : Logical Volume Manager PV : Physical Volume LU : Logical Unit 16 LU HDD : Hard Disk Drive HDD HDD HDD HDD HDD 24 SAS : Serial Attached SCSI FC : Fibre Channel Storage HDD: SAS 10,000 rpm 600GB 17

19 Evaluation Comparison Proposed approach Conventional approach: Feature is extracted off-line and add tag on event timestamp Items to compare Feature data-mart size Data preparation time Similarity search response time PMU data size 8 PMUs, 3 months data # of events = 7 PMU database size = GB 18

20 Simulation data # of PMU = Event #4,5 Event #6,7 5 7 Event #1,2,

21 Evaluation results 8 PMUs, 3 months data (experiment) Conventional Proposed Feature data-mart size 67.8MB - Data preparation time 3770 sec - Similarity search response time ~1.9 sec ~6.3sec Case: 170 PMUs, 2 years data(34tb) (estimated) Conventional Easy DB operation Flexible to integrate new knowledge Proposed Feature data-mart size 11.3GB - Data preparation time 178 h - Similarity search response time ~15.2 sec ~50.8 sec Additional feature data-mart is necessary in addition to PMU data, which makes DB operation complicated Data preparation time is time consuming. *: Assume that counter-measure should be prepared in minutes if an event happens Requirement performance* (a few min) is achieved DB flexibility is improved though response time is slightly decreased. 20

22 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Consideration Related works Summary & DEMO 21

23 Consideration PMU data load performance Conventional (Without Outlier Index) Proposed (With Outlier Index) 2.24 M records/sec 1.67 M records/sec Achieve requirements of sensor data store, which is 1.0 M records/sec Outlier index data size cost Conventional (Without Outlier Index) Proposed (With Outlier Index) 100% 105% (5% of overall) Small database size increase 22

24 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Related works Summary & DEMO 23

25 Related work Historical data platform IBM Informix (SIGMOD 2014) Achieves fast data retrieves with compression of time-series data Access data though SQL interface BTrDB (USENIX FAST 2016) Defines abstract access method to time-series data Compress the time-series data by partitioning the data into certain range and stores these on tree-format Only if sensor ID and Timestamp are key, data retrieval is fast Similar data retrieval from time-series data Dynamic Time Wrapping (VLDB 2008) Similarity matching method to allow more flexibility beyond uclid-distance Managing PMU Data Sets with Bitmap Indexes (IEEE SusTech 2014) Make bitmap index on PMU data Flexible time-series data retrieval is object 24

26 Contents Introduction Conventional approach and issue Our proposed approach: Latent similarity event search Evaluation Related works Summary & DEMO 25

27 Summary Propose latent similarity search Achieved fast performance Not need to prepare pre-defined feature data-mart by using PMU data Show evaluation results by using IEEE 179 bus model The similarity search response time achieves ~4.8 seconds if we use stored 2 years data from 179 PMUs Compared to conventional approach, response time is a little decreased but still meet the requirement (less than a few minutes) Simplify database operation in transmission system operation and achieve flexible similarity event search by integrating various types of features Makes it easy to integrate new knowledge into applications 26

28 DSS4LA Demo Decision support for grid operators with: o Wide-area situational awareness o Disturbance event recognition by high-speed historical data search o Countermeasures to prevent large-scale blackouts Hitachi, Ltd All rights reserved. 27

29