Big Data Access, Analytics and Sense-Making

Size: px

Start display at page:

Download "Big Data Access, Analytics and Sense-Making"

Alvin Stokes
6 years ago
Views:

1 Workshop on Data Analytics for the Smart Grid Pullman, WA August 28, 2017 Big Data Access, Analytics and Sense-Making Zhenyu (Henry) Huang, Ph.D., P.E., F.IEEE Laboratory Fellow/Technical Group Manager Pacific Northwest National Laboratory (PNNL)

information Velocity 1 sample / 4 seconds 30-120 samples / second ~200x faster Volume 8 terabytes / year

2 Deployment of a vast new phasor network is generating unprecedented real-time data April 2007 March 2015 Today SCADA data Emerging phasor data Improvement Variety voltage + current + phase angle, more information Velocity 1 sample / 4 seconds samples / second ~200x faster Volume 8 terabytes / year 1.5 petabytes / year ~200x more data Veracity unseen ms-oscillations oscillations seen at 10ms greater accuracy 2

3 Smart devices and 2-way communication offer new opportunities, greater complexity 60,000 Smart Meters 3

4 More diverse data add to the complexity Weather/climate data (e.g. PNNL ARM* Data) 300 instruments, 2000 data streams 24/7 500 GB/day rising to multiple TBs/day Curating 20 years data Market/business data Cyber/communication data Simulated data *ARM: Atmospheric Radiation Measurement Each contingency scenario generates 0.5M bytes data, adding up to TB scale Contingency Analysis Number of scenarios Serial computing on 1 processor WECC N-1 (full) 20,000 4 hours WECC N-2 (partial) 153, hours Parallel computing on 512 processors ~30 seconds 469x speed up ~3 minutes 492x speed up Parallel computing on 10,000 processors ~12 seconds 7877x speed up 4

5 Data volume comparison: grid vs. big science data KB 10 6 MB 10 9 GB TB PB

Making data accessible is a big challenge Organizing and converting data to application specific formats SQL Tables NoSQL CSV PI PTI Text files Time series XML JSON Redundancy: Underlined

6 Making data accessible is a big challenge Organizing and converting data to application specific formats SQL Tables NoSQL CSV PI PTI Text files Time series XML JSON Redundancy: Underlined steps has to be performed by every application for each type of data Operational or Planning Applications 1. Connect to n data sources 2. Get Data * n 3. Combine * n 4. Transform * n 5. Analyze

7 Making data accessible is a big challenge Organizing and converting data to application specific formats SQL Tables NoSQL CSV PI PTI Text files Time series XML JSON GOSS 1. Connect to n data sources 2. Get Data 3. Combine 4. Transform Operational or Planning Applications 1. Connect to GOSS 2. Request Data 3. Analyze GOSS = GridOPTICS Software System,

8 GOSS TM : link data to applications 8

9 Analytical Challenges in multi-domain datadriven reasoning 9

10 TeraFLOPS Computational challenges in keeping up with data cycles Dynamic state estimation: computational performance achieved 30ms target for regional systems (1000s buses). WECC-size system (16,000 buses) a 100 TF problem with 48ms performance on 3,000 cores. Remaining bottleneck is communication. Memory bandwidth advancements expect to meet the 30ms target Computational time per estimation step Mar Oct 2012 June 2014 Oct 2012 June Seconds ms

Mathematical challenges in handling non- Gaussian noise in power grid measurement 120 Histogram (pvalue=0.00%) pvalue=0.00% 100 80 Frequency 60 40 20 Noise extracted from PMU 0-0.22-0.2-0.18-0.16-0.

11 Mathematical challenges in handling non- Gaussian noise in power grid measurement 120 Histogram (pvalue=0.00%) pvalue=0.00% Frequency Noise extracted from PMU Measurements Noise property analysis Current data applications assume Gaussian noise. New mathematics needs to be developed and adapted for handling non-gaussian noise, such as Particle Filters, Gaussian Mixture Methods. 11

Advanced visualization for improving hydro state awareness

Develop new, novel visualization techniques and paradigms for

and integrating new data visualizations Current data display in

12 Advanced visualization for improving hydro state awareness (Hydromap) Modernize displays for hydro planning and operations Develop new, novel visualization techniques and paradigms for analyzing dynamic data Develop modular framework for deploying and integrating new data visualizations Current data display in need of modernization Interactive hydro map as a new visualization paradigm 12

Multi-dimensional wind visualization (Glyphs) Wind Visualization/Wind Forecast Visualization Wind visualization showing multidimensional data in glyphs Wind speed (length of tail) Wind

13 Multi-dimensional wind visualization (Glyphs) Wind Visualization/Wind Forecast Visualization Wind visualization showing multidimensional data in glyphs Wind speed (length of tail) Wind direction (angle of tail) Generation (size of head) Uncertainty (color of tail) Forecast variability Wholesale price Capacity Last hour generation SCE error code Generation difference from forecast 13

14 Historical hydro view using radial visualization Compares hydropower generation across different projects along Columbia and Snake rivers Alternative view shows generation across different sources such as hydropower, nuclear, renewables, and miscellaneous sources 14

Generation Processing Submission Processing

15 Data repository for public hosting Data Repository Web Portal User Data Repository Existing models & scenarios Download Processing Download request Use existing datasets Validated models & scenarios User-generated models & scenarios Generation Processing Submission Processing Case configuration 3 rd -party datasets Generate new datasets Submit datasets Anonymized Data Case Published Dataset Generation/ Anonymization Methods Data Tools New Case Request Private Datasets Dataset Metrics/ Validation Data Generation 15

16 Summary Grid data complexity is increasing with big volumes, diverse types, and various attributes. Such complexity poses significant challenges in data access, transformation, analytics, sense making. Math, computing and visualization technologies need to be developed to meet these challenges. GOSS as a big data platform. Multi-domain data reasoning and high performance computing. Modular visualization for information presentation. 16

17 Acknowledgement PNNL Researchers: (Data and Computing) Bora Akyol, Poorva Sharma, Steve Elbert, Shuangshuang Jin, Bruce Palmer, George Chin; (Power Engineering) Ruisheng Diao, Yousu Chen, Mark Rice, Shaobu Wang, Karen Studarus Former PNNL Researchers: Terrence Critchlow, Ning Zhou, Ning Lu, Pengwei Du Funding support by: PNNL Future Power Grid Initiative (FPGI) DOE Office of Electricity Delivery and Energy Reliability (OE) Advanced Grid Modeling Program DOE Grid Modernization Initiative DOE Advanced Scientific Computing Research (ASCR) Applied Math Program DOE Advanced Research Program Agency Energy (ARPA-E) Bonneville Power Administration (BPA) 17

Questions? Further Information: GridOPTICS: http://gridoptics.pnnl.gov/ GridOPTICS Software System (GOSS): https://github.

18 Questions? Further Information: GridOPTICS: GridOPTICS Software System (GOSS): Interactive Visualization and Demo Center: Zhenyu (Henry) Huang, Ph.D., P.E., F.IEEE Laboratory Fellow/Technical Group Manager Pacific Northwest National Laboratory 18

DOE GMLC Project Computational Science for Grid Management Computational Framework for Grid Applications

DOE GMLC Project Computational Science for Grid Management Computational Framework for Grid Applications 5 th GridOPTICS Workshop November 9-10, 2016 Richland, WA DOE GMLC Project 1.4.18 Computational Science for Grid Management Computational Framework for Grid Applications HENRY HUANG Chief Engineer/Technical