Optimal Data Compression Techniquesfor Smart Grid and Po

Transcription

1 Optimal Data Compression Techniques for Smart Grid and Power Quality Trend Data Faculty of Mechatronics Technical university in Liberec Research and development department KMB systems, s.r.o. ICHQP 2012, June 2012

2 Motivation: Smart grids, buildings, vehicles, energy producers and consumers > all generate an extreme amount of sensitive data comunication and storage is time/space demanding lossy algorithms are effective but these unpredictably deform the reality archived information is mostly jung - doesn t contain many useful features Project target: Format for efective storage and processing of PQ data (PQDIF, XLS or CSV is NOT storage friendly!) Is it possible to optimise the already best performing classical compression algorithm?

3 Motivation: Smart grids, buildings, vehicles, energy producers and consumers > all generate an extreme amount of sensitive data comunication and storage is time/space demanding lossy algorithms are effective but these unpredictably deform the reality archived information is mostly jung - doesn t contain many useful features Project target: Format for efective storage and processing of PQ data (PQDIF, XLS or CSV is NOT storage friendly!) Is it possible to optimise the already best performing classical compression algorithm?

4 Motivation: Smart grids, buildings, vehicles, energy producers and consumers > all generate an extreme amount of sensitive data comunication and storage is time/space demanding lossy algorithms are effective but these unpredictably deform the reality archived information is mostly jung - doesn t contain many useful features Project target: Format for efective storage and processing of PQ data (PQDIF, XLS or CSV is NOT storage friendly!) Is it possible to optimise the already best performing classical compression algorithm?

5 Tabulka : Compression ratios - assorted competitors Instrument(Mfg) File Original [kb] ZIP [kb] CR LZMA [kb] CR PQBox Sample PAS (Satec) Sample Sample Sample Sample MiQen (Iskra) MC760 Demo X4G_Demo PQInvest. (ELSPEC) Main 22KV ENA (ELCOM) QNA (Circutor) SPQ SPQ GridViz (Janitza) c c

6 Tabulka : Actual situation: RETIS, CETIS and ENVIS ratios File Original [kb] ZIP [kb] CR LZMA [kb] CR RETIS SMZ-Test SMZ-T CETIS Test Demo ENVIS 1.0 KMB SMY-Demo

7 SMPQ analyser: capacity of internal archives 256 MB Main archive (light blue) 15 MB PQ events (red) 11 MB PQ main archiv (blue) 1,5 MB LOG,1 MB HDO (RCS) (green) 0,7 MB daily profiles, 0,1 MB electricity meter (black) 1 MB reserve (for invalid flash blocks) (pink) PQ Event Trend and Ocilogram (saves to RAM)

8

9 Distribution of values of various measured quantities in the archived files.

10 Distribution of U, I, P a Q in a sample archive.

11 Distributions of values of selected harmonicks

12 vlastnosti komprimovaných dat Voltage harmonics absolute. Jan Kraus, Pavel Šte pán, Leoš Kukac ka relative to limits->

13 Selected loss-less compression techniques tested entropy encodings Huffman, Arithmetic, Range vocabulary methods (LZ77) DEFLATE 1, LZO combined methods BZIP2, LZMA 1 ZIP, GZIP

14 Type of compression and size of input. Voltage and current TRMS and harmonics included.

15 CR: LZ77 (ZIP) and LZMA versus input data size. Size of input versus CR for voltage (left) and active power (right) for ZIP and LZMA compression.

16 LZMA CR stability for charakteristical quantities.

17 LZMA compression of harmonics (U, orig. 700 kb)

18 LZMA by experiments turned out to be the best berforming algorithm for PQ trends compression. Builds on dictionary compression LZ77, extended with bit prediction in stream and range encoding of input data. Its good for application of prediction models to reduce symbol code variability in the compressed file: dictionary: higher propability to hit the term in dictionary entropy encoder: less symbols improve its efficiency

19 Modeling and optimization of a time serie.

20 Involved modeling techniques: relational models interval characteristics polynomial models spline models transform models (FFT, DWT) Evaluated model optimizations: effective encoding of values value ordering, differential encoding definition, size and distribution of modelling intervals local optimization dynamic interval length - RMSE, LogHist

21 Involved modeling techniques: relational models interval characteristics polynomial models spline models transform models (FFT, DWT) Evaluated model optimizations: effective encoding of values value ordering, differential encoding definition, size and distribution of modelling intervals local optimization dynamic interval length - RMSE, LogHist

22 Polynomial models - oscilating nature, not useful

23 Model of relations: Tabulka : CR for Pfh, Q fh modeled with P h, Q h. KMB1 P fh 1 P fh 2 P fh 3 Q fh 1 Q fh 3 Q fh 3 CR [] Ori Mdl Tabulka : CR improvement for LZMA while calculating U LL from U LN U LL [kb] U LN [kb] Sum Data CR U1 U2 U3 U12 U23 U31 [kb] [] KMB-1d Org Mdl FOS1 Org Mdl

24

25 Spline models: Various spline models. Green is a residuum for Akima spline model.

26 CR influences by number of nodes in model (Voltage, 40kB)

27 CR influences by number of nodes in model (power, 40kB)

28

29 Modely a Metriky - Běžně používané vztahy Residual Sum of Squares RSS = n = n (f (x i ) y i ) 2 ei 2 i=1 i=1 Root Mean Square Error RMSE = n MSE = (f (x i ) y i ) 2 1 n Mean Average Error n MAE = 1 n f (x i ) y i i=1 i=1

30 Akima Spline Model: removal of node and its influence in the surrounding intervals.

31 Proposed LogHist metric - motivation and definition For CR number of bits per symbol is crucial. LogHist splits symbols by code of its MSB. Sum of symbols in each group, multiplied by groups weight represents the weight of the evaluated interval and still is much less computationaly demanding than any LZ compression. Max LogHist = Hist i (int)(log N (i) + 1), i=0 where i is symbol code (ASCII, UTF), N defines (logarithmic) order of histogram bin and Hist i is a number of occurencies of symbol i in evaluated interval.

32 LogHist is not limited by number of input intervals.

33

34 Overview and Compression - key technique for savings, potentialy faster processing» not yet broadly accepted Different algorithms and data was benchmarked - specifics and limits have been identified Data from an existing implementation of SMPQ analyser have been used in our experiments - EN 50160(S) LZMA algorithm most effective existing solution for real applications. And it is still possible to improve it... Some described techniques are already implemented in ENVIS app: (SQL DB, CEA)

35 LZMA optimalizations Order and encoding is important for CR Before compression use any technique to limit the distribution of symbols Solution: prediction modeling + residuum Best results spline, agregation model and relation model s (interval spliting): LogHist metrics for affective buildup of models LZMA + agregation model with LogHist intervals + gradient optimization + diferential encoding

36 Emerging topics cluster examples Local model optimization towards the actual dictionary Timewise DB aggregation - ready in the next ENVIS software update Dynamic visualisation - use model instead of raw data Use modelling for data mining, statistical analysis and clustering

37 Thanks for your kind attention. I ll be glad to hear your opinions. Any comments regarding the advanced evaluation of PQ data, variable load/supply monitoring, general energy efficiency evaluation and archive processing is of my primar interrest. I will be also glad to later discuss possibilities for automatic data analysis and data mining in our "smart"world. Contact: Jan Kraus, TU in Liberec, jan.kraus@tul.cz

38 Harm. components of voltage, relative to EN limit. absolutne Jan Kraus, Pavel Šte pán, Leoš Kukac ka

39 3D scatterplot - cluster P, Q, a THD i. R and D

40 Scatter 4 quantities: P, Q, THD i, čas. R and D