StreamOLAP. Salman Ahmed SHAIKH. Cost-based Optimization of Stream OLAP. DBSJ Japanese Journal Vol. 14-J, Article No.

Size: px

Start display at page:

Download "StreamOLAP. Salman Ahmed SHAIKH. Cost-based Optimization of Stream OLAP. DBSJ Japanese Journal Vol. 14-J, Article No."

Anne Oliver
5 years ago
Views:

1 StreamOLAP Cost-based Optimization of Stream OLAP Salman Ahmed SHAIKH Kosuke NAKABASAMI Hiroyuki KITAGAWA Salman Ahmed SHAIKH Toshiyuki AMAGASA (SPE) OLAP OLAP SPE SPE OLAP OLAP OLAP Due to the increase of stream data sources, many Stream Processing Engines (SPEs) have been developed and deployed. Besides simple data filtering, demands for more sophisticated analysis of streams such as multi-level aggregation and summarization are increasing. OLAP is one of the common methods for systematic analysis of static data and has been studied intensively. Some existing works proposed application of OLAP to stream data. However, they lack some required features and do not consider use of SPEs for stream OLAP. Moreover, cost-based analysis to improve efficiency of stream OLAP has not been studied before. In this paper we present a stream OLAP architecture consisting of an SPE and an OLAP engine as a general framework for stream OLAP. Then, we propose a cost-based optimization scheme. To the best of our knowledge, this is the first proposal for cost-based optimization of stream OLAP. Moreover, we have implemented a runnable prototype system, and nakabasami@kde.cs.tsukuba.ac.jp kitagawa@cs.tsukuba.ac.jp salman@kde.cs.tsukuba.ac.jp amagasa@cs.tsukuba.ac.jp evaluated the proposed optimization scheme. The experimental results prove the effectiveness of the proposal. 1. OLAP [1] OLAP OLAP lattice lattice ( OLAP ) OLAP (SPE) SPE STREAM [2]Storm [3]S4 [4]Borealis [5] SPE SQL CQL [6] OLAP Jiawei Han OLAP Stream Cube [7] SPE SPE OLAP OLAP OLAP StreamOLAP StreamOLAP lattice CQL SPE OLAP lattice SPE lattice OLAP SPE (On-demand Query) OLAP OLAP [1] OLAP 1 Vol. 14-J, Article No. 3,

2 µ d d d d d dee e ede edd dedd d d 1: d ede ded ded d ee edd ded ede d eee ded ded d de eee dee de edd dd eed ed dee ee eed edd ede ded dde dee eee edd ede edd edd dde eed edd edd ee µ 2: drill-down roll-up drill-down roll-up TPC-H customerpartsupplier lineitem customerpartsupplier lineitem lineitem sales customer roll-up customer nation supplier drill-down supplier 2. 2 (SPE) (SPE) SPE STREAM [2]Storm [3] S4 [4]Borealis [5] SPE CQL [6] ( ) SPE OLAP SPE StreamOLAP SPE OLAP SPE part (sales) customer (nation) SPE Select part_, customer_nation, Sum(sales) From lineitem [Range 1 hour] Group By part_, customer_nation 3. OLAP Aouiche View [8]Talebi View OLAP [9]Santos OLAP partitioning OLAP [10]Harinarayan OLAP lattice [11]Joslyn lattice View [12] OLAP Jiawei Han Stream Cube [7]Stream Cube OLAP OLAP OLAP OLAP SPE Rui Zhang IP SPE Gigascope [13] Gigascope IP Stream OLAP lattice lattice OLAP SPE 4. StreamOLAP OLAP 4. 1 OLAP. StreamOLAP lattice partsuppliertime 3(a) 3(b) 3(c) lattice root (p s minute) (p s hour) 2 Vol. 14-J, Article No. 3,

3 (a) part (b) supplier minute hour (c) time (p_, s_, minute) (p_, minute) (minute) (s_, minute) (p_, s_, hour) (p_, hour) (hour) (d) their lattice (s_, hour) 3: part, supplier time lattice (p minute) (p hour) lattice 3(d) lattice lattice OLAP OLAP 3(d) lattice (p minute) OLAP (p s minute) OLAP OLAP OLAP OLAP OLAP (Interval of Interest (IoI)) IoI OLAP IoI OLAP StreamOLAP 4 StreamOLAP SPE OLAP SPE OLAP SPE IoI OLAP (SPE) SPE CQL 4 (part, supplier, time) (sales) 3 OLAP 4 CQL [6] SPE 4 CQL (s, minute) OLAP s minute sales SPE window 4 1 window 1 OLAP (s, minute) RSTREAM [6] window 4 s minute 1 CQL 1 StreamOLAP SPE OLAP OLAP a) Registered Query b) On-demand Query OLAP Registered Query Registered Query OLAP CQL SPE CQL OLAP OLAP CQL OLAP IoI IoI Registered Query OLAP OLAP 4 (s, minute) Registered Query IoI :00 9:59 10:00 9:00 IoI On-demand Query OLAP lattice Registered Query CQL SPE OLAP 3 Vol. 14-J, Article No. 3,

4 Stream Data (part, supplier, time, sales) 1 min d, A, 10:00:47, 300 c, B, 10:00:25, 500 a, A, 10:00:08, 100 Aggregation B, 10:00, 500 A, 10:00, 400 Window Stream Processing Engine Register Query RSTREAM [1 minute] ( SELECT s_, minute, SUM(sales) FROM stream [RANGE 1 minute] GROUP BY s_, minute New result preserved (p_, s_, minute) (p_, minute) (s_, minute) registered query (minute) (p_, s_, hour) (p_, hour) (s_, hour) (hour) C, 09:59, 720 B, 09:59, 280 C, 09:58, 300 A, 09:59, 100 A, 09:58, 560 Aggregation C, 09, B, 10, 500 B, 09, A, 10, 400 A, 09, OLAP Engine 1 hour Data Buffer C, 09:00, 120 B, 09:00, 500 A, 09:00, 400 Result Result Old result deleted User on-demand query 4: Stream OLAP On-demand Query OLAP Registered Query OLAP OLAP lattice () Registered Query 4 (s, hour) OLAP Ondemand Query Registered Query (s, minute) minute IoI 30 hour IoI 1 (s, hour) (s, minute) lattice IoI IoI OLAP Registered Query 4 (p, s, minute) (p, s, hour) Registered Query Ondemand Query Registered Query OLAP Registered Query On-demand Query OLAP q i ( ) p i OLAP q i p i q i Registered Query On-demand Query Registered Query Registered Query q i OLAP C R 1 OLAP q i CQL window w i CQL k i w i IoI k i k i Registered Query q i C R k i w i (1) On-demand Query On-demand Query q i Registered Query q r q ri 1 C O q ri IoI I ri q ri Ir i kr i w ri q i C O Ir i kr i w ri q ri CQL w ri k ri On-demand Query q i p i q i 4 Vol. 14-J, Article No. 3,

5 C O Ir i k ri p i w ri (2) On-demand Query q i Registered Query q ri Registered Query q r OLAP Registered Query n On-demand Query m StreamOLAP C C = C R n i=0 k i w i + C O m i=0 I ri k ri p i w ri (3) SPE OLAP SPE Registered Query q i 1 s i S S = n i=0 I i k i s i w i (4) 5. 3 lattice OLAP Registered Query On-demand Query lattice Q lattice root () r On-demand Query Registered Query r Registered Query Registered Query OLAP lattice Q out Q r Q o Registered Query On-demand Query Q r lattice root r Q o C pre processcost(q r, Q o ) ( 3) Q o 2 Registered Query Registered Query Registered Query Q r On-demand Query Q o C C C pre Bene f it Q o Bene f it Q r Q r C mins eqc Q r S storagecost(q r ) ( 4) Registered Query S max Q r Q o Registered Query OLAP mins eqc (root r) C Q out Algorithm1 Algorithm 1: Optimization Algorithm Input: a set Q of vertices of the lattice, the root vertex r of the lattice (r Q) Output: a set Q out of vertices (Q out Q) begin mins eqc newarray() S 0, Q r {r}, Q o Q\Q r while sums > S max do C pre processcost(q r, Q o ) Bene f it max, C min +, q min null for q Q o do Q r Q r {q}, Q o Q\Q r C processcost(q r, Q o) Bene f it C pre C if Bene f it > Bene f it max then Bene f it max Bene f it C min C, q min q mins eqc.add ((q min, C)) Q r q min, S spacecost(q r ) min cost minimum cost in mins eqc, Q out {r} for i = 0 to mins eqc.length() do if mins eqc[i].c = min cost then Q out Q out {mins eqc[i].d} break Q out Q out {mins eqc[i].d} 5. 4 partsuppliertimesales 4 partsuppliertime sales 3 lattice 1 1: OLAP Aggregate Dimension w i k i p i s i p, s, minute p, minute s, minute minute p, s, hour p, hour s, hour hour I (IoI ) OLAP 3600 S max 1,000,000,000 OLAP (lattice ) Registered Query root (p, s, minute) Registered Query Q o 5 Vol. 14-J, Article No. 3,

6 (p, minute) Registered Query C C R = 2C O Registered Query (p, s, minute) On-demand Query (s, minute) (p, s, hour) (s, hour) Registered Query (p, minute) On-demand Query C = / / / / / / / / 1 = On-demand Query Registered Query C 2 2: 1 C p, minute s, minute minute p, s, hour p, hour s, hour hour C (p, s, hour) root Registered Query Registered Query (s, minute) (p, minute) Registered Query (p, minute) Registered Query S 1,235,280,000 S max Registered Query Registered Query Registered Query (p, s, minute) (p, s, hour) (s, minute) 35,646,000 15,396,0836,401,083 Registered Query Registered Query StreamOLAP 4.2 StreamOLAP SPE OLAP SPE SPE () ucosminexus Stream Data Platform [14] OLAP Java Registered Query OLAP On-demand Query OLAP CUSTOMER custkey nationkey nation regionkey region TIME time minute hour LINEORDER orderkey custkey partkey suppkey linenumber datetime extedprice PART partkey mfgr brand type SUPPLIER suppkey nationkey nation regionkey region 5: nation region (a) customer mfgr brand (b) part type nation region (c) supplier minute hour (d) time 6: TPC-H [15] Patrick O Neil [16] TPC-H lineitem order lineorder lineorder lineorder 6018 SPE 5 6 Registered Query Ondemand Query 2 Frequency-based: Registered Query q i OLAP p i Random: Registered Query OLAP q i p i 6 Rand: OLAP q i [0, 1] p i ( p i = 1 1 ) AllHigh: OLAP [0.8, 1] p i AllLow: OLAP [0, 0.2] p i CoarseHigh: OLAP OLAP p i = 1 lattice Vol. 14-J, Article No. 3,

7 Processing time (s) Processing time (s) : (I (1)) 8: (I (2)) Proposed Frequency_based Random Proposed Frequency_based Random FineHigh: OLAP root p i = 1 root OneDimHigh: root 3 (p, s, minute) (p, s, hour) p i = 1.0 (p, minute) (s, minute) (p, hour) (s, hour) p i = 0.8 (minute) (hour) p i = 0.6 customer I (IoI ) 2 I I (1): OLAP I = 60 I (2): I = 60 I = 240 ( 3) Registered Query On-demand Query 1 C R C O C R = 5C O Registered Query Ondemand Query Registered Query OS: CentOS 6.5 (Linux el6.x86 64) : 31.1GB CPU: Intel (R) Core (TM) i : 3.40GHz StreamOLAP lineorder tuple/s S max 1GB I (1) 8 I (2) Frequency based Random x 6 y Registered Query ( ) On-demand Query Proposed Frequency basedrandom Frequency basedrandom CoarseHighFineHigh Registered Query SPE 1GB 7. SPE OLAP StreamOLAP SPE OLAP OLAP OLAP OLAP SPE SPE IoI OLAP [] [] [1] S. Chaudhuri et al., An Overview of Data Warehousing and OLAP Technology, ACM SIGMOD Record, 26 (1), pages 65-74, March [2] A. Arasu et al., STREAM: The Stanford Data Stream Management System, Technical Report, Department of Computer Science, Stanford University, [3] Storm, [4] L. Neumeyer et al., S4: Distributed Stream Computing Platform, In Data Mining Workshops (ICDMW), 2010 IEEE conference on, pages , 13 Dec [5] D. J. Abadi et al., The Design of the Borealis Stream Processing Engine, In Second Biennial Conference on 7 Vol. 14-J, Article No. 3,

8 Innovative Data Systems Research (CIDR 2005), Asilomar, CA, Jan [6] A. Arasu et al., The CQL continuous query language: semantic foundations and query execution, The VLDB Journal, 15 (2): , [7] J. Han et al., Stream Cube: An Architecture for Multi- Dimensional Analysis of Data Streams, Distributed and Parallel Databases, 18 (2): , Sep [8] K. Aouiche et al., Data Mining-based Materialized View and Index Selection in Data Warehouses, Journal of Intelligent Information Systems, Vol. 33, Issue 1, pages 65-93, August, [9] Z. A. Talebi et al., Exact and Inexact Methods for Selecting Views and Indexes for OLAP Performance Improvement, Proc. 11th International Conference on Exting Database Technology (EDBT 2008), pages , Nantes, France, March, [10] R. J. Santos et al., PIN: A Partitioning & Indexing Optimization Method for OLAP, Proc. 9th International Conference on Enterprise Information Systems (ICEIS 2007), Funchal, Madeira, Portugal, June, [11] V. Harinarayan et al., Implementing Data Cubes Efficiently, Proc ACM-SIGMOD Int. Conf. Management of Data (SIGMOD 1996), pages , ACM, Montreal, Canada, June [12] C. Joslyn et al., View Discovery in OLAP Databases through Statistical Combinatorial Optimization, Scientific and Statistical Database Management Lecture Notes in Computer Science, Vol.5566, 2009, pages [13] R. Zhang et al., Multiple Aggregations Over Data Streams, Proc ACM-SIGMOD Int. Conf. Management of Data (SIGMOD 2005), pages , ACM, Baltimore, Maryland, USA, June 14-16, [14] ucosminexus stream data platform, sdp/. [15] TPC-H, [16] P. O Neil et al., The Star Schema Benchmark and Augmented Fact Table Indexing, First TPC Technology Conference (TPCTC 2009), Lyon, France, Aug [17] J. Han et al., Efficient Computation of Iceberg Cubes with Complex Measures, In Proc ACM-SIGMOD Int. Conf. Management of Data (SIGMOD 01), pages 1-12, Santa Barbara, CA, May Kosuke NAKABASAMI OLAP Hiroyuki KITAGAWA ACMIEEE Salman Ahmed SHAIKH 2005 Mehran University of Engineering and Technology 2008 Communication Systems and Networks 2014 (DBSJ)International Association of Computer Science and Information Technology (IACSIT)Pakistan Engineering Council (PEC) Toshiyuki AMAGASA Web ACM IEEE 8 Vol. 14-J, Article No. 3,

A Comparison of Memory Usage and CPU Utilization in Column-Based Database Architecture vs. Row-Based Database Architecture

A Comparison of Memory Usage and CPU Utilization in Column-Based Database Architecture vs. Row-Based Database Architecture By Gaurav Sheoran 9-Dec-08 Abstract Most of the current enterprise data-warehouses