Sliding Window Based Method for Processing Continuous J-A Queries on Data Streams

Transcription

1 ISSN , CODEN RUXUEW Journal of Software, Vol17, No4, April 2006, pp DOI: /jos Tel/Fax: by Journal of Software All rights reserved J-A 1+, 1,2, 1 1,2, 1 (, ) 2 (, ) Sliding Window Based Method for Processing Continuous J-A Queries on Data Streams WANG Wei-Ping 1+, LI Jian-Zhong 1,2, ZHANG Dong-Dong 1, GUO Long-Jiang 1,2 1 (School of Computer Science and Technology, Harbin Institute of Technology, Harbin , China) 2 (School of Computer Science and Technology, Heilongjiang University, Harbin , China) + Corresponding author: Phn: , wpwang@hiteducn, Wang WP, Li JZ, Zhang DD, Guo LJ Sliding window based method for processing continuous J-A queries on data streams Journal of Software, 2006,17(4): Abstract: Sliding window join aggregation continuous queries (J-A queries for short) are often used in data stream applications The intuitive method for processing these queries is to construct steaming operator tree and execute the tree in pipeline The space cost of this method is Ο(α β), where α and β are the sizes of two sliding windows respectively To reduce the requirement of memory, which is the most expensive resource in data stream query processing system, two novel sliding window J-A continuous query processing algorithms and TC are presented in this paper, whose space cost are both Ο(α+β) Theoretical analysis and experimental results show that the algorithms are more effective Key words: : data stream; sliding window; join aggregation; continuous query ( J-A ),, Ο(α β), α,β, J-A TC, Ο(α+β), : ; ; ; : TP311 : A,, IP Web,,, Supported by the National Natural Science Foundation of China under Grant Nos , ( ); the Natural Science Foundation of Heilongjiang Province of China under Grant Nozjg03-05 ( ) Received ; Accepted

2 : J-A 741, [1 3],,,,,, [4] K, T, Kang,, unit-time-basis,, Lukasz, [6] Viglas, SHJ, Mjoin [7] Arasu [8], (, J-A ) J-A, J-A 1:, IP A, IP B, IP, : SELECT COUNT(*) FROM A[60 MINUTE], B[60 MINUTE] WHERE Asrc=Bsrc AND Adest=Bdest,A[60 MINUTE] B[60 MINUTE] A B 60 2:, A, BA (location,time,temperature)b (location,time,strength) A B 10, 40 C 06 5, : SELECT location, COUNT(*) FROM A[10 MINUTE], B[10 MINUTE] WHERE Alocation=Blocation and Atemperature>=40 and Bstrength>06 GROUP BY location HAVING COUNT(*)>5, J-A J-A,, 1 J-A 2 J-A J-A, (group by) J-A 2,, J-A J-A, 1( ) S={ s 1,t 1, s 2,t 2,, s i,t i, },s i t i 2( ) T,t>T, SW[t T:t] S T [5]

3 742 Journal of Software Vol17, No4, April 2006, t T, t S,SW[t T:t] t, 3( ) SW T K SWT K = SW[ TK T : TK ] J-A,, O(α β),, J-A : TC, 5 :COUNT,SUM,AVG, MAX MIN, COUNT,SUM AVG, TC 5, 1 Table 1 Symbol table 1 11 Symbol Meaning SWA, SWB Sliding window corresponding to stream A,B respectively Α, β Number of items in SWA, SWB respectively T Time size of sliding window SWA, SWB λ a Arrival rate of stream A M Number of hash buckets in the hash table of sliding window C Cost of accessing one item in sliding window Cw Cost of writing one item in sliding window σ Selectivity of join operator sizea, sizeb Item size of stream A, B respectively aσb Concatenation of item a and item b Join operator COUNT SUM, AVG COUNT SUM, COUNT, : J-A, K COUNT K+1 COUNT A B,, COUNT, A B, 1 1 A,B,C,D 4, A B,C D, : (B D) (A C) + (B\A) D + (D\C) B (B\A) (D\C) (A\B) (C D) (C\D) (A B) (A\B) (C\D) ( ) A,B SWA, SWB SWA,SWB TK T K, T T J A f, : :SWA,SWB,T J,f,CountValue,T :CountValue (1) Insert f into SWA (2) SetA null; SetB null; (3) f, SWA,SWB, SWA SWB SetA SetB ; (4) val1 COUNT({f} SWB); (5) val2 COUNT(SetA SWB);

4 : J-A 743 (6) val3 COUNT(SetB (SWA {f})); (7) val4 COUNT(SetA SetB); (8) CountValue CountValue+val1 val2 val3 val4; (9) Return CountValue;, COUNT(A) A f, SWA SWB TK, (1)~(3), SWA, SWB TJ,, T K CountValue = ( SWA TK SWB TK, ( SWA TJ SWB TJ ) 1 : SWA T ( SWA TJ SWB TJ ) = ( SWA TK SWB TK ) + ( SWA TJ \ SWA TK ) SWB TJ + ( SWB TJ \ SWB TK ) SWA TJ ( SWA TJ \ K ) ( SWB TJ \ SWB TK ) ( SWA TK \ SWA ) ( SWB SWB ) ( SWB TK \ SWB TJ ) ( SWA TK SWA TJ ) T J ( SWA TK \ SWA TJ ) ( SWB TK \ SWB TJ ) (1) T K : SWA \ SWA ={f} B,, SWB SWB TK, T J TK SWB TJ \ SWB TK =φ SWA TK \ SWA TJ =SetA, SWB TK \ SWB TJ =SetB (3), SWA TK SWA TJ =SWA\{f}, SWB TK SWB TJ =SWB (1), : SWA T SWB T = SWA T SWB TK + {f} SWB SetA SWB SetB (SWA\{f}) SetA SetB (2) J J K, COUNT HASH (1) (3) C w,(α+β)c SWB M, β/m ; 4 (β/m) C; (5) (6) SetA β/m C, SetB α/m C; (7),, SetA SetB C, C w +(α+β)c+(β/m) C + SetA β/m C+ SetB α/m C+ SetA SetB C T J (α+β+β/m+ SetA β/m+ SetB α/m+ SetA SetB ) C (3),,, SetA,SetB, O(α sizea+β sizeb+ SetA sizea+ SetB sizeb),seta SetB, 12 TC COUNT, TC COUNT,SUM AVG ; MAX, TC MAX MIN COUNT TC TC-COUNT TC-COUNT, SWA,SWB CV SWA,SWB a, SWA f, fcv= {g g SWB, gtimestamp>ftimestamp,ga=fa},fcv SWB f, SWB f f CV COUNT 2 SWA,SWB T; f SWA( SWB);fCV= {g g SWB( SWA);gtimestamp> ftimestamp,ga=fa}, a,timestamp T K, SWB g, f SWA TK, g fcv : T K, SWB g SWA TK f CV, CV, ga=fa gtimestamp>ftimestamp g, TK gtimestamp>t, T K ftimestamp>t, f SWA f SWA T,g fcv T K K 2,, CV CV COUNT 3 SWA,SWB T, f SWA( SWB),fCV= {g g SWB( SWA),gtimestamp T J TJ

5 744 Journal of Software Vol17, No4, April 2006 >ftimestamp,ga==fa}, : SWA SWB = fcv f SWA, SWB : r SWA SWB, r=f g,,f SWA,g SWB r, CV, ftimestamp>gtimestamp, gcv=gcv+1, fcv=fcv+1 r SWA SWB, 1 : SWA SWB = f CV f SWA, SWB f SWA,SWB TC-COUNT 2 3 COUNT TC-COUNT, T J A f,tc-count, HASH TC-COUNT :SWA,SWB,T j,f,t :CountValue (1) fcv 0;CountValue 0; (2) Insert f into SWA; (3) SWA SWB (4) SWB f HASH B (5) FOR tuple g B DO (6) IF ga==fa (7) gcv++; (8) FOR k SWA DO (9) CountValue CountValue+kCV; (10) FOR g SWB DO (11) CountValue CountValue+gCV; (12) Return CountValue; MAX TC TC-MAX TC-MAX TC-COUNT, SWA,SWB MV SWA,SWB a, B c, Bc [0,U], f SWA, fmv=max{gc g SWB,gtimestamp>ftimestamp ga= fa}; g SWB, : g c, f SWA, f timestamp > g timestamp g a = f a g MV = (4) 0, otherwise MV, 2 3, : f CV MAX{fc f SWA SWB}=MAX{fMV f SWA,SWB} (5) T J f,tc-max, HASH TC-MAX :WA,SWB,T j,f,t :MaxValue (1) fmv 0;MaxValue 0; (2) SWA SWB (3) IF f SWA (4) Insert f into SWA; (5) SWB f HASH B

6 : J-A 745 (6) FOR g B DO (7) IF ga==fa (8) gmv gc ; (9) ELSE (10) Insert f into SWB; (11) SWA f HASH B (12) FOR g B DO (13) IF ga==fa (14) gmv MAX(gMV,fc); (15) FOR k SWA DO (16) MaxValue MAX(MaxValue,kMV); (17) FOR k SWB DO (18) MaxValue MAX(MaxValue,kMV); (19) Return MaxValue;,TC-COUNT TC-MAX, CV MV, CV MV, COUNT MAX J-A, TC 5 J-A TC-COUNT TC-MAX, TC-COUNT TC-COUNT (1),, (2) C w (3) SWA,SWB, (α+β)c SWB M, β/m, (5)~(7) (β/m) C (8)~(11) (α+β)c, C w +2(α+β)C+β/M C (2α+2β+β/M) C (6) TC-COUNT CV, TC-COUNT Ο(α sizea+β sizeb+(α+β) SizeOf(Int)) (7) 2 J-A 1 J-A, ( group by) J-A J-A 21 J-A, J-A COUNT, TC G G COUNT Glist, SetA,SetB {f} SWB,SetA SWB,SetB (SWA\{f}),SetA SetB, Glist, COUNT, Glist HAVING, Glist, HAVING 22 J-A Viglas, SHJ, MJoin [7] 1, S 1 f,mjoin, f S 2, S 3,

7 746 Journal of Software Vol17, No4, April 2006, MJoin, S 1 S 2 S 3 S n 1 S n S 1 S 2 S 3 S n 1 S n Fig1 Mjoin operator 1 MJoin MJoin, J-A TC-COUNT J-A, MTC-COUNT S 1 f, MJoin r,r=f g 2 g n, g i S i,2 i n r g 2 g n g k, g k, r TC-COUNT, MTC-COUNT, CV r COUNT 1, g k CV 1, 3 : n S 1 S 2 S n = g cv (8) i= 1 g Si CV, COUNT 3, Visual C++, :CPU 24G HZ 256MB, J-A PC, TC 31 J-A ; 32 J-A 31 J-A PC, TC 2, 18 A,B : (1) ; (2) /,,3 2, PC PC,, TC,, SWA,SWB, SetA,SetB 1, TC 2,PC, ;TC COUNT, ; 3 3 PC, PC, ID, 4 3,PC ;TC CV,, ;

8 : J-A 747 Excution time (ms) PC TC Sliding window size (s) (λ a =λ h =100,M=100,σ=001) Memory used (MB) PC TC Sliding window size (s) (λ a =λ h =100,M=100,σ=001) Fig2 The effect of varying sliding window size on the algorithms execution time Fig3 The effect of varying sliding window size on the algorithms memory requirement , 20,HASH M 20, 20, 20, 500 / 5, Zipf, i 1 λ = 500 i /, 1 i 5 4 :, 2 i,pc TC ;, :, seta,setb, 4, TC ;,TC ; TC PC 5 3 : TC PC,, 3 Execution time (ms) PC TC Memory used (KB) Time elapsed (s) (T=20s, M=20, σ=005) Time elapsed (s) (T=20s, M=20, σ=005) Fig4 The effect of varying stream rate on the algorithms execution time Fig5 The effect of varying stream rate on the algorithms memory requirement J-A J-A, J-A J-A 321 J-A COUNT J-A GPC,G GTC 6 3, 100, 001,G, GPC PC TC

9 748 Journal of Software Vol17, No4, April 2006 Excution time (ms) PC TC Sliding window size (s) (λ a =λ h =100,M=100,σ=001) Fig6 The effect of sliding windows size on the algorithms that process sliding window J-A query contained groupby 6 J-A 322 J-A PC TC, COUNT J-A MPC MTC, 10, 100 /, :MTC MPC,MPC,, MTC Excution time (ms) MPC MTC Sliding window number (λ a =λ h =100,T=10s,M=100,σ=001) Fig7 The effect of sliding window number on the algorithms that process sliding window J-A query contained multi-way join 7 J-A 4 J-A,, O(α β),, J-A TC, O(α+β), TC References: [1] Motwani R, Widom J, Arasu A, Babcock B, Babu S, Datar M, Manku G, Olston C, Rosenstein J, Varma R Query processing, resource management, and approximation in a data stream management system In: Stonebraker M, Gray J, Dewitt D, eds Proc of the 1st Biennial Conf on Innovative Data Systems Research Asilomar: Online Proceedings,

10 : J-A 749 [2] Carney D, Cetintemel U, Cherniack M, Convey C, Lee S, Seidman G, Stonebraker M, Tatbul N, Zdonik S Monitoring streams a new class of data management applications In: Lochovsky FH, Wang S, Papadias D, eds Proc of the 28th Int l Conf on Very Large Data Bases Hong Kong: Morgan Kaufmann Publishers, [3] Madden S, Shah M, Hellerstein JM, Raman V Continuously adaptive continuous queries over streams In: Franklin MJ, Moon B, Ailamaki A, eds Proc of the 2002 ACM SIGMOD Int l Conf on Management of Data Madison: ACM, [4] Babcock AK, Babu S, Datar M Model and issues in data stream systems In: Popa L, ed Proc of the 21st ACM SIGACT-SIGMOD-SIGART Symp on Principles of Database Systems Madison: ACM, [5] Kang J, Naughton JF, Viglas SD Evaluating window joins over unbounded streams In: Dayal U, Ramamritham K, Vijayaraman TM, eds Proc of the 19th Int l Conf on Data Engineering Bangalore: IEEE Computer Society, [6] Golab L, Ozsu MT Processing sliding window multi-joins in continuous queries over data streams In: Freytag JC, Lockemann PC, Abiteboul S, Carey MJ, Selinger PG, Heuer A, eds Int l Conf on Very Large Data Bases Berlin: Morgan Kaufmann Publishers, [7] Viglas S, Naughton J, Burger J Maximizing the output rate of multi-join queries over streaming information sources In: Freytag JC, Lockemann PC, Abiteboul S, Carey MJ, Selinger PG, Heuer A, eds Int l Conf on Very Large Data Bases Berlin: Morgan Kaufmann Publishers, [8] Arasu A, Widom J Resource sharing in continuous sliding-window aggregates In: Nascimento MA, Özsu MT, Kossmann D, Miller RJ, Blakeley JA, Schiefer KB, eds Proc of the 30th Int l Conf on Very Large Data Bases Toronto: Morgan Kaufmann Publishers, (1975 ),,,, (1976 ),,, (1950 ),,,,,CCF,, (1973 ),,,