Multi-dimensional Selectivity Estimation Using Compressed Histogram Information*

Transcription

1 Multi-dimensionl Seletivity Estimtion Using Compressed Histogrm Informtion* Ju-Hong Lee Deo-Hwn Kim Chin-Wn Chung À Deprtment of Informtion nd Communition Engineering À Deprtment of Computer Siene Kore Advned Institute of Siene nd Tehnology ABSTRACT The dtbse query optimizer requires the estimtion of the query seletivity to find the most effiient ess pln. For queries referening multiple ttributes from the sme reltion, we need multi-dimensionl seletivity estimtion tehnique when the ttributes re dependent eh other beuse the seletivity is determined by the joint dt distribution of the ttributes. Additionlly, for multimedi dtbses, there re intrinsi requirements for the multi-dimensionl seletivity estimtion beuse feture vetors re stored in multi-dimensionl indexing trees. In the -dimensionl se, histogrm is prtilly the most preferble. In the multi-dimensionl se, however, histogrm is not dequte beuse of high storge overhed nd high error rtes. In this pper, we propose novel pproh for the multidimensionl seletivity estimtion. Compressed informtion from lrge number of smll-sized histogrm buets is mintined using the disrete osine trnsform. This enbles low error rtes nd low storge overheds even in high dimensions. In ddition, this pproh hs the dvntge of supporting dynmi dt updtes by eliminting the overhed for periodil reonstrutions of the ompressed informtion. Extensive experimentl results show dvntges of the proposed pproh.. ITRODUCTIO The dtbse query optimizer hooses n effiient exeution pln mong ll possible plns by estimting the ost of eh pln. One of the most importnt ftors for omputing the ost of pln is the seletivity, whih is defined s the rtio of the number of dt in query result to the totl number of dt in dtbse. The ury of the seletivity estimtion signifintly ffets the seletion of n effiient pln. The seletivity n be estimted using vriety of sttistis tht re ept in dtbse tlog. The sttistis for the seletivity estimtion usully pproximtes the *This reserh ws supported by the tionl Geogrphi Informtion Systems Tehnology Development Projet nd the Softwre Tehnology Enhnement Progrm 000 of the Ministry of Siene nd Tehnology of Kore. Permission to me digitl or hrd opies of ll or prt of this wor for personl or lssroom use is grnted without fee provided tht opies re not mde or distributed for profit or ommeril dvntge nd tht opies ber this notie nd the full ittion on the first pge. To opy otherwise, to republish, to post on servers or to redistribution to lists, requires prior speifi permission nd/or fee. SIGMOD 99 Phildelphi PA Copyright ACM /99/05...$5.00 dt distribution of dtbse. There re two lsses in seletivity estimtion problems ording to the dimensionlity. One is the -dimensionl seletivity estimtion nd the other is the multi-dimensionl seletivity estimtion. The estimtion of the result size of query with single ttribute predite depends on the dt distribution of the ttribute. This se is the -dimensionl seletivity estimtion problem. Regrding the multi-dimensionl seletivity estimtion, there re severl pplitions tht require it. The optimiztion of query referening multiple ttributes from the sme reltion needs it, beuse the result size of the query depends on the joint dt distribution of the ttributes tht is represented s multi-dimensionl spe PI97. So do the optimiztion of fuzzy queries for multimedi repositories CG96, F96, F98 nd dtbse rning for seleting resoures in distributed environment suh s the World Wide Web CSZS97, beuse the feture vetors of multimedi dt re stored in multi-dimensionl index trees. A vriety of tehniques were proposed bsed on how to pproximte the dt distribution. An exellent survey nd the txonomy of vrious seletivity estimtion tehniques ppered in MCS98, CR94, PIHS96. -dimensionl seletivity estimtion tehniques re lssified into four tegories: the prmetri, the urve fitting, the smpling, nd the non-prmetri. Among these lsses, the histogrm method in the non-prmetri lss is the most preferble beuse it pproximtes ny dt distribution nd requires resonbly smll storge with low error rtes. And it does not inur run-time overheds. Severl histogrm tehniques were proposed in order to redue estimtion errorspihs96. For the multi-dimensionl seletivity estimtion, severl estimtion tehniques were proposed: the method using the multilevel grid file(mlgf)wkw94, the singulr vlue deomposition(svd), Hilbert numbering, PHASED, nd MHIST PI97. These re ll bsed on histogrm tehniques. And these were proposed under the ssumption tht histogrm method is lso effiient in the multi-dimensionl seletivity estimtion s it is so in the - dimensionl se. However, the sitution of the multi-dimensionl se is very different from tht of the -dimensionl se. In order to hieve low error rtes, the size of histogrm buets must be smll. As the dimension inreses, the number of histogrm buets tht n hieve low error rtes inreses explosively. This is beuse the number of histogrm buets is in inverse proportion to the dimension th power to the normlized onedimensionl length of prtitioned multi-dimensionl buet s expressed by n eqution below. It uses severe storge overheds problem. # of buets, 0<<, dim where is the -dimensionl length of buet.

2 Therefore, it is impossible to mintin resonbly smll storge with low error rtes in high dimensions. Also it is diffiult to prtition multi-dimensionl spe into disjoint histogrm buets effiiently so tht the error rtes re ept smll. From prtil point of view, these methods nnot be used in dimensions higher thn three. Another problem is tht ll methods exept the MLGF method nnot reflet dynmi dt updtes immeditely to the sttistis for the estimtion. This leds to n dditionl overhed suh s the periodil reonstrution of sttistis for the estimtion. In this pper, motivted from the bove problems, we propose novel pproh for the multi-dimensionl seletivity estimtion. The ontents nd ontributions re s follows: Compressed informtion from lrge number of smll-sized buets is mintined using the disrete osine trnsform (DCT). This enbles low storge overheds nd low error rtes even in high dimensions. This n be hieved from the ft tht DCT n ompress the informtion remrbly. Tht is, low error rtes n be hieved by smll-sized buets nd low storge overheds n be hieved by ompressing lrge mount of histogrm buet informtion. As nother ontribution, s fr s we now, this is the first pplition in whih DCT is used in high dimensions. DCT hs been widely used in the imge nd signl proessing re usully in -dimensionl domin. Therefore, we lso extend DCT from two dimension to high dimensions. In ddition, this method hs the dvntge tht it is not neessry to reonstrut sttistis for seletivity estimtion periodilly, beuse it reflets dynmi dt updtes into the sttistis for the estimtion immeditely. An extensive set of experiments show tht the method proposed in this pper requires low storge overheds, hieves low error rtes, nd provides fst omputtions of the estimtion even in high dimensions. The pper is orgnized s follows: In Setion, we desribe - dimensionl nd multi-dimensionl seletivity estimtion tehniques s well s their dvntges nd disdvntges. In Setion 3, we introdue the disrete osine trnsform. In setion 4, we explin how disrete osine trnsform n be used in the multidimensionl seletivity estimtion. In Setion 5, we show experimentl results nd disuss them in detil. Finlly, onlusions re mde in Setion 6.. RELATED WORK First, we briefly desribe -dimensionl seletivity estimtion tehniques nd explin multi-dimensionl estimtion tehniques, nd then disuss their problems.. One-dimensionl Seletivity Estimtion Seletivity estimtion tehniques n be lssified into four tegories: the prmetri method by model funtions Chri83, the urve fitting method by generl polynomil funtions CR94, SLRD93, the smpling method HSS95, nd the nonprmetri method by histogrms Io93, IP95, PIHS96, JKMPSS98. The prmetri method pproximtes the dt distribution of n ttribute to model funtion suh s norml, exponentil, Person, Zipf funtion, nd omputes free prmeters for the model funtion under the ssumption tht the dt distribution well fits the seleted model funtion. The dvntge of this method is tht it requires little storge, inurs low omputtion overheds, nd provides urte results when the dt distribution fits the seleted model funtion. However, if the dt distribution does not fit the model funtion, the error rtes of estimtion results will be very high. And we must now priori whih model funtion fits the tul dt distribution. If the tul dt distribution does not fit ny nown model funtion, we nnot use this method. The urve fitting method ws proposed to get more flexibility thn the prmetri method. This method uses generl polynomil funtion in fitting the tul dt distribution. The dvntge of this method is tht it n pproximte ny dt distribution. However, it hs the negtive vlue problem nd the rounding error propgtion problem. So, we must be reful to use this method. The smpling method is minly used for sttistil queries tht hve ggregte funtions. It retrieves smple dt from dtbse nd pplies the smple dt to query in order to get sttistis of the query. The smpling method must te enough smple dt to hieve the desired ury. The query optimiztion tht requires frequent seletivity estimtions nnot use this method due to its high performne overheds. The histogrm method is the most ommon non-prmetri method. The histogrm method divides the dt distribution into set of smll disjoint intervls, in other words, buets, to pproximte the dt distribution, nd stores some sttistis in eh buet suh s vlue rnge nd the number of dt in buet. The histogrm method is bsed on the uniform distribution ssumption whih mens tht dt in buet re uniformly distributed. The seletivity estimtion using histogrm is s follows: First, ll buets overlpping with the query re seleted. The sttistis in eh buet is used to ompute the number of dt tht stisfy the query. The numbers of the stisfied dt from eh buet re summed up to get the finl estimtion result. The histogrm method is prtilly the most preferble mong the ones in four lsses beuse it is possible to me histogrm tht pproximtes ny dt distribution with resonbly smll storge nd low error rtes. Therefore it is widely used in mny ommeril dtbses. The histogrm method is gin lssified into vrious methods ording to how to prtition the dt distribution into buets in order to minimize the estimtion error: the Equi-width, the Equi-depth, the MxDiff, the V-optiml method, et. In the Equi-width, the widths of the buets re equl, nd the number of dt in eh buet pproximtes the dt distribution. In the Equi-depth, eh buet hs the sme number of dt, so the widths of the buets re different. In the MxDiff, there is buet boundry bewteen two djent vlues when the differene of these vlues re mong the lrgest. In the V-optiml, the sum of weighted vrines of buets is minimized. The V- optiml method hs been shown to be the most urte histogrm method IP95, JKMPSS98.. Multi-dimensionl Seletivity Estimtion The optimiztion of fuzzy queries for multimedi repositories needs multi-dimensionl seletivity estimtion tehnique. ChudhuriCG96 used the result using the orreltion frtl dimension BF95 s the seletivity estimtion. However, the seletivity using the orreltion frtl dimension is the verge of the estimtion results for the sme shpe nd size queries nd n be prtilly used in two nd three dimensions. For queries with multiple ttributes, there is n estimtion method tht uses multidimensionl file orgniztion lled the multilevel grid file (MLGF) WKW94. MLGF prtitions the multi-dimensionl dt spe into severl disjoint nodes, lled grids, tht t s histogrm buets. A new field, ount, is dded to eh grid node for sving the number of dt in the grid. The seletivity is estimted by essing grid nodes overlpping with query. This

3 method supports dynmi dt updtes beuse MLGF itself is dynmi ess method. And it urtely estimtes the result size of query. However, MLGF suffers from the dimensionlity urse BBK98 tht mens severe performne degrdtion in high dimensions. Also the method hs the mintenne overhed of MLGF. So, the method n not be pplied in dimensions higher thn three. Reently, Poosl et l. proposed severl useful methods for the multi-dimensionl seletivity estimtion PI97: The Singulr Vlue Deomposition (SVD), the Hilbert numbering, the PHASED, nd the MHIST methods. These methods re bsed on the -dimensionl histogrm method under the ssumption tht the histogrm n lso be used in the multi-dimensionl seletivity estimtion. So, these methods prtition the joint dt distribution into disjoint buets. The SVD method deompose the joint dt distribution mtrix J into three mtries U, D, nd V tht stisfy J=UDV T. Lrge mgnitude digonl entries of the digonl mtrix D re seleted together with their pirs, left singulr vetors from U nd right singulr vetors from V. These singulr vetors re prtitioned using ny one-dimensionl histogrm method so s to be used s histogrm buets of the ttributes. There re mny effiient SVD lgorithms, but the SVD method n be used only in two dimension. The Hilbert numbering method onverts the multi-dimensionl joint dt distribution into the -dimensionl one nd prtitions it into severl disjoint histogrm buets using ny one-dimensionl histogrm method. The buets mde by this method my not be retngles. Therefore, it is diffiult to find the buets tht overlp with query. The estimtes my be inurte beuse it does not preserve the multi-dimensionl proximity in - dimension. The PHASED method prtitions n n-dimensionl spe long one dimension hosen rbitrrily by ny onedimensionl histogrm method, nd repets this until ll dimensions re prtitioned. The MHIST is n improvement to the PHASED method. It selets the most importnt dimension in eh stte nd prtitions it. From the V-optiml point of view s n pplied prtitioning method in MHIST, the dimension tht hs the lrgest vrine is the most importnt dimension. The experiments in PI97 showed tht MHIST tehnique is the best mong vriety of multi-dimensionl histogrm tehniques. However, even though it produes low error rtes in -dimensionl ses, it hs reltively high error rtes in the 3-dimensionl spe (0-30 %) nd the 4-dimensionl spe (30-40%). This demonstrtes tht it is not esy to segment multi-dimensionl spes into disjoint histogrm buets effiiently. These methods nnot be used in dimensions higher thn three. In ddition, the dtbse system must reonstrut the sttistis periodilly in n environment where dt is updted frequently beuse the method do not support dynmi dt updtes. 3. DISCRETE COSIE TRASFORM The disrete osine trnsform hs been widely used in the imge nd signl proessing res usully in the -dimensionl domin beuse it hs the power to ompress informtion. However, we should use the multi-dimensionl DCT for ompressing the histogrm informtion. Therefore, we briefly desribe the definition of the -dimensionl DCT, the -dimensionl DCT nd extend them to the multi-dimensionl DCT. 3. Definition of Disrete Cosine Trnsform & For series of dt F & = (f(0),f(),,f(-)), DCT oeffiients, G = (0),),,-)), re defined s follows: (n + ) uπ u) = u f ( n) os n= 0 for u = 0, u = 0,,- u = for u 0 & F = (f(0),f(),,f(-)) is reovered by the inverse DCT defined s follows: (n + ) uπ f ( n) = u u) os, n=0,,- u= 0 -dimensionl DCT ws extended to -dimensionl DCT s follows: Let F be n M mtrix representing the - dimensionl dt nd G be the -dimensionl DCT oeffiients of F. Then the element (v) of G is given by M uv (m + ) uπ (n + ) vπ v) = f ( m, n)os os M m= 0 n= 0 M where u = 0,,M- nd v = 0,,- By the seprbility property RY90, Lim90 of the -dimensionl DCT, v) n be rewritten s follows: M (n + ) vπ (m + ) uπ v) = u v f ( m, n)os os M m= 0 n= 0 M Its inverse is s follows: M (n + ) vπ (m + ) uπ f ( m, n) = u v v)os os M u= 0 v= 0 M ow we generlize the bove to the -dimensionl DCT reursively s follows: Let F be -dimensionl dt. Let u(t)=(u,,u t ) (u,,u ) nd n(t)=(n,,n t ) (n,,n ) for <t nd u i = 0,, i -, n i = 0,, i - for i. Let G be DCT oeffiients of F. We define G(u(t)), F(u(t)) s follows: t (nt + ) utπ G( u( t)) = u G( u( t ))os t t n = 0 t t (n + ) uπ G( u()) = u f ( n,..., n )os n = 0 t (nt + ) utπ F( n( t)) = n F( n( t ))os t t u = 0 t t (n + ) uπ F( u()) = n..., u ) os n = 0 Then, -dimensionl DCT oeffiients is given by u,,u ) = G(u()). And the inverse DCT trnsform is given by f(u,,u ) = F(u()). 3. Properties of Disrete Cosine Trnsform DCT hs mny desirble properties s follows: () DCT is liner trnsform. Let F C be DCT nd α,β be the slr vlues, nd let x,y be the generl -dimensionl dt. Then the following linerity holds: F ( α x + βy) = αf ( x) + βf ( y) C () DCT is seprble. This mens tht the -dimensionl DCT n be redued to the -dimensionl DCT whih enbles the rowolumn deomposition whih is the bsis of fst lgorithms. (3) DCT preserves the energy in the trnsformed domin s Prsevl s theorem sys tht f ( n = (,..., ),..., n ) n ( (,..., ),..., n g u u u u n i, u i = 0,, i -, i=,, C C )

4 (4) DCT hs the property of energy omption. DCT redues the orreltion mong trnsformed oeffiients. This property is relted to the energy omption. Tht is, if dt djent to eh other in the dt distribution re highly orrelted, DCT n redue the orreltion between djent trnsformed oeffiients. And if the frequeny spetrum of dt distribution is sewed in whih the mgnitudes of low frequeny oeffiients re lrge while those of high frequeny oeffiients re smll, we n disrd the high frequeny oeffiients without seriously ffeting the originl dd distribution AFS93. Sine disrding the high frequeny oeffiients uses n error, we mesure this error s the men squre error (MSE). * MSE = ( f ( n,..., n ) f ( n,..., n )) ( n,..., n ) n i = 0,, i -, i=,, where f * (n,,n ) is omputed by pplying the inverse DCT with trunted DCT oeffiients. There re mny other trnsforms suh s the disrete Fourier trnsform (DFT), the Hrr trnsform, the Hdmrd Trnsform, nd the Krhunen Loeve Trnsform (KLT). They differ in energy omption nd in omputtionl requirements. From the energy omption point of view, KLT is the best trnsform. Tht is, KLT is the trnsform tht minimizes the MSE for trunted oeffiients. However KLT hs serious prtil problem. There is no omputtionlly effiient lgorithm for KLT. However, DCT hs good energy omption property s well s omputtionlly effiient lgorithms. Also the energy omption power of DCT is superior to ll other trnsforms exept KLT RY90,Lim90. Therefore DCT is most widely used in vrious pplitions. Typil pplitions of DCT re the visul telephony nd the joint photogrphi expert group (JPEG). 4. SELECTIVITY ESTIMATIO USIG DISCRETE COSIE TRASFORM As explined in Setion nd, histogrm method nnot be diretly used in the multi-dimensionl seletivity estimtion. As lterntives, we n onsider prmetri nd urve-fitting methods. The former hs the sme onstrint in multi-dimensionl spe s in the -dimensionl spe, tht is, the model funtion should fit the dt distribution in some degree. When the onstrint does not hold, the ury degrdes. The ltter uses polynomil funtion for fitting urve. But it uses n independent vrible for every dimension nd the number of oeffiients in multi-vrible polynomil funtion inreses rpidly s the dimensionlity inreses. It lso suffers from the problems of the osilltion (negtive vlues) nd rounding errors. We propose urve-fitting method using DCT. In this method we use uniform grid s histogrm buets in multi-dimensionl spe. From now, this grid is lled uniform histogrm buet. In se dt distribution is highly orrelted, DCT mes it possible for few dt items to represent the whole dt by ompressing informtion of the dt distribution. We lso n get the originl distribution by the inverse trnsformtion with low error rtes. This method solves the problem of the high storge overheds nd higher error rtes in high dimensionl spes, sine it uses lrge number of smll-sized multi-dimensionl histogrm buets while ompressing informtion from histogrm buets. There re vrious onsidertions to estimte the multi-dimensionl seletivity by using DCT: oeffiients smpling, dt distribution, DCT omputtion nd mintenne, nd seletivity omputtion. First, we onsider the effiient smpling method to selet lowfrequeny oeffiients tht hve lrge vlues. Seond, we desribe wht is the onstrint of the dt distribution to ompress the histogrm informtion effiiently. Third, we explin how to support dynmi dt updtes to reflet it to the sttistis immeditely. Fourth, we desribe how to simply lulte the seletivity estimtion. 4. Geometril Zonl Smpling The size of the histogrm buet should be mintined smll enough to get low error rte in high dimensionlity. The number of DCT oeffiients trnsformed, however, inreses exponentilly s the dimensionlity inreses. If we hoose pproprite oeffiients fter ll oeffiients re omputed, it uses severe omputtion overhed. Therefore, we must hoose nd ompute only the oeffiients tht re estimted to hve lrge vlues. To selet the pproprite DCT oeffiients, we use the -dimensionl geometri zonl smpling tehnique tht is used frequently in the re of digitl signl proessing RY90, Lim90 nd extend it to multi-dimensionl tehnique. Only those trnsformed oeffiients within speified zone re proessed, with the remining ones set to zero. This seletion orresponds to low frequeny filtering. There re severl zonl smpling tehniques: The tringulr, the reiprol, the spheril, nd the retngulr zonl smpling. Fig. 0,,, 3,...,- 0,,, 3,...,- 0,,, 3,...,- () Tringulr 0,,, 3,...,- () Spheril ()~(d) shows only -dimensionl ses of 4 geometril zonl smpling methods for esy visuliztion. The tringulr method is to selet the oeffiients within the tringle in -dimensionl se s shown in Fig.(). It selets DCT oeffiients, u,u ), suh tht the sum of u nd u is less thn or equl to given vlue b, tht is, u +u b for u =0,, - nd u =0,, -. In multidimensionl se, it selets DCT oeffiients, u,,u n ), suh n tht u b for u i = 0,, i -. We now the number of DCT i= i oeffiients by this smpling with lemm. 0,,, 3,...,- Lemm ) The number of DCT oeffiients seleted by the tringulr zonl smpling is given by n+b C min(n,b), if the ondition 0,,, 3,...,- 0,,, 3,...,- (b) Reiprol 0,,, 3,...,- (d) Retngulr Fig. Geometril Zonl Smpling in -dimensionl se

5 b i is stisfied. Tble shows vrious vlues of n nd b. b= b= b=3 b=4 b=5 b=6 n= C = 3C =3 4C =4 5C =5 6C =6 7C =7 n= 3 C =3 4C =6 5C =0 6C =5 7C = 8C =8 n=3 4 C =4 5 C =0 6 C 3 =0 7C 3 =35 8C 3 =56 9C 3 =84 n=4 5 C =5 6 C =5 7 C 3 =35 8C 4 =70 9C 4 =6 0C 4 =0 n=5 6 C =6 7 C = 8 C 3 =56 9C 4 =6 0C 5 =5 C 5 =46 n=6 7 C =7 8 C =8 9 C 3 =84 0 C 4 =0 C 5 =46 C 6 =94 Tble. The number of DCT oeffiients seleted by the tringulr zonl smpling The reiprol method is to selet the oeffiients suh tht the multiplition of indies is less thn or equl to given vlue b. n Tht is, the seletion is mde by the onstrint ( u + ) b for u i = 0,.., i -. This method hooses more high-frequeny vlues in eh dimension thn the previous method. The spheril zonl smpling method is to selet the oeffiients suh tht the sum of the squre of indies is less thn or equl to given vlue b, tht n is, u b for u i = 0,, i -. It hooses the oeffiients within i i= the re of irle in the -dimensionl se nd sphere in the 3- dimensionl se. The retngulr zonl smpling method hooses the oeffiients suh tht the mximum vlue of indies is less thn or equl to given vlue b, tht is, mx(..., un ) b for u i = 0,, i -. It hooses the oeffiients within the re of retngle. Tble shows the smpling rtio of eh zonl smpling methods. As the dimensionlity inreses, the number of oeffiients hosen by the tringulr zonl smpling nd the reiprol zonl smpling inreses slowly, while the totl number of histogrm buets inreses explosively. However, the number of seleted oeffiients by the spheril nd retngulr zonl smpling method inreses somewht rpidly. i= i 4. Dt Distributions In order to be ble to ompress gret number of histogrm buets into smll mount of informtion with low estimtion error rtes by using DCT, the dt distribution should hve ertin hrteristis. The distribution should hve high orreltion mong dt items. Tht is, the frequeny spetrum of the distribution should show lrge vlues in its low frequeny oeffiients nd smll vlues in its high frequeny oeffiients AFS93. If the dt distribution does not follow the bove hrteristis, tht is, dt re totlly independent of djent dt, we nnot hve the benefits of energy omption nd nnot redue the number of oeffiients without distorting the originl dt distribution. We believe tht dt in rel dt distribution re highly orrelted. There re mny ses tht dt re orrelted. It is nturl for the joint dt distribution of multiple ttributes from reltion to hve lusters in most ses, sine the ttributes re in generl losely dependent eh other PI97. Atully in the res lie dt mining, the tehniques to find suh lusters re prtilly used for extrting useful nowledge from lrge volume of dtbses GRS98, EKSWX98, ZRL96, H94. The lustering effet n lso be seen in multimedi dtbses lie imges nd in sptil dtbses EKSX96, SCZ98. The lrge-sized shpes of luster orrespond to lrge-vlued low frequeny oeffiients while smll-sized vritions in it orrespond to smll-vlued high frequeny oeffiients. Therefore, the men squre error between the tul dt distribution nd the distribution reovered by seleted low oeffiients is usully smll. Bsed on these observtions, we n redue the number of multi-dimensionl histogrm buets remrbly. In generl, s the sewness of dt distributions grow or the number of lusters inreses, the number of lrge-vlued high frequeny oeffiients tends to inrese. It mens more oeffiients re needed to eep low error rtes. 4.3 Dynmi Dt Updte It is importnt to reflet dynmi dt updtes to the sttistis for estimting seletivity immeditely in the environment where dt re frequently inserted or deleted. Exept the MLGF method, most of multi-dimensionl seletivity estimtion tehniques, suh s MHIST, SVD, PHASED, nd Hilbert numbering, nnot reflet dynmi dt updtes into the histogrm immeditely. In other words, when the number of dt updtes rehes ertin threshold, the histogrm should be reonstruted entirely. In # of seleted oeffiients (% rtio to # totl buets) # of totl Tringulr Reiprol Spheril Retngulr dim i buets b=6 b=4 b= b= (.%) 4(.6%) (0.44%) 6(0.64%) (0.54%) 86(0.56%) 87(0.56%) 64(0.4%) (0.4%) 53(0.3%) 305(0.6%) 56(0.5%) (0.46%) 6(0.3%) 973(0.97%) 04(%) (0.35%) 333(0.3%) 88(.%) 4096(.6%) (0.%) 477(0.058%) 8080(0.98%) 6384(%) (0.8%) 60(0.036%) 77(.3%) 65536(3.9%) Tble. The rtio of the number of seleted oeffiients by the zonl smpling to the totl number of uniform histogrm buets

6 Seletivity of query q = b b b... f ( x, x,..., x ) dx dx... dx b... g u u Z u... u u u u x dx (,...,,..., ) os( )... ) b os( u πx ) () π () dx ontrst, our proposed method n reflet dynmi dt updtes to the sttistis for estimting the seletivity with resonble overheds. This is enbled beuse DCT is liner trnsform. Its proess is s follows: When dt is newly inserted, the vlues of its DCT oeffiients re omputed nd dded into existing DCT oeffiients. In se of deletion, the vlues of DCT oeffiients of the deleted dt re omputed nd subtrted from existing DCT oeffiients. Therefore, we n immeditely reflet dt insertions nd deletions into the sttistis for estimting the seletivity by proessing only the updte dt. Exmple ) We show n exmple for -dimensionl se. Let F be the urrent uniform histogrm buets nd G be the urrent DCT oeffiients of F. Let F be some dt updtes whih represents tht one dt in (0,) nd two dt in (,) re deleted nd two dt in (,0) re newly dded. And let G be DCT oeffiients of F. Let F be the finl uniform histogrm buets nd G be finl DCT oeffiients of F. Then F = F + F nd G = G + G. 0 F = F = F 0 = DCT G = DCT G 3 DCT 4 G = = Seletivity Estimtion of Rnge Queries There re two inds of methods to ompute the seletivity of rnge query. The first method finds ll histogrm buets within the query rnge using the inverse DCT, nd then omputes the seletivity s the histogrm method does. It ssumes the uniform dt distribution within buet lie the existing histogrm methods. The seond method omputes the seletivity using the integrl of the inverse DCT funtion sine the funtion is ontinuous osine funtion. The former method needs the inverse DCT omputtion for eh buet informtion while the ltter simply omputes the seletivity without the omputtion for eh buet informtion ount sine it omputes the integrl of the inverse DCT funtion only for the intervl of the query rnge. Sine the inverse DCT funtion nturlly supports the ontinuous interpoltion between ontiguous histogrm buets, the seond method provides urte results. The following is the expression of the integrl to estimte the seletivity of rnge query. First, we show the dimensionl se nd generlize it to the -dimensionl se. Let q be -dimensionl query. The rnge of q is x b, y d, whih is represented s (~b, ~d). We ssume the dt spe is normlized s (0,) n. The x oordinte is divided into prtitions nd y oordinte is divided into M prtitions. Then i th positions of x,y (x i nd y i ) re s follows: i + i + xi =, yi = M Then we n rewrite the inverse DCT funtion f(m,n) in setion 3. s follows: M f ( x, y) = u v v)os( xvπ ) = = os( yuπ ) M u 0 v 0 d b f ( x, y) dxdy Seletivity of query q = d b M = u v v)os( xvπ ) os( yuπ ) dxdy M u= 0 v= 0 d M b = u v v) os( xvπ ) dx os( yuπ ) dy M u= 0 v= 0 d b g v Z u uv v) os( uπy) dy os( vπx) dx (, ) M where Z is the set of seleted oeffiients from zonl smpling ow, we generlize the bove integrl to the -dimensionl se. Let q be -dimensionl rnge query. The rnge of the query q is i x i b i for i, whih is represented s ( ~b,, ~b ). The x i oordinte is devided into i prtitions. Then the seletivity is expressed s formul (), (). 5. EXPERIMETAL EVALUATIO In order to mesure the ury of the proposed method in estimting the result sizes of queries, we onduted omprehensive experiments over n environment ontining vrious syntheti dt distributions nd vrious queries. All dt re generted in the normlized dt spe (0,) n. We were not ble to me detiled omprisons with the previous resultswkw94, PI97 beuse the existing methods showed high errors in high dimensions beyond 3 dimension. For exmple, MHIST shows somewht high errors in the 3-dimension (0~30%) nd the 4-dimension (30~40%), nd the MLGF method nnot be used in dimensions higher thn three. Syntheti dt re generted with 50K reords whih rnged from to 0 dimensions. We generted dt with vrious distributions:. orml distribution : The dt points follow (0,σ ) where σ = 0.4 for ~4 dimensions, σ =.0 for 5~0 dimensions.. Zipf distribution: The dt points follow the Zipf distribution where z = 0.3 for ~5 dimensions, z = 0. for 6~0 dimensions. The Zipf distribution is defined s follows:

7 z i where i =,,.., f ( i) = z z z 3. Clustered distribution: 5~5 norml distributions re overlpped in dt distribution. DCT oeffiients re lulted s follows: A multidimensionl spe is prtitioned into lrge number of uniform histogrm buets suh tht the number of prtitions in eh dimension is the sme s those of others. The totl number of buets is in proportion to the dimension th power of the number of prtitions in one dimension. In low dimensions, if the totl number of buets is not quite lrge, we red dt sequentilly nd ount the number of dt in eh buet nd store them in the rry of min memory. Then we lulte only DCT oeffiients tht re seleted by the zonl smpling using DCT. In high dimensions, sine the number of buets is very lrge, we nnot fford the memory spe for ounting the number of dt in ll buets. So, we used n X-treeBKK96 to get groups of dt tht re lose to eh other by essing nodes of the X-tree. This enbles to get the number of dt in smll group of buets t time for lulting DCT oeffiients. The seletivity estimtion method proposed in this pper is evluted for rnge queries of the form ( X b )& & ( n X n b n ), where 0 i,b i. Four sets of 30 queries were mde suh tht eh set represents different rnge of seletivity: lrge( 0.3), medium( 0.067), smll( ), very smll ( 0.003). There re two query models for the probbility distribution of queries PSTW93, BF95: the rndom model, the bised model. The rndom model ssumes tht queries re uniformly distributed in the dt spe. Tht is, every prt of dt spe is eqully liely to be queried. The bised model ssumes tht queries re more highly distributed in high-density regions. Tht is, eh dt is eqully liely to be queried. Most pplitions follow the ltter model. For exmple, in GIS pplitions, users re not liely to query the re of dessert but re liely to query populted res lie ity. In imge dtbse pplitions, most of users my browse the imges from dtbse nd pi up the most similr imge tht they wnt from the browsed imges nd serh imges similr to it. This mens tht queries re loted more frequently in dense re in the dt spe. So, we dopt the bised model s query model in these experiments. For eh query, we generted 30 bised queries. The query results re ompred with the estimtions using the proposed method in this pper. A perentge error is used for the ury of n estimtion result: Perentge error = query result size - estimted result size 00 % query result size 5. Storge Requirements nd Seletivity Estimtion Time The proposed method requires the storge of the sttistis for estimting the seletivity. The mount of the storge for the method is proportionl to the number of DCT oeffiients seleted by zonl smpling. We onvert the multi-dimensionl indies of DCT oeffiient to n one-dimensionl vlue nd vie vers. Therefore, one DCT oeffiient needs 4 bytes for storing its vlue nd 4 bytes for storing its index. 8 bytes re required for storing one DCT oeffiient. If one use 00 DCT oeffiients for estimting the seletivity, 800 bytes nd some boo eeping bytes re required. From the seletivity lultion formul (), we n estimte the the seletivity omputtion time s follows: If is the dimension nd α is the time to ompute the sine funtion, the time to ompute the seletivity is given by **α*(the number of seleted DCT oeffiients). Tble 3 shows the typil seletivity estimtion time. In Sun Ultr II, α is mesured s bout µ se. dimension # DCT= 50 # DCT = 00 # DCT = µ se 600 µ se. m se µ se. m se.4 m se µ se.8 m se 3.6 m se Tble 3. The seletivity omputtion time in Sun Ultr II It follows tht the proposed method is effiient for time nd spe. 5. Effet of Zonl Smpling The zonl smpling selets low frequeny oeffiients. Tht is, it ts s low frequeny filter. Its effetiveness n be mesured by the men squre error. But this requires ll vlues of uniform histogrm buets by the inverse DCT, whih is very time onsuming job. So, insted we mesure the effetiveness of the zonl smpling by perentge errors of queries. We me 30 queries for eh test nd verged their results. The effiieny of the zonl smpling is ffeted by distributions. We mde experiments for 3 different distributions in the 6-dimension: () orml distribution () Zipf distribution (3) Clustered 5 distribution (tht hs 5 lusters). We pply the three zonl smpling methods to these dt. We drop the retngulr zonl smpling in the 6-dimension beuse the number of seleted DCT oeffiients by retngulr zonl smpling inreses very rpidly with smll b vlue s indited in Tble. The results re shown in Fig. ~4. The results show tht the reiprol zonl smpling is the best for ll distributions. The tringulr zonl smpling method is the seond. The spheril zonl smpling showed the worst performne. However, there re some threshold fter whih there is no differene between three zonl methods. Therefore, when we use few DCT oeffiients, the reiprol zonl smpling is the best. 5.3 Effet of Dimension nd Query Size In Fig. 5~7, we show the results of vrious query sizes in vrious dimensions. Query sizes re lrge, medium, smll, very smll. The dimensions re vried s, 4, 6, 8, 0. The dt distribution is the lustered 5 distribution. We use the reiprol zonl smpling method s setion 5. shows tht the reiprol zonl smpling is the best. Fig. 5 shows the results for using only 00 DCT oeffiients. Fig. 6 for 500 DCT oeffiients nd Fig. 7 for 000 DCT oeffiients. As the dimension inreses, the error rtes inrese slightly, but the verge error of queries is below 0 %. This results show tht the method in this pper n be used for high dimensionl dt spes. As the query size is deresed, the error rtes inrese. This is nturl result beuse the perentge error is mgnified by the slight differene between n estimtion size nd query result size when the query result is smll. 5.4 Effet of Dt Distributions The dt distribution hs impts on the error rtes for estimting

8 the seletivity. Fig. 8~0 shows the results for vrious distributions. The Zipf is sewed distribution. As the dimension inrese, the sewness of the Zipf lso inrese exponentilly. Therefore, the error rtes inrese. However, s expeted, we verified the ft tht the more we use DCT oeffiients, the more urte the results re. The error rtes of the norml nd the lustered 5 distributions inrese very slightly. This mens tht the sewness of the norml nd the lustered distribution inreses very slightly s the dimension inreses. In ddition, sine the lustered distribution is the most ommon phenomenon in mny pplitions, the proposed method n be widely used in rel world. 5.5 Effet of Dt Spe Prtition A multi-dimensionl spe is prtitioned into lrge number of uniform histogrm buets. The number of DCT oeffiients is the sme s tht of the histogrm buets. But 000 DCT oeffiients tht re seleted by the tringulr zonl smpling re omputed nd sorted. To show the effets of the number of prtitioned buets, we prtition multi-dimensionl spe into severl different wys. The p in Fig. ~4 mens the number of prtitions in one dimension. We find the verge result size of 30 medium-size queries nd estimte the size of the queries with only the indited number of DCT oeffiients in the X-oordinte (numdct) in Fig. ~4. Then we lulte perentge errors. We found some interesting fts. As the number of prtitions (p) inreses, the ury lso inrese. The more DCT oeffiients we use for estimting the seletivity, the more urte the result is. There is some threshold fter whih the ury is not hnged. In 3 dimensionl se, if p=5, the threshold of the number of DCT oeffiients is 30 with less thn % error. Tht is, it is suffiient to hve 30 DCT oeffiients for estimting the seletivity with low error rtes. 6. COCLUSIO In this pper, we proposed novel pproh for estimting the multi-dimensionl seletivity. The histogrm is not dequte in high dimensions beuse the desired high ury requires smllsized histogrm buets, however we hve tremendous storge overhed s the dimension inreses. To solve this problem, we used the disrete osine trnsform whih is n informtion ompression tehnique in order to ompress the informtion of lrge number of histogrm buets. We hieved the high ury by using smll-sized buets, nd lso low storge overhed by smll mount of ompressed informtion. Extensive experiments showed the proposed method is superior to the previous ones with the following dvntges: () The previous methods ould not support multi-dimensionl seletivity estimtion, prtiulrly, more thn three dimensions. But our method supports high dimensionl seletivity estimtion with high ury. () Our method elimintes the periodil reonstrution of the sttistis for estimting the seletivity beuse it n reflet dynmi dt updtes to the sttistis immeditely. (3) Our method simply lultes the seletivity using the integrl of osine funtions. It lso lultes the estimtion urtely beuse it nturlly supports the interpoltion between the djent buets. For the future reserh, we pln to investigte the seletivity estimtion of the nerest neighbor query. 7. REFERECES AFS93 R. Agrwl, C. Floutsos, A. Swmi. Effiient Similrity Serh In Sequene Dtbses. Foundtions of Dt Orgniztions nd Algorithms Conferene, 993. BBK98 S. Berhtold, C. Bohm, H. Kriegel. The Pyrmid Tehnique: Towrds Breing the Curse of Dimensionlity. ACM SIGMOD Conferene, pp.4-53, 998. BKK96 S. Berhtold, D. Keim, H. Kriegel. The X-tree: An Index Struture for High-Dimensionl Dt. th VLDB Conferene, pp. 8-39, 996 BF95 A. Belussi, C. Floutsos. Estimting the Seletivity of Sptil Queries Using the Correltion Frtl Dimension. th VLDB Conferene, 995. CR94 C. Chen.. Roussopoulos. Adptive Seletivity Estimtion Using Query Feedb. ACM SIGMOD Conferene, pp. 6-7, 994. CSZS97 W. Chng, G. Sheiholeslmi, A Zhng, T. Syed- Mhmood. Effiient Resoure Seletion in Distributed Visul Informtion Systems. ACM Multimedi Conferene, pp. 03-3, 997. CG96 S. Chudhuri, L. Grvno. Optimizing Queries over Multimedi Repositories. ACM SIGMOD Conferene pp. 9-0, 996,. Chri83 S. Christodoulis. Estimting reord seletivities. Informtion Systems Journl, 8(), pp05-5, 983. EKSWX98 M. Ester, H. Kriegel, J. Snder, M. Winner, X. Xu. Inrementl Clustering for Mining in Dt Wrehousing Environment. 4th VLDB Conferene, pp , 998. EKSX96 M. Ester, H. Kriegel, J. Snder, X. Xu. A Density Bsed Algorithm for Disovering Clusters in Lrge Sptil Dtbses with oise. In Pro. nd Int. Conf. on Knowledge Disovering nd Dt Mining, 996. F96 R. Fgin. Combining Fuzzy Informtion from Multiple Systems. In Pro. of the 5th ACM Symposium on Priniples of Dtbse Systems, 996. F98 R. Fgin. Fuzzy Queries in Multimedi Dtbse Systems. In Pro. of the 7th ACM Symposium on Priniples of Dtbse Systems, pp. -0, 998. GRS98 S. Guh, R. Rstogi, K. Shim. CURE: An Effiient Clustering Algorithm for Lrge Dtbses. ACM SIGMOD Conferene, pp , 998. HSS95 P.J. Hs, J.F. ughton, S. Seshdri, nd L. Stoes. Smpling bsed estimtion of the number of distint vlues of n ttribute. th VLDB Conferene, 995. Io93 Y. Ionnidis. Universlity of Seril Histogrms. 9th VLDB Conferene, pp , 993. IP95 Y. Ionnidis, V. Poosl. Blning Optimlity nd Prtility for Query Result Size Estimtion. ACM SIGMOD Conferene, pp , 995. JKMPSS98 H. Jgdish,. Koud, S. Muthurishnn, V. Poosl, K. Sevi, T. Suel. Optiml Histogrms with Qulity Gurntees. 4th VLDB Conferene, pp , 998. Lim90 J.S. Lim. Two Dimensionl Signl And Imge Proessing. Prentie Hll, 990. MCS98 M.V. Mnnino, P. Ch nd T. Sger. Sttistil profile estimtion in dtbse systems. ACM Computing Surveys, 0(3), 988. H94 R. g, J. Hn. Effiient nd Effetive Clustering Methods for Sptil Dt Minig. 0th VLDB Conferene, 994. PSTW93 B. Pgel, H. Six, H. Toben, P. Widmyer. Towrds n Anlysis of Rnge Query Performne in Sptil Dt

9 Strutures. In Pro. of the nd ACM Symposium on Priniples of Dtbse Systems, 993. PIHS96 V. Poosl, Y.E. Ionnidis, P.J. Hs, E.J. Sheit. Improved Histogrms for Seletivity Estimtion of Rnge Predites. ACM SIGMOD Conferene, pp , 996. PI97 V. Poosl, Y.E. Ionnidis. Seletivity Estimtion Without the Attribute Vlue Independene Assumption. 3th VLDB Conferene, pp , 997. RY90 K.R. Ro, P. Yip. Disrete Cosine Trnsform Algorithms, Advntges, Applitions. Ademi Press, 990. SCZ98 G. Sheiholeslmi, W. Chng, A. Zhng. Semnti Clustering nd Querying Heterogeneous Fetures for Visul Dt. ACM Multimedi Conferene, pp. 3-, 998. SLRD93 W. Sun, Y. Ling,. Rishe, nd Y. Deng. An Instnt nd urte size estimtion method for joins nd seletions in retrievl-intensive environment. ACM SIGMOD Conferene, 993. WKW94 K.Y. Whng, S.W. Kim, G. Wiederhold. Dynmi Mintenne of Dt Distribution for Seletivity Estimtion, VLDB Journl. Vol.3, o., pp. 9-5, 994. ZRL96 T. Zhng, R. Rmrishnm, M. Linvry. BIRCH: An Effiient Dt Clustering Method for Very Lrge Dtbses. ACM SIGMOD Conferene, pp. 03-4, 996. \ Œ Œ } Ž šžœ ~ ŽŒ wœ x ~ Œ ž Œ Fig.. orml distribution, dimension=6, one-dimensionl prtition=0 Fig. 5. Clustered 5 distribution, number of DCT oeffiients = 00 \ Œ Œ } Ž šžœ ~ ŽŒ wœ x ~ Œ ž Œ Fig. 3. Zipf distribution, dimension=6, one-dimensionl prtition=0 Fig. 6. Clustered 5 distribution, number of DCT oeffiients = 500 \ \ \ \ \ Œ Œ } Ž šžœ ~ ŽŒ wœ x ~ Œ ž Œ Fig. 4. Clustered 5 distribution, dimension=6, one-dimensionl prtition=0 Fig 7. Clustered 5 distribution, number of DCT oeffiients = 000

10 yš Œ n \ \ h^ h h h \ Fig. 8. number of DCT oeffiients = 00 Fig.. dimension = 3, Clustered 5 distribution Query size = medium yš Œ n h h h\ h h ^ b \ d b ^ \ Fig. 9. number of DCT oeffiients = 500 yš Œ n Fig. 0. number of DCT oeffiients = 000 Fig.. dimension= 5, Clustered 5 distribution, Query size=medium h h hb h h ^ ^ ^ b Fig. 3. dimension = 7, Clustered 5 distribution, Query size= medium \ \ ^ h h ^ b \ Fig. 4. dimension = 0, Clustered 5 distribution, Query size = medium