Agile Data Streaming for Grid Applications

Transcription

1 Agle Data Streamng for Grd Applcatons Wen Zhang, Junwe Cao 2,3*, Ysheng Zhong,3, Lanchen Lu,3, and Cheng Wu,3 Department of Automaton, Tsnghua Unversty, Bejng 00084, Chna 2 Research Insttute of Informaton Technology, Tsnghua Unversty, Bejng 00084, Chna 3 Tsnghua Natonal Laboratory for Informaton Scence and Technology, Bejng 00084, Chna * Correspondng emal: jcao@tsnghua.edu.cn Abstract The grd s proposed to be the new computng nfrastructure for cross-doman resource sharng and collaboraton n vrtual organzatons. Whle most current work n grd computng communty focus on adaptablty of grd resource management, aglty s proposed n ths work as another essental requrement, amng to provde mass customzaton and personalzaton on grd computng servce provson. Ths work s focused on data streamng aspects of grd aglty. Snce volumes of data streams are usually extremely hgh but avalable bandwdth and storage are often very lmt n a grd envronment, an enablng envronment for grd data streamng applcatons s mplemented wth supports of on-demand data transfers and just-n-tme data cleanups n ths work. The mplementaton ncludes performance sensors, predctors and a scheduler, wth real-tme measurements, predcton and schedulng capabltes. The GrdFTP s utlzed and data transfer schedules are performed usng the on-the-fly adjustable GrdFTP parallelsm. Expermental results show that the data streamng envronment s agle enough to meet dynamcally changng applcaton data processng requrements and scale well regardng to storage and bandwdth usage.. Introducton Aglty s consdered as an essental feature to many complex system engneerng, e.g. busness [], manufacturng [6] and software development [9]. Aglty represents flexblty of response to dynamcally changng users requrements. Aglty provdes mass customzaton and personalzaton of servces wth extreme flexblty wthout loosng effcency. Ths work s focused on aglty for grd data streamng applcatons. Data management s one of the most challengng ssues n grd mplementaton. Whle most exstng research on data grds prefer to a brng-program-todata approach, grd applcatons such as astronomcal observatons, large-scale smulaton and sensor networks may requre brng-data-to-program supports. For example, LIGO (Laser Interferometer Gravtatonal-wave Observatory) [6,3] s generatng TB scentfc data per day and tryng to beneft from processng capabltes provded by the Open Scence Grd (OSG) [20]. Snce most OSG stes are CPU-rch and storage-constrant wth no LIGO data avalable, data streamng supports are requred n order to utlze OSG CPU resources. In a data streamng scenaro, data are transferred and processed constantly as f data were always avalable from local storage. Meanwhle, processed data have to be cleaned up to save space for the subsequently comng data. In ths work, a data streamng envronment s mplemented wth supports of on-demand data transfers and just-n-tme data cleanups. The mplementaton ncludes several modules, responsble for real-tme measurements, performance predcton, and resource schedulng, respectvely. The envronment s ntegrated wth Condor [9] for job allocaton and Globus [4] for data transfers usng GrdFTP [3]. Data transfer schedules are performed usng the on-the-fly adjustable GrdFTP parallelsm. Expermental results show that aglty of our data streamng envronment can meet dynamcally changng applcaton data processng requrements wth good scalablty. Some exstng work on data streamng management are derved from database management systems, e.g. STREAM [4], Aurora [7], NagaraCQ [8], StatStream [22], and Ggascope [0]. Most of data streamng research n context of grd computng [5,5,7,8] are applcaton specfc and schedulng ssues are not addressed. The followng software mplementaton s most smlar to the work descrbed n ths paper: Pegasus [2,2]. Pegasus handles data transfers, job processng and data cleanups n a workflow manner. Data transfers and processng are executed sequentally. In our envronment, data

2 streamng s performed smultaneously as jobs are processed. By carefully optmzng data streamng, our envronment makes requred data avalable n an on-demand and just-n-tme manner. Streamlne [,2]. Streamlne schedules streamng applcatons on HPC resources usng heurstc methods. Streamlne s bascally stll a job scheduler wth consderaton of data streams; our envronment s focused on schedulng data streamng nstead of job processng wth consderaton of system aglty. Ths paper s organzed as follows: Secton 2 provdes an overall ntroducton to data streamng for grd applcatons, whch s mplemented n detals n Secton 3, usng repertory polces, real-tme measurements, performance predcton and schedulng technques. Expermental results are ncluded n Secton 4 and the paper concludes n Secton Grd Data Streamng A detaled ntroducton to grd requrements for data streamng s provded n ths secton. Gven an example of grd applcatons, two major data management methods are proposed: on-demand data transfers and just-n-tme data cleanups, whch are mplemented n Secton Grd requrements Tradtonal grd computng applcatons prefer to a brng-program-to-data approach. Snce the cost for transferrng large mount of data s always hgh, t s natural that users try to submt ther jobs to computers wth data already avalable and only return data analyss results nstead of transferrng raw data. Grd applcatons such as astronomcal observatons, large-scale smulaton and sensor networks may requre workng n a brng-data-toprogram manner. There may be scenaros that owners of data, although ready for data sharng, may be reluctant to allow remote users to execute processng programs on stes for the sake of securty or lack of computatonal resources. For example, n the LIGO project, there are not enough computatonal resources avalable at observatores though data have to be generated from observatores. In order to utlze CPU cycles avalable from the OSG, LIGO data have to be streamed to OSG stes for processng. On the other hand, t s not always the case that transferrng large amount of data requres hgh bandwdth and local storage space. It s not necessarly ether that all data are transferred to local at one tme before the processng program s actually started. In ths case, data can be processed n the form of streams, and correspondng mplementaton requres supports of on-demand data transfers and just-n-tme data cleanups descrbed n the followng sectons On-demand data transfers It s not always the case that data should be transferred as fast as possble. For data streamng applcatons, data transfers should be on-demand nstead of spontaneous. Ths s to say, data transfers can be controlled to make requred data avalable accordng to actual data processng speeds. If data arrve too fast and cannot be processed n tme, accumulated data may requre large amount of storage space over tme; f data transfer speed s lower than processng, correspondng jobs may become dle and computatonal power s not fully utlzed. A data streamng envronment has to support multple applcatons, each requrng dfferent data types. All applcatons have to share lmt network bandwdth and storage space. For example, most OSG stes are CPU rch but storage lmt. Data transfers have to be scheduled n order to share bandwdth save local storage and maxmze CPU usage for data processng. The prevalng prncple here s enough s ok, not the faster transfers the better, nor the more data n storage the better. For data streamng applcatons, data transfers and processng take place smultaneously. The data streamng envronment ams to provde data for applcatons transparently and seamlessly as f jobs are processng data that were always avalable from local storage. Ths can be acheved va careful schedulng especally wth constrants of bandwdth and storage Just-n-tme data cleanups Besdes data transfer schedules, a cleanup procedure s ntegrated n the envronment to remove obsolete data and save storage space for subsequently comng data. Processed data should be removed from local storage as soon as possble. Just-n-tme cleanups are of great mportance n data streamng envronment snce the whole volume of data are usually extremely hgh. There maybe stuatons that multple applcatons are sharng a same set of data. In ths case, data can be only removed after all jobs are processed, otherwse the same set of data have to be transferred for multple tmes. 3. The Envronment Implementaton As shown n Fgure, mplementaton of the data streamng envronment s descrbed n ths secton, ncludng ndvdual processors for performance sensng, predcton, schedulng and executon of data transfers and cleanups.

3 Scheduler Predctor Data Transfer Job Manager Data Cleanup Sensor Sensor Sensor Storage Networks CPUs Globus GrdFTP Condor Fgure. The data streamng envronment Sensors. The grd s a dynamc envronment, where performance of resources, ncludng networks, processors and storage, are changng over tme. For example, the network s shared by many data transfers and the competton makes avalable bandwdth for each transfer vary from tme to tme, so real-tme performance nformaton s requred. Some sensors are developed to collect such nformaton, whch can be used as nputs of the performance predctor. Predctor. The predctor apples some predcton algorthms to forecast performance for a perod of tme n the future. Predcton results are nputs of the scheduler. Scheduler. The scheduler collects performance predcton results (e.g. data transferrng and processng speeds) and decdes when to transfer data, launch jobs and clean up processed data. Actual executons call external functons provded by Globus and Condor, as shown n Fgure. Our optmal goal s to make full use of computatonal resources, requrng guarantee of data provson, and mnmze bandwdth and storage consumptons. 3.. Real-tme measurements Performance nformaton s of mportance to predcton of both data transfers and processng. Ths can be acheved va real-tme performance measurements. Expermental results on GrdFTP scalablty are ncluded n ths secton as an example. We apply GrdFTP as the data transfer protocol for cross-doman data replcaton, parameters of whch can be adjusted to get optmal transferrng performance, ncludng parallelsm, TCP buffer sze and buffer sze. The parallelsm has the most drect mpact on data transfer speeds. Our experments show that the optmal rm number of data channels s between 8 and 0, as shown n Fgure 2. The curve stands for average tme of 20 experments for transferrng a data fle of 2 GB n seconds, usng dfferent parallelsms. It s obvous the data transfer speed ncreases dramatcally wth the GrdFTP parallelsm changng from to 8. When the parallelsm reaches over 0, the ncrement does not result n better performance further. It s a non-trval task to determne the proper amount of bandwdth to be allocated for each applcaton runnng n the Condor pool n terms of utlzaton and qualty of servce (QoS) satsfacton. As far as our assgnment algorthm concerns, t s transformed to set approprate GrdFTP parallelsms for applcatons. For convenence, the parallelsm s set to, 2, 4, 6 or 8 accordng to data processng speed and status of network. For example, f a certan processng program can consume data of 2 GB n 230 seconds, accordng to Fgure 2, the parallelsm can be set to 4 or 6 to guarantee data supply wth mnmum bandwdth. Average Transmsson Tme (seconds) Transmsson Parallelsm Fgure 2. Comparson of data transfer tmes usng dfferent GrdFTP parallelsms Snce the grd s a dynamc and resource sharng envronment, the same GrdFTP parameter may result n dfferent actual data transfer speed over tme, and data processng speed n the Condor pool may also vary over tme. A well set parallelsm may not always match data requrements over a long tme, whch makes t necessary to evaluate these parameters perodcally. In a tme nterval, f the GrdFTP can not supply enough data, ts parallelsm should be set to ts upper neghbors, e.g., from 6 to 8. But we must be cautous to set the parallelsm to a lower level unless the redundant transferrng speed s observed n several successve ntervals. If even the hghest parallelsm can not meet data processng requrements, the correspondng processor has to be nevtably dle and wat for more avalable data; f the data transferrng speed s hgh enough, some repertory polcy should be appled to avod data overflow.

4 3.2. Repertory polces A repertory strategy wth lower and upper lmts for each type of data s appled for the scheduler to decde the start and end of data transfers and ensure only reasonable local storage s requred. For most OSG stes, the total storage s lmt and must be shared by many types of data. On the other hand, t s meanngless to keep processed data. Repertory polces are defned to lower and upper lmts of storage space for each type of data. The lower lmt s used to guarantee that data processng can survve network collapse when no data can be streamed from sources to local storage for a certan perod of tme, whch mproves system robustness and ncreases CPU resource utlzaton. Ths s also llustrated n Secton 4 wth detaled expermental results. The upper lmt for each applcaton s used to guarantee that the overall amount of data n local storage does not exceed avalable storage space. Our strategy can be denoted as M(l, u ), where l and u stand for lower and upper lmts for data type (=,2,,n) and n s the total number of data types. Lower and upper lmts are manly used as thresholds to control start and end tmes of data transfers: when data amount scratches the lower lmt, more data should be transferred untl the amount reaches the upper lmt. Snce there are also data cleanups nvolved, data amount keeps changng and vares between lower and upper lmts Performance predcton Data transferrng and processng speeds must match each other so as to gan optmzed performance wth hgh throughput, so t s necessary to mplement realtme montorng and predcton. Onlne measurement and predcton of CPU usage s ncluded n ths secton. There are many avalable predcton methods, ncludng nonlnear tme-seres analyss, wavelet analyss, rough and fuzzy sets. Our mplement of fractal predcton for CPU usage can be calculated rapdly wth reasonable precson, whch can meet our requrement though not necessarly the best algorthm. The fractal dstrbuton can be descrbed as: C N =, D r where r s the sample tme, an ndependent varable; N s the usage percent of CPU, a varable correspondng to r; C s a constant to be calculated and D stands for the fractal dmenson. Defne a seres of ntal data N j (j=,,m) and the aggregate sum of th order can be calculated as: j Nk, = k=. S, j = j > S, k, k= The fractal dmenson D,j and the constant C,j can be calculated as: S, j+ rj D, j = ln ln S r, j D, j C, j S, j rj j+ =, where r j =j ( j=,,m-). Here the second order aggregated sum s appled snce for most cases ts predcton result s good enough and ts computng overhead s reasonable. In ths case the predcton s: D2, m S2, m * m N m+ = S D 2, m S, m 2, m ( m + ) Table. Fractal predcton of CPU usage (m=5) N j S,j S 2,j S 3,j S 4,j D,j D 2,j D 3,j D 4,j Predcted Measured Relatve % N 6 N 6 error One of predcton results s gven n Table. N j s an average CPU usage (%) durng a 0 seconds nterval. Whle D,j vares to some extent, D 2,j, D 3,j and D 4,j approach to a stable value respectvely. Ths means that fractal dmensons tend to be fxed and can be appled to make predcton. As shown n Table, the relatve error of the predcton N 6 s reasonably good. A predcaton of 00 samples s ncluded n Fgure 3. Whle there are some dfference between the predcton and measurements, predcton results does follow the trend of CPU usage measurements Schedulng of data streamng The amount of data n storage vares over tme and can be descrbed as the followng: Q () t = I () t d () t =,2,, n Q ( 0) = 0 =,2,, n, t, () t where Q () I and d () t stand for the varyng rate, assgned transferrng bandwdth and processng speed for data type (=,,n). () t s the dervatve of the Q

5 data amount n storage, whch reflects ntegrated effects of nput and cleanup of data. Data s streamed to local storage on-demand and then processed and removed. If I (t)= d (t), there s Q () t = 0, whch means that all data s processed and removed as soon as they are transferred, and theoretcally no extra storage s necessary at any tme. CPU Usage (%) predcton measurements Sample No. Fgure 3. Comparson of 00 predcton and measurements of CPU usage Data processng programs run constantly to process avalable data. If no data s locally avalable, they have to be dle and wat whch wastes computatonal resources. So d (t) can be descrbed as: 0 Q () t = 0 d () t = =,2,, n. > 0 Q () t > 0 Sometmes we can just take d (t) as a constant, and then we have: 0 Q () t = 0 d () t = =,2,, n. d Q () t > 0 As mentoned before, the total storage avalable s lmt and the constrant s: n = Q () t S, where S s the total avalable storage. As ntroduced n repertory polces, every type of data has ts lower and upper lmts on storage space, L and U : () t U =,2, n L Q, It s necessary to set values for lower and upper lmts for each data type snce these defne when to start and end transferrng data. In ths work these values are set manually, and eventually some heurstc methods can be appled to fnd more optmal values. I (t) represents data transferrng speeds and can be controlled to optmze performance n terms of throughput wth the followng constrant: n = I () t I() t, where I(t) s the total avalable local bandwdth at a gven tme t. In an actual envronment, I (t) can be adjustable usng carefully selected GrdFTP parallelsm p for each data type. Possble values of p are, 2, 4, 6, and 8 and can be selected as: ~, I I ( t,) ~ 2, I ( t,) < I I ( t,2) ~, p = 4, I ( t,2) < I I ( t,4) ~ 6, I ( t,4) < I I ( t,6) ~ 8, I > I ( t,6) where ~ I ( t ) represents the data transferrng requrement (n another word, data processng speed) for data type at a gven tme t. I (t,p) s the estmated data transferrng speed usng the GrdFTP parallelsm p for data type at a gven tme t. These can be measured and predcted usng methods proposed n Sectons 3. and 3.3. Wth above processes and constrants, the optmzaton goal s to mnmze the waste of computatonal resources, or n another word, to maxmze throughput of each data type: T f max d () t dt =,2,, n, 0 or the whole throughput for all types of data: n T f d () t dt T n max f = max d ()dt t, 0 = 0 = or, f each type of data processng has dfferent prvleges or weghts: n T f d () t dt T n max ω f = max ωd ()dt t 0 = 0 = where T f s the evaluaton tme span and ω s the weght of data type (=,,n). There could be many other schedulng goals, e.g. farness of data transferrng and processng. Many exstng polces n job schedulng can also be appled for schedulng of data streamng. These wll be addressed n our future work. 4. Expermental Results A prototype data streamng envronment s mplemented to evaluate algorthms proposed n Secton 3. A Condor pool s bult to manage over 0 workstatons n the laboratory, provdng computatonal and storage resources. Data to be processed are avalable n remote stes rather than machnes n the Condor pool, and Globus GrdFTP s adopted for data streamng. 4.. Experment desgn An LIGO data analyss program s used as a case study of data streamng applcatons. It reads n data

6 from both observatores and calculates statstcs for gravtatonal burst analyss. LIGO data for analyss are transformed from raw data to bnary data fles n reduced sze usng an nternatonal standard data format. Data processng programs are submtted to the Condor pool and allocated to avalable computatonal nodes accordng to ther requrements. Processng programs run constantly on host machnes, and requred data are transferred from data sources on remote stes to local storage and fed to correspondng processng programs. Our data streamng envronment wth several ndvdual daemons, ncludng sensors, the predctor, the scheduler and data transfer and cleanup agents, s ndependent from applcatons. It montors processng programs, collects nformaton about data transferrng and consumng to decde parameters dynamcally,.e. upper and lower lmts of repertory polces as thresholds of data transfers, GrdFTP parallelsms, and so on. Snce the grd s a dynamc envronment, schedulng s carred out over and over agan and makes new decsons when necessary, e.g. new applcatons jon or data processng speeds and network stuatons change Performance evaluaton - scalablty Three repertory polces are used for performance evaluaton of storage scalablty, whch are llustrated n Fgure 4. Contnuous streamng wthout cleanup. Data transfers are not under any control,.e. data are transferred constantly, and no cleanup procedure s nvolved, so data accumulates rapdly, as shown n Fgure 4. Contnuous streamng wth cleanup. The cleanup process s added. Whle the requred storage space scales better, data stll accumulates over tme. Ths s because data transfers are faster than processng. Although processed data can be removed mmedately, data watng to be processed stll accumulates and requred storage space ncreases. On-demand streamng wth cleanup. In ths scenaro, the schedulng algorthm proposed n Secton 3 s appled. Snce data transfers are controlled usng lower and upper lmts, data transfers would stop at upper lmts and data overflow would not happen. From expermental results gven n Fgure 4 we can nfer that our data streamng and schedulng mechansm should scale well wth applcatons requrng hgh volumes of data and long-term runs. Wth storage lower lmts, our repertory polces also result n tolerance to unexpected stuatons, e.g. network falures. Ths guarantees applcatons can survve network falures whch dsable data streamng from remote stes. Fgure 5 shows such an nstance. Storage Usage (MB) 2.5 x contnuous streamng wthout cleanup contnuous streamng wth cleanup on-demand streamng wth cleanup Tme (mnutes) Fgure 4. Performance evaluaton on storage scalablty usng 3 repertory strateges Durng a many hours data streamng perod, a network broken happens n the mddle for a short perod of tme. Whle the amount of local avalable data (represented usng data fle numbers) drops below the lower lmt temporarly, the applcaton s stll runnng and after the network recovers, data storage resumes to a normal level. Data Fle Number Tme (mnutes) Fgure 5. Storage usage tolerant to an unexpected network falure 4.3. Performance evaluaton - aglty We nvestgate aglty of our data streamng mplementaton by settng up an experment wth dynamcally changng data processng speeds. Ths could happen especally when multple processes are sharng CPU resources. Four typcal scenaros are desgned n the experment and a schedulng scheme of data streamng for 4 processng programs s llustrated n Fgure 6. data. Snce the data applcaton has a very hgh data processng speed, the GrdFTP parallelsm s set to the hghest 8 to meet the requrement. Even data has been streamed as fast as possble, t s obvous that data processng stll has to be dle and wat for avalable data occasonally.

7 data2. As shown n Fgure 6, the data processng speed of data2 applcaton decreases over tme, leadng to a dynamcally changng schedulng scheme for data transferrng. The GrdFTP parallelsm starts from 8 and fnally decreases to 2 snce processng speed becomes low after 4 mnutes. Processng Speed (MB/s) data data data 2 data 3 data Tme (mnutes) p=8 data2 p=8 p=6 p=6 p=2 5. Conclusons Ths work s focused on aglty aspects of data streamng for grd applcatons. On demand data transfers and just-n-tme data cleanups are proposed to mplement system aglty and meet applcaton requrements wth consderaton of scalablty. In order to utlze computatonal resources wth lmt storage and bandwdth, large amount of data have to be streamed for processng. A prototype of a data streamng envronment s mplemented, together wth Condor for CPU allocaton and Globus GrdFTP for data transfers. Some repertory polces are defned, performance measurement and predcton methods are proposed, and schedulng algorthms are mplemented. Expermental results show good scalablty and aglty of our envronment. Ongong work ncludes the consderaton of data sharng scenaros among multple data processng applcatons. Also some heurstc schedulng algorthm s under development for refned performance optmzaton. We are also nvestgatng the method to combne data streamng schedulng wth tradtonal job schedulng so that system aglty could be mproved further. data3 p=4 p=8 p=2 p=6 Acknowledgement data4 p= p= p= p= Tme (mnutes) Fgure 6. Performance evaluaton on aglty of data streamng to dynamcally changng data processng speeds data3. The data processng speed of data3 applcaton changes dramatcally over tme. The data streamng envronment has to adjust the GrdFTP parallelsm p on-the-fly between 2 and 8. data4. In contrast wth data applcaton, data4 s slow so p= can always satsfy the processng requrement, even wth some ntervals when no data transfers happen. In all of above scenaros except data, data transfers start and stop automatcally, meetng applcaton requrements as well as avodng data overflows. In ths work, we provde an ntegrated grd resource schedulng scheme for data streamng applcatons wth combnaton of consderatons of relevant resources, e.g. CPUs, storage and bandwdth. Ths mproves the system aglty to serve ndvdual data processes n a customzed and personalzed way, though applcaton requrements are dynamcally changng. Ths work s supported by Mnstry of Educaton of Chna under the 2/5 project Natonal Unversty Instrument and Resource Sharng Systems, and Mnstry of Scence and Technology of Chna under the natonal 863 hgh-tech R&D program (grants No. 2006AA0Z237, No. 2007AA0Z79 and No. 2008AA0Z8). Junwe Cao would lke to express hs grattude to Prof. Erk Katsavounds of LIGO (Laser Interferometer Gravtatonal-wave Observatory) Laboratory at Massachusetts Insttute of Technology for ther long-term collaboraton supports. References []. B. Agarwalla, N. Ahmed, D. Hlley, and U. Ramachandran, Streamlne: a Schedulng Heurstc for Streamng Applcatons on the Grd, n Proc. SPIE Multmeda Computng and Networkng, Vol. 607, [2]. B. Agarwalla, N. Ahmed, D. Hlley, and U. Ramachandran, Streamlne: Schedulng Streamng Applcatons n a Wde Area Envronment, Multmeda Systems, Vol. 3, No., pp , [3]. B. Allcock, J. Bester, J. Bresnahan, A. L. Chervenak, I. Foster, C. Kesselman, S. Meder, V. Nefedova, D. Quesnal, and S. Tuecke, Data Management and

8 Transfer n Hgh Performance Computatonal Grd Envronments, Parallel Computng, Vol. 28, No. 5, pp , [4]. A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, I. Nshzawa, J. Rosensten, and J. Wdom, STREAM: The Stanford Stream Data Manager, IEEE Data Eng. Bull., Vol. 26, No., pp.9-26, [5]. V. Bhat, S. Klasky, S. Atchley, M. Beck, D. McCune, and M. Parashar, Hgh Performance Threaded Data Streamng for Large Scale Smulatons, n Proc. 5 th IEEE/ACM Int. Workshop on Grd Computng, pp , [6]. D. A. Brown, P. R. Brady, A. Detz, J. Cao, B. Johnson, and J. McNabb, A Case Study on the Use of Workflow Technologes for Scentfc Analyss: Gravtatonal Wave Data Analyss, n I. J. Taylor, D. Gannon, E. Deelman, and M. S. Shelds (Eds.), Workflows for escence: Scentfc Workflows for Grds, Sprnger Verlag, [7]. D. Carney, U. Cetnterne, M. Chernack, et. al., Montorng Streams: A New Class of Data Management Applcatons, n Proc. Int. Conf. on Very Large Data Bases, pp , [8]. J. Chen, D. DeWtt, F. Tan, and Y. Wang, NagaraCQ: A Scalable Contnuous Query System for Internet Databases, n Proc. ACM SIGMOD Int. Conf. on Management of Data, pp , [9]. A. Cockburn, Agle Software Development, Addson- Wesley, 200. [0]. C. Cranoe, T. Johnson, V. Shkapenyuk, and O. Spatscheck, Ggascope: a Stream Database for Network Applcatons, n Proc. ACM SIGMOD Int. Conf. on Management of Data, pp , []. S. Davs, and B. J. Pne II, Mass Customzaton: The New Fronter n Busness Competton, Harvard Busness School Press, 999. [2]. E. Deelman, J. Blythe, Y. Gl, C. Kesselman, G. Mehta, K. Vah, et. al., Mappng Abstract Complex Workflows onto Grd Envronments, J. Grd Computng, Vol., No., pp , [3]. E. Deelman, C. Kesselman, G. Mehta, L. Meshkat, L. Pearlman, et. al., GrPhyN and LIGO, Buldng a Vrtual Data Grd for Gravtatonal Wave Scentsts, n Proc. th IEEE Int. Symp. on Hgh Performance Dstrbuted Computng, pp , [4]. I. Foster and C. Kesselman, Globus: A Metacomputng Infrastructure Toolkt, Int. J. Supercomputer Applcatons, Vol., No. 2, pp. 5-28, 997. [5]. G. Fox, H. Gadgl, S. Pallckara, M. Perce, R. L. Grossman, et. al., Hgh Performance Data Streamng n Servce Archtecture, Techncal Report, Indana Unversty and Unversty of Illnos at Chcago, July 2004 [6]. P. T. Kdd, Agle Manufacturng: Forgng New Fronters, Addson-Wesley, 994. [7]. S. Klasky, S. Ether, Z. Ln, K. Martns, D. McCune and R. Samtaney, Grd-Based Parallel Data Streamng Implemented for the Gyroknetc Torodal Code, n Proc. ACM/IEEE Supercomputng Conf., [8]. R. Kuntschke, T. Scholl, S. Huber, A. Kemper, A. Reser, et. al, Grd-Based Data Stream Processng n e-scence, n Proc. 2 nd IEEE Int. Conf. on e-scence and Grd Computng, [9]. M. Ltzkow, M. Lvny, and Matt Mutka, Condor A Hunter of Idle Workstatons, n Proc. of 8 th Int. Conf. on Dstrbuted Computng Systems, pp. 04-, 988. [20]. R. Pordes for the Open Scence Grd Consortum, The Open Scence Grd, n Proc. Computng n Hgh Energy and Nuclear Physcs Conf., Interlaken, Swtzerland, [2]. A. Ramakrshnan, G. Sngh, H. Zhao, E. Deelman, R. Sakellarou, K. Vah, K. Blackburn, D. Meyers, and M. Samd, Schedulng Data-ntensve Workflow onto Storage-Constraned Dstrbuted Resources, n Proc. 7 th IEEE Int. Symp. on Cluster Computng and the Grd, Ro de Janero, Brazl, pp , [22]. Y. Zhu and D. Shasha, Statstream: Statstcal Montorng of Thousands of Data Streams n Real Tme, Techncal Report TR , CS Dept, New York Unversty, 2002.