Realization of GGF DAIS Data Service Interface for Grid Access to Data Streams

Transcription

1 Reaization of GGF DAIS Data Interface for Grid Access to Data Streams Ying Liu, Beth Pae, Nithya Vijayakumar Indiana University Boomington, IN IU-CS TR 613 ABSTRACT As the computation power of hardware increases, rea-time processing of arge amount of streaming data becomes possibe. Streams are widey recognized as one of the major data resources. Meanwhie, many efforts are spent on the grid framework research, which enabes peope and appications across physica and administrative domains to easiy share data and computationa resources. Our work everages the cutting-edge technoogies in both areas to define a new grid-accessibe streaming query and management abstraction, named Virtua Stream Store. In this paper, we describe our reaization of the interfaces specified in the GGF DAIS Data proposa for streaming resources. I. INTRODUCTION In this technica report, we describe a reaization of the DAIS Data specification [7] for a nove data resource. The resource is an abstract resource that is a coection of data streams that taken together define a body of information meaningfu to a community. We foow the framework provided by OGSA Data proposa [4]. It is recommended that the reader be famiiar with the OGSA Data s proposa [4] and the more genera Grid Data Specification [7]. The atter defines various porttypes that are extended in this document. At the same time, this document can be viewed as a parae document to other reaization documents, incuding the Reationa Reaization [5] and XML Reaization [6]. The rest of this technica report is organized as foows. We provide an overview of the specification in the section II. In section III, we show how the system works using four use-cases. In section IV, we discuss the interface reaization in detai. Concusion is given in section V. II. SPECIFICATION OVERVIEW Virtua Stream Store (VSS) is an abstract coection of streams and computationa resources [1]. More specificay, it has two major components. First, it contains a coection of distributed data streams that are reated by coherence and meaning, and share a common communication substructure. Second, the virtua stream store has access to a set of computationa resources ocated in physica proximity to data streams. The computationa resources are where query processing is carried out. The Grid Data (GDS) is a web service that provides a document-based access to a data resource. It adheres to the OGSA Data s proposa [4] and as such is the interface to the GDS is through three genera porttypes: Data Access, - 1 -

2 Data Factory, and Data Management. The GDS used in our work is based on the GDS of OGSA-DAI [10] with a few extensions and exceptions necessitated by the needs of the virtua stream store mode. A GDS exists in a one-to-one reationship with a Virtua Stream Store. With the exception of a pug-in that taks to a specific stream system, the GDS is agnostic to any specific impementation of the stream system. The GDS tracks a active queries and computationa resources that beong to its virtua stream store. dquob [11] is an exampe of a back-end continuous query stream processing system. The dquob stream system provides dynamic query execution over data as the data moves from providers to consumers in event channes. The VSS is a ogica coection of streams that share meaning and coherence. dquob server shown in Figure 1, is a persistent service that contros access to the dquob stream system [2]. A stream query processing system can generate streams that beong to one or more VSS; hence a VSS is in a many-to-one reationship with the backend dquob system. More generay, dquob can be repaced by any stream processing system impementation, for exampe, the Stanford Stream server [8] as shown in Figure 1. n :1 VSS Factory VSS Registry VSS1 User Appications GDS1 Cataog VSS2 Query Panner Rowset dquob Server GDS2 Cataog Query Panner Rowset GDS3 Cataog VSS3 Query Panner Rowset Stream Server Figure 1 Virtua Stream Store components and interaction with users and backend stream processing systems. The streaming reaization is a specification of data services and interfaces for a streaming data resource. We define Grid Data as a persistent service that serves a singe Virtua Stream Store [1]. As defined before, a Virtua Stream Store is an abstract coection of streams and computationa resources. The Grid Data depicted in Figure 2 is a grid service composed of four sub-service categories. A sub-service category can be viewed as a functiona unit that exports a unique set of interfaces from the other functiona units

3 The various functionaities that must be supported in the Virtua Stream Store can be viewed as faing into four sub-service categories, where each virtua interface is composed of methods that cross three genera port types: I. SQL Logica Interface: the data resource is a stream processing system, hence is accessed by means of a continuous query anguage. This anguage is typicay an extension of SQL. There is a singe persistent instance of the SQL Data. II. Rowset Logica Interface: exports a set of interfaces that enabe a user to manipuate the resuts of a continuous query. Each Rowset Data can serve for severa user submitted queries. The Rowset Data is in existence for the duration of the often ong-ived continuous query. III. Stream Pubish Interface: We view a data resource as a set of streams. The users of this service are stream providers. Through this service a stream provider can add or remove a stream and aso change its characteristics. IV. Admin Logica Interface: Provides grid access for administrator to manage the streams and the computationa resources in the system. A virtua interface exports the standard set of interfaces as defined in [7], DataAccess, DataFactory, and DataManagement. The DataAccess interface aows access to the data resource itsef. This coud be a query issued to a DBMS, to update a set of rows, or retrieve a tupe from a resut set. The DataFactory interface aows for the instantiation of a new instance of a service. The DataManagement interface aows management activities on the data service itsef. The interfaces and operations of the GDS are summarized in Figure 2. The detais are expained section IV. Sub SQL Logica Interface Rowset Log. Inter. Stream Pub. Logica Interface Operation SQLAccess:sqQuery SQLAccess:dropQuery SQLFactory:create SQLMgt:istQueryStatus RowsetAccess:getTupeTi RowsetAccess:getTupes RowsetFactory:create PubishAccess:createChanne PubishAccess:registerStream PubishAccess:pubishData PubishAccess:unregisterStream PubishAccess:removeChanne DataAccess Interface DataFactory Interface DataManagement Interface Admin Log. Inter. AdminAccess:addComputationaEement AdminAccess:dropComputationaEement Figure 2 Four sub-services of GDS - 3 -

4 III. TOUR OF SPECIFICATION BY EXAMPLE In this section, we demonstrate functionaity of VSS through four use-cases. Case I. Administrator Aice wants to set up a new instance of a VSS named VSS1. How does she go about doing this? VSS Factory service shown in Figure 1, is a persistent service that responds to request for creating a new instance of Virtua Stream Store. To instantiate a new instance of VSS means to instantiate the GDS, cataog service, query panning service and the rowset service. GDS is the gateway to the Virtua Stream Store and it cas other services internay. The API discussed in technica report is impemented in the GDS. A persistent VSS Registry service shown in Figure 1, keeps track of the GDS handes for different VSS instances. The information of a input data streams and computation nodes registered to a specific VSS instance is maintained by its cataog service. The query panner records information of a the queries and based on this information and generates a query pan for the incoming query. The rowset service heps users manage streaming query resuts. A above services are buit on Open Grid s Architecture (OGSA) mode [3, 11] and impemented according to the OGSI specification [9]. Besides starting these persistent services, Aice aso needs to specify the pubish/subscribe protoco used by the backend stream processing system and its query anguage. Then a new VSS1 is created and it is ready to take new streams, computation resources, continuous queries and foow-up rowset queries. Case II: How does Stream provider Bob, pubish his streams into an existing VSS1? Bob first obtains the hander of the GDS (say GDS1 ) corresponding to VSS1 by searching through the VSS Registry service. Then he pubishes streams into VSS1 using GDS1 s create_channe() interface. Since each GDS is associated with ony one VSS, Bob need not specify the name of the VSS instance to which he pubishes streams. But he needs to specify the name of the channe aong with data format information of be added to the underying pubish/subscribe system. create_channe (channename, datatype); The first parameter to the operation is a channename which is meaningfu ogica name for Bob to refer this channe ater. The second is type definition for this channe. After this, VSS wi register this new channe and its type (format) information and return a goba unique channe identifier for the new channe. Later Bob uses channename to refer this channe and pubish data to it as shown beow. pubish_data (channename, streamname); But the VSS system maps (Bob, channename) to the system-wide unique channeid for reference. Case III: User David submits 3 ong running queries (q1, q2,q3) on VSS1 through GDS1 and is handed a pointer to three RingBuffers (one for each query) on the Rowset Data s, from which he can retrieve resuts

5 David must first contact the registry to obtain the hande for GDS1 corresponding to VSS1. Then he submits the queries through GDS1 s interface SQLAccess:sqQuery(). To use this interface, David needs to specify the query string and query name. Associating a name with the query enabes easy identification of the query. VSS1 creates a new RingBuffer on the rowset service for each query submitted. The pointers (IDs) to the newy created RingBuffers are returned to David. RingBufferIDList = sqquery(qname1,qexp1,qname2,qexp2,qname3,qexp3); The streamed resuts are returned and stored in the corresponding ringbuffers. Case IV: User Mary submits one ong running query to VSS1 through GDS1 that is exacty the same as one of the queries that David submitted. This case is simiar to the previous one from the user point of view. But the system notices that Mary s query is exacty the same as an executing query. To save system resources, we wi et Mary and David share the same RingBuffer. However, for security reasons, if either of them specify that they do not want to share their RingBuffers, VSS wi create a new RingBuffer for Mary and David each. IV. DATA SERVICE INTERFACE The individua interfaces are defined as foows: A. SQL Logica Interface SQL Data is a sub-service which virtuaizes the data resource as a SQL database. User communicates with a data resource through SQL-based continuous query anguage. The service interfaces can be cassified into DataAccess interface, DataFactory interface and DataManagement interface as shown in Figure 2. SQLAccess:sqQuery operation is used to direct pre-defined continuous query to data resources. The continuous query is expressed in SQL ike anguage. And user can give his continuous query a ogica name, which can be used to refer to this query ater. We overoad this operation with two different parameter sets. One is for a singe query submission; the other is used to submitting severa queries at a time. Both operations are isted beow. RingBufferID = sqquery(queryname, querystring); This operation accepts a singe continuous query. It has two inputs, querystring and queryname. querystring is defined in the continuous query anguage accepted by the backend stream processing system. queryname is the ogica name of SQL query that wi be used as the hande for this query. The output is a rowsetptr, which is the pointer to the rowset data service. The rowset data service is where the resut is stored. The rowset data service is expained in detai a itte ater. RingBufferIDList = sqquery(qname1,qexp1,qname2,qexp2 qnamen,qexpn) ; - 5 -

6 This operation overoads the previous one with different input, output parameters. User can submit severa queries at a time with this operation. The input can have severa different pairs of (qname, qexp), each of them specifies one submitted query. Accordingy the output is a ist of pointers to the ring buffers created in the rowset set service that hods resuts for these queries. Possibe exceptions from these operations are InvaidQuery, InvaidQueryName and ConfictQueryNumber. InvaidQuery means a query either semanticay incorrect or fais during the execution. InvaidQueryName means that query name either vioates naming scheme or aready exists. ConfictQueryNumber points out that there is confict between the number of queries submitted and the number of names user specified. SQLAccess:dropQuery operation can stop/drop a particuar continuous query from the system. True/Fase = dropquery (queryname); The input of this operation is queryname. It is the ogica name of the query to be dropped from VSS. The output is a booean vaue indicating success or faiure of the operation. A possibe exception message is invaidqueryname, indicating the specified query name does not exist in the system for this user. SQLFactory:create operation beongs to DataFactory interface. It is used to create a new data service, which can be an empty service (no resources associated), a service with associated resources or a session between cient and user. Based on different requirements, we overoad this operation with different input parameters. True/Fase = create(vssname); This operation has a singe parameter, VSSName. The new created service is has no associated resources initiay. The input parameter is VSSName, which is the name of VSS. The output indicates whether the operation succeeds or not. True/Fase = create(vssname, resourceconfig); This operation has two input parameters. VSSName is the ogica name for new VSS and resourceconfig is the fie where the resource configuration is described. Possibe exceptions are invaidvssname and invaidconfig. invaidvssname can be caused by the vioation of the naming scheme or the specified name aready being used. invaidconfig indicates that either the configuration fie in the wrong format or it specifies some resources which do not exist. SQLManagement:istQueryStatus This operation is used to check the status of specific query

7 querystatus = istquerystatus (queryname); The input parameter is the queryname, which is the name of specified query. The output, querystatus, is the status of specified query, which can be PLAN, DEPLOY, WAIT and ACTIVE. The status indicates the different stages for the query: under the pan generation stage, under depoyment, waiting for the incoming data or under execution. Possibe exception is invaidquery, which can be caused by specifying a non-existing query. B. Rowset Data RowsetAccess:getTupeTi operation is used to access tupe received at timestamp Ti in the ring buffer. tupeptr = gettupeti(timestamp, ringbufferid); The inputs of this operation are timestamp and ringbuffername. timestamp specify the time when the interested tupe is received. ringbufferid indicates the targeted ring buffer. Possibe exceptions are invaidringbuffer and tupesnotavaiabe. invaidringbuffer indicates that the specified ring buffer does not exist. tupesnotavaiabe means that the user specified tupes are not avaiabe unti the query fired time. RowsetAccess:getTupes: the operation is used to access tupes faing into a specific time period. resutptr = gettupes(starttime, endtime, returndatamode, ringbufferid); There are four inputs: starttime and endtime are timestamps to define the interested time period. returndatamode indicates how the resut data wi be returned. There are two modes: BULK_RETURN and STREAM_RETURN. The former chunks a the data together and return the buk. STREAM_RETURN mode returns data continuousy as a stream. ringbufferid specifies the ringbuffer where the tupes are cached. Under the BULK_RETURN mode, the output is the pointer to the storage where a the chunked data is stored. Under the STREAM_RETURN mode, the output is the pointer to the channe where the stream comes. Possibe exceptions are invaidtimerange, invaidringbuffer, tupenotavaiabe. invaidtimerange means starttime is arger than endtime. invaidringbuffer indicates that specified Ringbuffer does not exist on the service. tupenotavaiabe indicates satisfied tupes are sti not avaiabe in the ringbuffer. RowsetDataFactory:create operation is used to create a new Rowset Data. In a VSS, there must be at east one Rowset Data as it is a persistent service component for VSS. Typicay an instance of Rowset is started when the VSS is created. At the beginning, a the RingBuffers are created on that Rowset. Later, for system s scaabiity, new Rowset can be created. This create ca is issued by the system without user - 7 -

8 invocation. When the user submits a query into system and the maximum number of ringbuffers is reached on the main Rowset Data, system wi automaticay create an auxiiary Rowset Data. rowsetptr = create(); The operation has no input parameter and the output is the pointer to the new created Rowset Data. C. Stream Pubishing PubishingAccess:createChanne operation is used to create new pubishing channe in the system. This operation is overoaded with two separate parameter sets. One is to create a new channe at a time. The other enabes to create severa channes once. True/Fase = createchanne (channename, datatype); The inputs to this operation are datatype and channename. datatype is the data type or dataformat definition for the new channe. channename is reference name for the new created channe, which must be unique in the whoe system. The output indicates whether the operation succeeds or not. True/Fase = createchanne (channenamelist, datatypedeffie); This operation can create severa channes at a time. The input parameters are channenamelist and datatypedeffie. datatypedeffie is a formatted fie which incudes a ist of data type definitions. ChanneNameList is the ist of referenced names of new created channes. Output indicates the operation status. There is not partia success. Possibe exception is confictname which means that the specified name is aready taken by some channes. Another possibe exception is that invaiddatatype which indicates the datatype definition fie does not foow the pre-defined formats. PubishAccess: registerstream operation is used to register a new stream resource to an existing channe. streameid = registerstream(channename, streamformat); The inputs are channename and streamformat. channename specify where the streaming data wi go. streamformat defines the format of the data to be pubished. Output is the system-wide unique stream ID that wi be used ater as a reference for the stream. Possibe exception is that nonexistchanne and confictformat. nonexistchanne means the specified channe does not exist in the system. confictformat means the stream has different dataformat from the channe s format

9 PubishAccess: pubishdata operation can be used to pubish the stream to a specific channe. True/Fase = pubishdata (streamid, streamdata); The inputs are streameid and streamename. streamid is the system-wide identifier got from the stream registration operation. The streamdata points to the data to be pubished to the system. Possibe exceptions are nonexiststreamid. nonexiststreamid means that no such stream has been registerd. PubishAccess: unregisterstream operation is to unregister the stream from the channe. True/Fase = unregisterstream (channename, streamid); Inputs are channename and streamid. channename specifies the channe where the stream wi un-register from. streamid specifies which stream wi be un-registered. Possibe exceptions are nonexistchanne, nonexiststream and invaidoperation. nonexistchanne means the specified channe does not exist, nonexiststream means no such stream is registered to the channe. invaidoperation means this operation is not aowed to be issued by the user, as VSS ony aows the stream provider to un-register the stream from the channe. PubishAccess: removechanne operation is used to remove channes from the VSS. True/Fase = removechanne(channename); The input is channename which is the ogica name of channe to be removed. Ony the provider who creates the channe can request to remove that channe. A possibe exception is nonexistchanne, which means the channe does not exist. Another exception is invaidoperation which means this channe is not aowed to be removed as there are sti some streams attached to it. D. Admin AdminAccess:addComputationNode operation adds a new Computationa Node in a Virtua Stream Store. nodeid = addcomputationnode(hostname, configfie); - 9 -

10 The inputs are hostname and configfie. hostname can be used to identify the contributing computer on the Internet. The configfie is the name of a formatted fie which provides some information for this host. The information can be the memory size, cpu speed, ocation and etc. The resources provider can even specify how much they want their resources to be shared in the configfie. The output is the goba unique indentifier for this computation node. Possibe exceptions are existhost and invaidconfig. existhost means that this host has been aready registered as computation node in this VSS. invaidconfig indicates the configuration fie vioates the pre-defined format. AdminAccess:removeComputationNode operation aims to remove a computation node from the Virtua Stream Store. True/Fase = removecomputationnode(nodeid); The input of this operation is nodeid, which is the goba unique identifier for the node, generated when the node is added into VSS. The output shows the execution status. Possibe exceptions are invaidnode and invaidaccess. invaidnode means the specified node does not exist. And invaidaccess indicates that user don t have permission to remove this node because ony the resource provider can do this. V. CONCLUSION Data streams are main data resource for many appications, such as Weather forecasting, Stock market financia anaysis, Network traffic monitoring and so on. The management of streaming resources pays a key roe in these appications. At the same time, Open Grid Architecture (OGSA) [3, 11], aims to faciitate the sharing of data and computing resources, which is needed by these streaming appications. In this paper, we described the reaization of OGSA Grid Data in Virtua Stream Store, a stream query and management system. By providing grid access to streams, Virtua Stream Store enabes sharing of data and computation resources among different streaming appications. ACKNOWLEDGMENT The authors woud ike to thank Mario Antonioetti and Nei Chue Hong at EPCC for their hepfu discussion and support. REFERENCES 1. B. Pae, Architecture for Accessing Data Streams on the Grid, In Proceedings of 2nd European Across Grids Conference (AxGrids), B. Pae, K. Schwan, Dynamic Querying of Streaming Data with the dqoub System, In Journa IEEE Transactions on Parae and Distributed Systems, 14(4),

11 3. I. Foster, H. Kishimoto, A. Savva, D. Berry, A. Djaoui, A. Grimshaw, B. Horn, F. Macie, F. Siebenist, R. Subramaniam, J. Treadwe, J. Von Reich, Open Grid s Architecture, Version 1.0, Goba Grid Forum Informationa Document, Continuousy updated. 4. I. Foster, S. Tuecke., J. Unger, OGSA Data s, Goba Grid Forum 9, M. Antonioetti, A. Krause, S. Hastings, S. Langea, S. Maaika, J. Magowan, S. Laws, N. W. Paton, Grid Data Specification: The Reationa Reaization, Goba Grid Forum 9, M. Antonioetti, A. Krause, S. Hastings, S. Langea, S. Maaika, S. Laws, N. W. Paton, Grid Data Specification: The XML Reaization, Goba Grid Forum 9, M. Antonioetti, M. Atkinson, S. Maaika, N. W. Paton, D. Pearson, G. Riccardi, Grid Data Specification, Goba Grid Forum 9, R. Motwani, J. Widom, A. Arasu, B. Babcock, S. Babu, M. Datar, G. Manku, C. Oston, J. Rosensteain, R. Varma, Query Processing, Resource Management, and Approximation in a Data Stream Management System, In Proceedings of Conference on Innovative Data Systems Research (CIDR), S. Tuecke, K. Czajkowski., I. Foster, J. Frey, S. Graham, C. Kesseman, T. Maquire, T. Sandhom, D. Sneing, P. Vanderpit, Open Grid s Infrastructure, Goba Grid Forum 8, M. Antonioetti, M. Atkinson, R. Baxter, A. Borey, N. P. Chue Hong, B. Coins, N. Hardman, A. C. Hume, Aan Knox, M. Jackson, A. Krause, S. Laws, J. Magowan, N. W. Paton, D. Pearson, T. Sugden, P. Watson, M. Westhead, Design and impementation of Grid database services in OGSA-DAI, In Journa Concurrency and Computation: Practice and Experience, 17 (2-4, ), I. Foster, D. Gannon, The Open Grid s Architecture Patform, Goba Grid Forum 7, Working Draft,