GRID Stream Database Managent for Scientific Applications

Transcription

1 GRID Stream Database Managent for Scientific Applications Milena Ivanova (Koparanova) and Tore Risch IT Department, Uppsala University, Sweden

2 Outline Motivation Stream Data Management Computational GRIDs GSDM Distributed Architecture Project status Next steps GSDM Demo

3 Motivation: LOFAR/LOIS Application Data Fabric with nodes Data Processing Plant User post processing on-line hybrid data processing cluster Control Center(s)

4 Motivation: LOFAR/LOIS Application Geographically distributed set of electromagnetic sensors and emitters Data products: Raw data streams (beams) Very high data volume and rate Complex numerical data Continuous queries: filtering, reduction, combining of data streams User-defined computational functions Problem: High performance processing of distributed continuous queries over many data streams. Utilizing GRID infrastructure for achieving highperformance

5 How stream data are different? Streams peculiarities: Infinite Representation of substreams of limited size: windows Continuous Queries (CQs) Immediate processing of elements, followed by archiving or deletion Order preserving and non blocking processing Stream specific operations, e.g., windows join, moving average DBMS techniques for streams: approximate query answering, adaptability, data reduction, multi-query optimization

6 GRID Computing Heterogeneous sets of clusters of computers For applications with much need for CPU cycles Toolkits (e.g. GLOBUS, CONDOR-G) provide transparent access to GRID clusters and parallelization of jobs Batch processing (upload compute download) High data volume bottleneck

7 GSDM as a Computational GRIDs Application High data flow rate and large data volumes require high performance Parallelism and distribution Varying computer resource needs because of varying number of queries and streams, varying stream rates Dynamic resource allocation Computational GRID infrastructure

8 GSDM: GRID Stream Database Approach Manager for Scientific Data Distributed and parallel Stream-oriented Main-memory Object-Relational DBMS Utilization of GRID infrastructure for achieving high-performance

9 GSDM Scenario Query Coordinator 1 Application CQ1 Metadata Manager Application CQ2 Working Node 5 Query Coordinator 2 Working Node 1 Working Node 3 Working Node 2 Working Node 4 GRID Data Beam1 Data Beam 2 Data Beam 3 Legend: denotes data flow

10 GSDM Distributed Architecture Four types of nodes Metadata manager Coordinator server Working node Application

11 Metadata and Coordinator Server Functions: store metadata about the system decompose, install and activate CQs assign queries to working nodes start and kill working nodes coordinate processing at working nodes

12 System Metadata Type GSDM name(gsdm) -> Charstring working_node(gsdm) -> Boolean coordinator(gsdm) -> Boolean cur_load(gsdm) -> Real Type Query qid(query)->charstring query_string(query)-> Charstring producers(query) -> Bag of GSDM consumers(query) -> Bag of GSDM installed(query,gsdm) -> Boolean active (Query,GSDM)-> Boolean

13 Working Node Function: to process continuous queries over streams List of active CQ CQ Execution loop Stream Consumer receives pushed data and incoming commands scheduler picks next CQ ready for execution executes the CQ over current stream windows Continuous Query Producer sends streamed result to the consumer (application or working node)

14 GSDM Working Node Architecture Continuous Query Manager Application Query Executor Stream Consumer Cont Q Producer Query Executor Cont Q Producer Continuous Query Manager Plug-ins GSDM Working Node Stream Consumer Stream Consumer Stream Consumer Beam 1 Beam 2 GSDM Stream Data sources

15 GSDM Command Line User Interface Commands sent from application Executed at coordinator monitor(charstring qstring) -> Charstring qid; run(charstring qid) -> Boolean; stop(charstring qid) -> Boolean; status(charstring qid) -> <Charstring qstring, Charstring prodname>;

16 GSDM Communication Primitives User Node Monitor Query Qid Run Qid Status Qid Start node Register WN Metadata and Coordinator Install query qid, qstring Installed qid Activate qid Activated qid Working Node Deactivate qid Deactivated qid Uninstall qid Uninstalled qid Kill node

17 Current status GSDM prototype architecture Initial GSDM prototype implementation Formulation of continuous queries with user-defined computational functions from space physics UDP receiver of radio on the Internet Data beam simulator

18 Related Work in Stream DBMS Relatively low stream rates, centralized architectures Relational representations and relational operations modified for stream processing: selection, join, aggregations Our system High rates of LOIS/LOFAR streams Distributed architecture User-defined functions over non-relational data representations

19 GSDM requirements for GRID Interactive job: users can install and stop CQs interactively Accessibility & security issues Data delivery directly to the working nodes, e.g. inside of clusters High-performance communication between GSDM servers running on different GRID resources

20 Next steps Evaluation of initial prototype implementation and development Optimization of distributed stream database queries Utilize GRID infrastructure functionality for resource brokering and management New GRID services for streaming data

21 GSDM Demo Scenario CQ Application (UI) Name Server Metadata and Coordinator Server CQ Results Working Node 1 Application GSDM

22 Demo scenario Database schema start_udp_consumer(char s) -> integer; stop_udp_consumer() -> integer; change_udp_scale(real i) -> integer; get_udp_packet() -> < vector of complex, vector of complex, vector of complex >; Example CQ create function visualized_fft()->boolean as select repaint(v,6.0) from vector of real v, vector of complex x, vector of complex y, vector of complex z where v= vect_log_magnitude(fft(windowed_fft(x))) and get_udp_packet()= <x,y,z>;