SciDB An Open Source Data Base Project. Paul Brown* *medical emergency trumped presence

Transcription

1 SciDB An Open Source Data Base Project by Paul Brown* *medical emergency trumped presence

2 Outline Science data Why science folks are unhappy with RDBMS Our project what we are doing about it

3 O(100) petabytes

4 Nearest neighbor queries, time series queries

5 Snow Cover in the Sierras

6 Why SciDB? Big science very unhappy with RDBMS Astronomy HEP Fusion Bio Remote sensing Oceanography

7 Why? Experience of Sequoia 2000 (mid 1990s) Tried to use Postgres for science databases Failed badly Main science data type is an array horribly inefficient to simulate arrays on top of tables Required features absent (provenance, uncertainty, version control) SQL operations wrong (regrid not join)

8 Why SciDB? Net result Mentality of roll your own from the ground up for every new science project Realization by the science community that this is long-term suicide Community seemingly wants to get behind something better Great commonality of needs among domains

9 A Little Context XLDB-1 (Oct 2007) Message from previous slides came across loud and clear Dewitt/Stonebraker agreed to move the ball down the field

10 A Little Context Asilomar (March day workshop people) Flesh out requirements (biggie is open source, commercial quality, petabyte-scale DBMS) Considerable commonality across science disciplines Core team of science and DBMS types identified to push things forward Research issues identified

11 A Little Context The Next Year Initial design completed Along with an initial implementation Shown at VLDB/Lyons Recruiting of initial team Detailed use cases specified

12 Funding Situation Tried and failed to get $$$ from NSF/DOE/NASA Tried and failed to get $$$ from foundations Tried and failed to get $$$ from industry Last resort was VC s Company (Zetics) funded in March 10

13 Present Day Structure About 25 employees/consultants/volunteers co-ordinated by Suchi Raman Design co-ordinated by Mike Stonebraker and Paul Brown Support/marketing/business/website co-ordinated by Marilyn Matz

14 Data Model Nested multi-dimensional arrays natural representation for spatially or temporally ordered data Array cells can be tuples of values or other arrays extensible type system Arrays can be very sparse user-definable handling for null or missing data

15 Data Storage Arrays are chunked (in multiple dimensions) in storage Chunks are partitioned across a collection of nodes Each node has processor and storage (shared nothing) Chunks have overlap to support neighborhood operations

16 Architecture Shared nothing cluster 10 s 1000 s of nodes Commodity hardware Queries refer to arrays as if not distributed Query planner optimizes queries for efficient data access & processing Query plan runs on a node s local executor/storage manager Runtime supervisor coordinates execution Application Language Specific UI Runtime Supervisor Query Interface and Parser Plan Generator Node 3 Node 2 Node 1 Local Executor Storage Manager Application Layer Server Layer Storage Layer

17 AQL: an array & analytics query language DBMS-style operations Filter, combine (join) Array operations Multiply Transpose Extensible Postgres style UDFs Interfaces to other open source packages MatLab R

18 Other Features Which Science Guys Want (These could be in RDBMS, but Aren t) Uncertainty Data has error bars Which must be carried along in the computation (interval arithmetic)

19 Other Features Provenance (lineage) What calibration generated the data What was the cooking algorithm In general repeatability of data derivation

20 Other Features Time travel Don t fix errors by overwrite I.e. keep all of the data Spatial support Named versions Recalibration usually handled this way

21 Status Open source SciDb 0.5 available now From our web site Good for PoCs Not good for Pbytes Better versions are imminent Come to the community meeting this afternoon for a roadmap