DATABASES AND THE CLOUD. Gustavo Alonso Systems Group / ECC Dept. of Computer Science ETH Zürich, Switzerland

Transcription

1 DATABASES AND THE CLOUD Gustavo Alonso Systems Group / ECC Dept. of Computer Science ETH Zürich, Switzerland AVALOQ Conference Zürich June 2011

2 Systems Group Enterprise Computing Center

3 Some results of ECC Impact on Direct on Products and Releases TA Models, Privacy (CreditSuisse) Stream Federator (SAP) Exploratory work and alternative designs Global Dictionary, Global TAs (SAP) Design of New Products Crescando (Amadeus) Hadoop (Amadeus, CreditSuisse) Semantic Search (CreditSuisse) All projects based on Real products Customer requirements Real data, real loads

4 DATABASES: The cornerstone of enterprise architectures

5 Relational Databases No real alternative to relational database engines Transactional guarantees Recovery guarantees Existing investments Constant need for larger and larger deployments to cope with modern loads

6 Relational Databases today Peak load provisioning = license a huge database engine to cope with peak loads Scalability = relational engines scale but only at an exponential cost in licenses, hardware, and administrative overhead Complexity = Huge costs for tuning, maintenance and administration Price = exorbitant costs for the performance

7 CLOUD COMPUTING: Change in model

8 The key to cloud computing TODAY= Secondary sector economy (manufacturing) Focus is on production processes CLOUD= Tertiary sector economy (services) Focus is on business processes Subject to completely different forms of regulations and behavior expectations 8

9 AMADEUS: reaching for the clouds

10 Amadeus Workload Passenger Booking Database ~ 600 GB of raw data (two years of bookings) single table, denormalized ~ 50 attributes: flight no, name, date,..., many flags Query Workload up to 4000 queries / second latency guarantees: 2 seconds today: only pre canned queries allowed Update Workload avg. 600 updates per second (1 update per GB per sec) peak of updates per second data freshness guarantee: 2 seconds Problems with State of the Art Simple queries workonly because of mat. views multi month project to implement new query / process Complex queries do not work at all

11 Traditional engines break down Update Load in Updates/sec MySQL Query 50th MySQL Query 90th Synthetic Workload Parameter s 20'000 MySQL Query 99th 9'000 8'000 15'000 10'000 Query Latency in msec 7'000 6'000 5'000 4'000 3'000 Query Latency in msec 5'000 2'000 1' Performance depends on workload parameters changes in load (updates, columns accessed) > huge variance Unpredictable performance, impossible to tune correctly

12 Many problems Exhaustive benchmarking and analysis shows: Lack of scalability with number of cores Problems with I/O with number of cores Load interaction problems Unpredictable performance Increasingly expensive tuning

13 Amadeus requirements Predictable (= constant) Performance Meet SLAs on latency and data freshness Affordable Cost compared to mainframe / current license Maintain Consistency monotonic reads (ACID not needed) Suitable for modern hardware main memory, NUMA, large data centers

14 CRESCANDO: rethinking data processing

15 What is Crescando? A distributed (relational) table: main memory on NUMA horizontally partitioned distributed within and across machines Query / update interface SELECT * FROM table WHERE <any predicate> UPDATE table SET <anything> WHERE <any predicate> monotonic reads / writes (SI within a single partition)

16 Work unit = Clock Scan QUERIES UPDATES BUILD QUERY INDEX FOR NEXT SCAN READ CURSOR WRITE CURSOR DATA IN CIRCULAR BUFFER (WIDE TABLE)

17 Crescando on 1 Machine (N Cores) Scan Thread Scan Thread Input Queue (Operations) Split Scan Thread Scan Thread Merge Output Queue (Result Tuples)... Input Queue (Operations) Scan Thread Output Queue (Result Tuples)

18 Crescando in a Data Center (N Machines)

19 Implementation Details Optimization decide for batch of queries which indexes to build runs once every second (must be fast) Query + update indexes different indexes for different kinds of predicates e.g., hash tables, R trees, tries,... must fit in L2 cache (better L1 cache) Probe indexes Updates in right order, queries in any order Persistence & Recovery Log updates / inserts to disk (not a bottleneck)

20 What is different? No threads (work unit = core) No synchronization across work units Data partitioned across work units Work units across cores and machines No indexes on data, no materialized views Constant performance Performance determined by design Dynamic scalability / elasticity

21 Linear scalability on modern hardware

22 Status Going live summer 2011

23 The road ahead Many exciting projects SharedDB SwissBox Cloud computing Barrelfish FPGA data processing Solid systems research with real impact

24 Systems Group Enterprise Computing Center