Datenbank-Virtualisierung

Transcription

1 Datenbank-Virtualisierung Prof. Dr.-Ing. Wolfgang Lehner DB-Stammtisch,

2 Outline of the Presentation Virtualization Goals and Trends Machine Virtualization Storage Virtualization Impact on Database Systems Database Systems in Virtual Environments Ressource Provisioning Multitple Layer interactions Conclusion 2

3 INTRODUCTION 3

4 ... it s everywhere Any problem in computer science can be solved with another layer of indirection. (David Wheeler) 4

5 ... it s simple! 5

6 What is Virtualization? Separating the abstract view of computing resources from the implementation of these resources A layer of indirection between abstract view and implementation of resources Hides implementation details Controls mapping from abstract view to implementation Adds flexibility and agility to the computing infrastructure Can be used to solve many problems related to provisioning, manageability, security, lower TCO, fewer computing resources Virtualization will be a $20 billion market by (IDC, Jan 2007) 6

7 Why virtualization? Virtualization adds flexibility and agility to the computing infrastructure Can be used to solve many problems related to provisioning, manageability, security, Pool and share computing resource Simplify administration and management Improve fault tolerance For organizations: Lower total cost of ownership for computing infrastructure Fewer computing resources More resilient and simpler to manage 7

8 A typical scenario in data centers Requirements Response time < 2s Throughput > 100 rq/sec Requirements Response time < 5s Throughput > 50 rq/sec Pay 100$ Customer A Customer B Pay 50$ Run auction site Run trading site 8 Shared Data Center 8

9 Hosting applications server Linux Web server Linux Database server Linux Data Center Common idiom: One-to-one mapping of applications to nodes 9 9

10 server Linux Solution: Virtualization Web server Linux Database server Linux Consolidate server Linux Web server Linux Virtualization Database server Linux Improved utilization using consolidation 10 10

11 Microsoft Massive Data Center Northlake, Illinois (US) 11

12 12

13 Resource requirements Resource requirements What does Virtualization enable? Server consolidation The vast majority of server are grossly underutilized Typical setup today: one machine per application (DBMS, web server, mail server, ) Provisioned for peak load. Usually under-utilized Instead, can run multiple applications on virtual machines that share the same physical machine Reduces both CapEx (Capital EXpenditure) and OpEx (Operational EXpenditure) Save hardware costs, administration costs, power, etc. P 1 P 2 P 12 < P 1 + P 2 13 Time Time

14 What does Virtualization enable? Migration of VMs (both storage and CPU/memory) Enables live load balancing Facilitates maintenance Need to bring Machine 1 down for maintenance VM App Migrate VM to Machine 2 (live migration) VM App Can now bring Machine 1 Down without service interruption 14 Machine 1 Machine 2 (spare) Shared Storage

15 What does Virtualization enable? High availability Allows a small number of generic servers to back up all servers VM App Detect failure (heartbeat) VM App Machine 1 Machine 2 Image of VM (stand by) VM App Shared Storage 15 Restart VM on Machine 2 from image on shared storage

16 What can be virtualized the big four. 16

17 17 Virtual Machines

18 18 Virtual Storage

19 MACHINE VIRTUALIZATION 19

20 Machine Virtualization A virtual machine "abstracts" the computing resources of a physical machine into virtual resources Introduces a level of indirection between virtual resources and physical resources End users only see the virtual resources Can install their operating systems and run their applications on the virtual machines A Virtual Machine Monitor (or Hypervisor) is a software layer that implements the mapping from virtual resources to physical resources 20

21 Machine Virtualization (2) Strong isolation between virtual machines Flexible mapping between virtual resources and physical resources Can have more virtual resources than the corresponding physical resources Can reallocate physical resources among VMs Pause, resume, checkpoint, and migrate virtual machines 21

22 Hypervisor Architectures 22

23 Paravirtualization CPU and Memory Paravirtualization extends the guest to allow direct interaction with the underlying hypervisor reduces the monitor cost including memory and System call operations. gains from paravirtualization are workload specific Device Paravirtualization places a high performance virtualization-aware device driver into the guest paravirtualized drivers are more CPU efficient (less CPU overhead for virtualization) Paravirtualized drivers can also take advantage of HW features, like partial offload 23

24 Virtual Machines: Usage and Repositories Software development and testing Multiple environments for development and testing Software deployment Preconfigured virtual appliances Repositories of virtual appliances on the web 24

25 Why not Use Virtualization? Performance penalty Indirection through VM adds overhead Can be reduced through better virtualization support Hiding details of physical resources Some applications (e.g., DBMS!) make decisions based on assumptions about the physical resources 25

26 ... some numbers (SQL Server within VMware) 26

27 STORAGE VIRTUALIZATION 27

28 Spectrum of Storage Virtualization 28

29 Storage Virtualization Storage virtualization is a layer of indirection that allows the definition of virtual storage devices Basis Operations create, destroy grow, shrink virtual devices change properties size, performance, reliability RAID physical devices mirror 29 virtual devices

30 Benefits of Storage Virtualization Minimize/avoid downtime Non-disruptive LUN Creation, Expansion, Deletion Non-disruptive Data Consolidation, Data Migration Non-disruptive RAID Level Changes Improve performance Distribute and balance storage load Dynamic storage provisioning control placement Consistent Interface for All Storage Applications Tiering of Storage Using Heterogenous Storage / Replication Across Heterogenous Storage Improving Backup/Restore, Archival, etc Processes Application 1 Application 2 virtual devices Application 1 Application2 physical storage pool 30

31 Additional Capabilities File System Level Versioning, snapshots, pointin-time copies Local and remote mirroring Migration of virtual devices supports provisioning, hierarchical storage management Auto-administration policy-based management Storage QoS and performance isolation Database Level Storage snapshots and versioning backup and recovery concurrent applications Storage replication and mirroring dynamic DBMS provisioning Dynamic storage resource allocation accommodate workload fluctuations e.g., resizable tablespaces CPU and memory in the storage system caching offload DBMS functions 31

32 Storage Virtualization and DBMS-Tuning Characteristics of physical storage hidden from the DBMS (and DBA) How to Layout DB objects? Costing query plans Physical design Set DBMS parameters? page/extent sizes Prefetching costs 32

33 Storage Virtualization and DBMS-Tuning (2) should not be there!!! 33

34 End-to-End Physical Design Goal Solve both DB design and storage design problems while preserving virtualization. Two approaches 1. export virtual device characteristics to DBA weakens virtualization DBA/DBMS must model I/O costs DBA involved in layout 2. describe DBMS storage workload to storage administrator storage virtualization preserved storage administrator responsible for layout need to estimate DBMS storage workload can accommodate multiple storage clients 34

35 End-to-End Physical Design (2) 1. Ignore problem interdependency: DB design, then storage system design 2. Exploit existing design tools 3. Predict storage workload at design time 35 Ashraf Aboulnaga, Cristiana Amza, Kenneth Salem: Virtualization and databases: state of the art and research challenges. EDBT 2008:

36 The Layout Optimization Problem Map different physical objects to physical disk configurations observe capacity constraints avoid interference and balance load - size - workload Current Practice stripe-everything-everywhere separate index from data Oguzhan Ozmen, Kenneth Salem, Jiri Schindler, Steve Daniel: Workload-aware storage layout for database systems. SIGMOD Conference 2010: capacity - performance model 36

37 Layout Optimization: Experiments TPC-H (scale factor 5): Homogenous Devices 37

38 Layout Optimization: Experiments (2) TPC-H (scale factor 5)/TPC-C (sf 90): Disk+SSD 38

39 NODE VIRTUALIZATION AND DBMS 39

40 Databases: Why Use VMs for databases? Virtualization at hypervisor level provides the best abstraction Each DBA has their own hardened, isolated, managed sandbox Strong Isolation Security Performance/Resources Configuration Fault Isolation Scalable Performance Low-overhead virtual Database performance Efficiently Stack Databases per-host 40

41 DBMS on top of Virtual Machines... yet another application? No! Virtualization poses several interesting research questions for database systems Understanding the performance of database systems on virtual machines Configuring and tuning virtual machines running database systems Taking advantage of virtualization capabilities in the database system 41

42 Performance Overhead native Linux (L), XenoLinux (X), VMware workstation 3.2 (V) and User-Mode Linux (U). 42

43 Detailed Study Run TPC-H / PostgreSQL on bare machine Xen virtual machine on identical hardware Overhead of virtualization is not unacceptably high Can be made lower with better virtualization support Some new types of overheads are introduced and must be carefully managed 43

44 Resource Provisioning What level of resources to give to each DBMS? Configuring VM parameters How to tune the DBMS for a given level of resources? Configuring the DBMS parameters Need a model of how resource allocation affects database performance Need optimization or control algorithms to decide on the optimal resource allocation statically and dynamically 44 Ahmed A. Soror, Umar Farooq Minhas, Ashraf Aboulnaga, Kenneth Salem, Peter Kokosielis, Sunil Kamath: Automatic virtual machine configuration for database workloads. SIGMOD Conference 2008:

45 Number of CPUs Dynamic Workloads Bursty load 5 Bad response time Wasted resources 4 3 Peak 2 1 Average 0 Time (every 5 mins) Provisioning for dynamic workloads is hard! 45 45

46 Resource Provisioning - Design Advisor Example Workload 1: TPC-H (10GByte) Workload 2: TPC-H (10GByte) only Q18 (CPU intensive) (132 copies) Virtualization design advisor 20% of CPU capacity to Workload 1 80% to Workload 2 46

47 Virtualization Design Advisor I Performance Goals Performance Modell Workload Decription Configuration Advisor Ressource Allocation On-line Monitoring Performance Data Ressource Adjustment Ressource Allocation (Dynamic) 47

48 Some Experiments I Workload Definition based on TPC-H Q18 is one of the most CPU intensive queries Q21 is one of the least CPU intensive queries Workload Units C: 25x Q18 I: 1x Q21 Experiment: Sensitivity to workload Resource Needs W1 = 5C + 5I W2 = kc + (10-k)I (increase of k -> more CPU intensive) DB2 Postgres 48

49 Some Experiments II Workload Settings W3 = 1C W4 = kc Workload Settings W5 = 1C W6 = ki 49

50 DB STACK OPTIMIZATION IN VIRTUALIZED ENVIRONMENTS 50

51 Virtualization at Multiple Levels Take application/workload characteristics into account Scenario 1 different apps run on the same HW-platform Scenario 2 multiple instances of the same app (e.g. DBMS) run on the same HW-platform but for different users Organize virtual machines based on application characteristics Lesson 1 Group jobs with contrary resource requirements (CPU, I/O, ) Lesson 2 Group jobs with contrary SLA requirements 51

52 Mulitple Levels for DB-Scenario SQL Workload DB Layer Instance Layer DB Server Layer DB DB DB DB DB Instance Instance DB Server Physical DB Layout Advisor Indexes, materialized views, Memory Management Self tuning memory Bufferpools, Sort, Hash Join, Workload Manager VM Layer HW Layer VM VM VM VM Tools Monitoring Resources Configuration Migration 52

53 Layer Interactions General shortcoming: Static Environment Assumptions Advisors and tuning mechanisms expects static hardware environment VM expects static (peak) resource requirements Automatic VM configuration assumes fixed physical design and simple rule-of-thumb configuration Interactions between configuration components improve performance/utilization DBMS Configuration HW/VM Configuration 53

54 System/HW DBMS Component Interaction Analysis SQL Workload (or some abstraction) Physical Design (Design Advisor) Indexes Materialized Views Multidim. Clustering Denormalization Memory Management (STMM) Bufferpools Sort, Hashjoin, Lock, Other DB Configuration Compression? Main Memory Cache? Rowstore/Colstore? capacity! bandwidth??? mem! CPU? Storage (SAN, ) Capacity Bandwidth? VM Main memory CPUs speed # CPUs 54

55 Observation: Index Cost Explain (optimizer) does not list index maintenance costs Schema: tab1(id int, value int, rand varchar(35)) Index: create index idx1 on tab1(value) Query: update tab1 set value = value Implications? (design advisor ignores downside of indexes) execution with index execution without index 55

56 Observation: Memory <---> Estimated Costs Example: a SORT operator reacts on memory assignment Schema: tab1(id int, value int, rand varchar(35)) Index: create index idx1 on tab1(value, id) Query: select id, value from tab1 order by value small sort heap: 4837 large sort heap: 1553 no sort:

57 Observation: CPU <---> Estimated Costs Example: a SORT operator reacts on CPU assignment Schema: tab1(id int, value int, rand varchar(35)) Index: create index idx1 on tab1(value, id) Query: select id, value from tab1 order by value slow CPU: 1553 fast CPU: 1479 no sort: 683 (was 716) 57

58 Observation: Physical Design <---> VM Configuration Observation: Physical design decisions (advises) are based on I/O and storage bounds not CPU or memory. Index (vs. no index) Materialized view (vs. no materialized view) Normalized schema (vs. denormalized schema) estimated workload costs physical design 1 estimated workload costs physical design 1 physical design 2 physical design 2 CPU main memory 58

59 CONCLUSION 59

60 Conclusion Virtualization Powerful mechanisms for improving computing infrastructure Adopted by a wide range of organizations Database systems are increasingly being run in virtualized environments Significantly changes the operating environment At the same time can be very useful Many opportunities for database researchers 60