Adaptive Query Processing on Prefix Trees Wolfgang Lehner

Size: px

Start display at page:

Download "Adaptive Query Processing on Prefix Trees Wolfgang Lehner"

Erin May
5 years ago
Views:

1 Adaptive Query Processing on Prefix Trees Wolfgang Lehner Fachgruppentreffen, TU München Prof. Dr.-Ing. Wolfgang Lehner

2 > Challenges for Database Systems Three things are important in the database world: performance, performance and performance. Bruce Lindsey M. Stonebraker Extreme data Extreme performance Dynamo Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 2

> Challenges for Database Systems Extreme Performance

time (schema definition) - during database lifetime

(scheduling, ) Extreme Flexibility Extreme Data Prof.

3 > Challenges for Database Systems Extreme Performance Flexibility in Database Systems + during deployment time (schema definition) - during database lifetime (schema evolution) - during query runtime (scheduling, ) Extreme Flexibility Extreme Data Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 3

comes second querying Web-Tables role-based object

statistical operators query processing data model-

operating system & hardware StorageClassMemory

4 Database System > Flexibility from feet applications data comes first, schema comes second querying Web-Tables role-based object models Open Data platforms Flexibility is cross-cutting alternative query model (drill-beyond) record management logging an persistency HW/SW DB-CoDesign (MV)CC domain-specific data models Demand flexibility statistical operators query processing data model- related storage model- related Provide flexibility operating system & hardware StorageClassMemory MegaCore systems TeraByte-Capacity GPUs FPGAs/FPPAs Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 4

> Example 1: Data-driven Schema Design 1998 2002 2006

Dimensions/Weight Dimensions/Weight/ Form Factor

USB, WiFi Dimensions/Weight GSM, UMTS, HSPA+, Bluetooth,

Display DotMatrix Display 2.0, QVGA, Color 3.

3, 800 480, Touch Standby/Talk Time Standby/Talk Time

#Names, #SMS #Names, #SMS Memory/Card Slot Memory

Ringtones/MP3 Player/Browser Browser/Exchange

5 > Example 1: Data-driven Schema Design GSM GSM TriBand GSM, GPRS, Bluetooth, FM Dimensions/Weight Dimensions/Weight/ Form Factor Dimensions/Weight/ Form Factor GSM, UMTS, Bluetooth, USB, WiFi Dimensions/Weight GSM, UMTS, HSPA+, Bluetooth, FM, USB, WiFi, agps, HDMI Dimensions/Weight DotMatrix Display DotMatrix Display 2.0, QVGA, Color 3.5, , Touch 4.3, , Touch Standby/Talk Time Standby/Talk Time Standby/Talk Time Battery Capacity Battery Capacity #Names, #SMS #Names, #SMS Memory/Card Slot Memory Memory/Storage/Ca rd Slot Polyphonic Ringtones MP3 Ringtones/MP3 Player/Browser Browser/Exchange Browser/Exchange 3MP Camera 3MP Camera Front/Rear Camera/HD Video CPU, OS CPU, GPU, OS Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 5

6 > Example 2: Web Data Management Web Data: A Multitude of Data Models and Sources Example 1: Queries spanning millions of data sources with as many schemata Example 2: Augment local datasets with (external) third-party information select from cust, where nations.gdp > $1B Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 6

> Example 3: Energy-Adaptive High-Speed Computing

Efficient Computing Optical Interconnect adaptive

ng technologies (e.g. 3D stacking of Si/III-V

on-chip/on-package antenna arrays analog/digital

7 > Example 3: Energy-Adaptive High-Speed Computing Platform SFB912: HAEC Highly Adaptive Energy Efficient Computing Optical Interconnect adaptive analog/digital circuits for e/o transceiver embedded polymer waveguide packaging technologies (e.g. 3D stacking of Si/III-V hybrids) 90 coupling of laser Radio Interconnect on-chip/on-package antenna arrays analog/digital beamsteering and interference minimization 100Gb/s GHz channel 3D routing & flow management Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 7

operator 1 Intermediate Indexed Tables Operators exchange clustered es 3 aggregate 1 Prefix Treebased Indexes

8 > QPPT Overview 3 Composed Operators Construction of composed operators, like multi-way-select-join-group-sort Avoids Intermediate Materialization 2 Cooperative Operators Operators optimize their output for the next operator 1 Intermediate Indexed Tables Operators exchange clustered es 3 aggregate 1 Prefix Treebased Indexes CIDR In-Memory Database Cluster Indexes Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 8

9 > Prefix Trees Prefix Trees Unbalanced Balanced read/write performance Main memory-optimized Efficient parallel access B+-Trees Balanced Low update performance Disk-optimized Low scalability Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 9

10 > Index Landscape KISS-Tree 32bit key Buzzard NUMA optimized Clustered Column Store Scan optimized Adaptive Radix Tree (ART) Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 10

Heterogeneous in-memory structure Combination of direct and indirect addressing Takes advantage of virtual memory

11 > KISS-Tree Overview Properties Specialized version for 32bit keys Latch-free updates Order-preserving 2-3 memory accesses per key Comparable fast to reported order-preserving in-memory es for read access BUT: High update performance Heterogeneous in-memory structure Combination of direct and indirect addressing Takes advantage of virtual memory management Enables different compression mechanisms DAEMON 2012 Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 11

> Level 1: Virtual Level Decimal Key: 327914 0000 0000 0000 0101 0000 0000 11 10 1010 Split key in 3 fragments 16bit (f 1 ) 5 10bit (f 2 ) 6bit (f 3 ) Calculate corresponding 2nd node address from

12 > Level 1: Virtual Level Decimal Key: Split key in 3 fragments 16bit (f 1 ) 5 10bit (f 2 ) 6bit (f 3 ) Calculate corresponding 2nd node address from the first fragment Direct addressing No memory required for level 1 Level 2 Node Node Node Node Node Node Node Node Requirement for Level 2: All nodes have to be stored sequentially in memory Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 12

> Level 1: Virtual Level Decimal Key: 327914 0000 0000 0000 0101 0000 0000 11 10 1010 Split key in 3

4 KB 4 KB Page Directory Memory Management Unit (MMU) Hardware Operating System 4 KB Physical Memory Address

13 > Level 1: Virtual Level Decimal Key: Split key in 3 fragments 16bit (f 1 ) 10bit (f 2 ) 6bit (f 3 ) Level 2 Virtual Memory Address 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB Page Directory Memory Management Unit (MMU) Hardware Operating System 4 KB Physical Memory Address 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 13

14 > Level 2: On-demand Level Decimal Key: bit (f 1 ) 10bit (f 2 ) 6bit (f 3 ) Change over to indirect addressing 1024 buckets per node containing a compact pointer to the 3 rd level node 256 MB maximum Level 2 Virtual Memory Address 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 4 KB 32 bit = 1024 pointer to L3 leaf nodes = potential L3 leaf nodes of a size between 2 0 and 2 6 Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 14

> Level 2: On-demand Level 0000 0000 0000 0101 0000 0000 11 10 1010 4 KB 10bit (f 2 ) 32bit 1 2 3 1022 1023 Number of existing elements in L3 leaf

each of the 64 node sizes possible on the third level Each segment consists of a maximum of 2 26 blocks of the respective node size 1 63 64 2 26

15 > Level 2: On-demand Level KB 10bit (f 2 ) 32bit Number of existing elements in L3 leaf node (compression) 6bit 32bit 26bit Position of L3 leaf node in corresponding memory segment Allocation of consecutive virtual memory segments for each of the 64 node sizes possible on the third level Each segment consists of a maximum of 2 26 blocks of the respective node size blocks of size 1 pre-allocated 2 26 blocks of node size 64 pre-allocated Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 15

16 > Level 3: Compression Decimal Key: bit (f 1 ) 10bit (f 2 ) 6bit (f 3 ) possible node sizes Bitmap indicates which bucket is in use only existing values are stored 64bit Bitmask s Read-copy-updates Compare-and-swap No in-place updates possible (lost updates) Thread-Local Memory Management Subsystem Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 16

> Duplicate Handling Efficient duplicate handling necessary for query processing

Segment Size 64Byte 128Byte 4KB Physical Memory (4KB Pages) Hardware Prefetching

sequentially in 4KB blocks blocks grow exponentially until reaching 4KB trade-off

17 > Duplicate Handling Efficient duplicate handling necessary for query processing Scanning a linked list results in random memory accesses Page Size Limit Growing Segment Size 64Byte 128Byte 4KB Physical Memory (4KB Pages) Hardware Prefetching Boundary Page boundaries are a barrier for hardware prefetchers store values sequentially in 4KB blocks blocks grow exponentially until reaching 4KB trade-off between scan performance and memory consumption Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 17

> Prefix Tree Performance Insert/Update Performance Lookup Performance Performance depends on the number of memory accesses per key KISS-Tree: 3 memory accesses per key

18 > Prefix Tree Performance Insert/Update Performance Lookup Performance Performance depends on the number of memory accesses per key KISS-Tree: 3 memory accesses per key Latency Hiding with Batch Updates/Lookups KISS-Tree performs better than hash table implementations Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 18

19 > Cooperative Operators Each operator adjusts its output to the requirements of the successive operator traditional operators mostly build internal es anyway skip plain tuple exchange overhead aggregate provide high-quality input to each operator date.datekey lineorder.orderdate supplier.suppkey lineorder.suppkey part.partkey lineorder.partkey date supplier part lineorder Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 19

20 > Composed Operators (1) Materialization is costly; thus try to completly avoid it! aggregate integrate operators that materialize large intermediate results date.datekey lineorder.orderdate lineorder.suppkey supplier.suppkey part.partkey lineorder.partkey date supplier part lineorder Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 20

21 > Composed Operators (2) Materialization is costly; thus try to completly avoid it! 3-way-select-join aggregate date.datekey lineorder.orderdate supplier.suppkey 3 lineorder.partkey date supplier part lineorder Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 21

22 > Evaluation (1) Star Schema Benchmark (SSB) Scale Factor = 15 Intel i7-3960x 15 MB LLC 3.3 GHz (Turbo Mode disabled) 32 GB Main Memory Ubuntu Single-threaded execution Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 22

23 Execution Time [ms] > Evaluation (2) MonetDB Commercial DBMS DexterDB Q1.1 Q1.2 Q1.3 Q2.1 Q2.2 Q2.3 Q3.1 Q3.2 Q3.3 Q3.4 Q4.1 Q4.2 Q4.3 (*) : we are in touch with MonetDB; performance numbers are under investigation Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 23

Batched KISS-Tree Lookups/Inserts SSB Query 4.

24 > Evaluation (3) SSB Query 1.1 Single Join Select-Join operator avoids intermediate result materialization Batched KISS-Tree Lookups/Inserts SSB Query Joins over 5 Tables Multi-way join operator avoids intermediate result materialization Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 24

25 > Adaptive QPPT (1) Flexible Query Execution on Adaptive Hardware aggregate date.datekey lineorder.orderdate supplier.suppkey 3 lineorder.partkey date supplier part lineorder Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 25

part lineorder supplier date lineorder > Adaptive QPPT (2) Parallel Running Query 3 Board 1 Board 2 3 3 reduce hops by wireless point-to-point interconnects operator

26 part lineorder supplier date lineorder > Adaptive QPPT (2) Parallel Running Query 3 Board 1 Board reduce hops by wireless point-to-point interconnects operator data / data operator shipping adaptive hardware components (e.g., trade cache for compute units) Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 26

27 CMOS (industry focus) > Comprehensive Approach: 9 Paths More-Shots-on-Goal Information Processing Devices & Circuits Materials & Functions H HAEC: Highly Adaptive Energy-Efficient Computing A Silicon Nanowires B Carbon F Orchestration G Resilience C Organic D Biomolecular Assembly E Chemical I Biological Systems Prof. Dr.-Ing. Wolfgang Lehner Adaptive Query Processing on Prefix Trees 27

28 Adaptive Query Processing on Prefix Trees Wolfgang Lehner Fachgruppentreffen, TU München Prof. Dr.-Ing. Wolfgang Lehner

Main-Memory Centric Data Management Open Problems and Some Solutions

Main-Memory Centric Data Management Open Problems and Some Solutions Wolfgang Lehner Technische Universität Dresden Prof. Dr.-Ing. Wolfgang Lehner > The Reality: The Petabyte Age New Realities TB disks